I do what superjedi does, but after taking a couple other steps. I find where the actual edge lines should be and mark them with a fine-tip marker. Then I grind down to that all the way around with a handheld electric finishing sander. That will just leave a bump in the middle of the back of the ear that I then sand down flat using the method above. I rotate the earpiece slowly as I go, so I don't accidentally take too much off one side without noticing. Yeah, it takes a while, but I prefer having that degree of control over material removal. You can always take more off. It's harder to put back on.
--Jonah