Be careful if you decide to sand it using a machine. I used my drill to do mine and it caught on a bad bit in the helmet, leapt out of my hands and tried to take several layers of skin off my hands and arms.
sand it by hand buddy
Very good advice, but you're using a drill. Have you ever used a Black and Decker "Mouse" hand sander? It's a different animal and what I used to clean out the interior of that helmet on reader was most kind to link. A drill is a rotary tool, where a sander is a vibrational tool. Both remove material differently. With a drill, there is a risk of drag -- as you pointed out. With a vibrational sander like the "Mouse" you have no such danger except just letting it go and letting it eat away at the inside of your helmet without control. But it won't eat through it like acid. Just pick it back up. There is a huge difference in danger to your hands unless you are purposely sanding your hands.
I'm not saying the "Mouse" is the holy grail tool to use, but there are other makes and shapes of handheld sanders depending on your needs. I've worked on five Vader kits so far, and the "Mouse" has been extremely helpful to me though I do wish I had something a bit more specialized.
But I do also need to point out that until godaking shows us a photo of what kind of FG he is dealing with, the advice we're giving is a guess in the dark. If he's dealing with the swirly random pattern FG mat, then simple hand sanding with a certain grit level of sand paper would be fine. If he's dealing with what I dealt with in the Golden Armor product (the cottage cheese chunky stuff with very thick pokey FG) then I would suggest reconsidering using an electic sander or Dremel properly.
So, godaking, what are you actually dealing with?