I had the exact same problem, and I used the same approach you did ... almost. I sanded, shot a layer of automotive primer, let it dry, sanded with 800 grit, shot another layer of primer, let it dry overnight, sanded again with 800 grit, and then RnBed.
So It's the same process, except I used a buffing cloth instead of my hand. I tried my hand, but there are 2 problems that made me abort that method. #1) My hand began sweating too much and I noticed streaks on the armor (perhaps this happened to you.) #2) Sometimes my hand would get "caught" in the process (i.e. squealing tires vs. non-squealing tires ... or static friction vs. kinetic friction.)
I used the buffing cloth ... just a very smooth cloth (or an undershirt), and was buffing to the point that I got a blister on my hand. I had to build up A LOT of heat to get it even and shiny.