You definately want to stay away from the cheapy, dollar store variety of velcro as the tape peels right away from the velcro. If you don't trust the sticky backing of the industrial stuff, you can get the sew-on kind at most fabric stores (it comes in rolls) and glue it to your armor.
GOOP is a great glue for sticking velcro to cloth, but it seems to take forever to dry on anything solid. Superglue will do the trick if you prep the armor first with a rough grit sandpaper. If the surface is smooth, the possibility of it popping off is likely. I have also had pretty good luck with Elmer's ProBond, but you have to keep an eye on it because it expands as it dries. You'd definately want to clap the velcro into place.