I think you'll find people have used either method with success.
I'm a velcro fan myself. I attached my armor using industrial strength velcro. I used 'goop' glue to attach one part to the back of my armor pieces, and 'no-sew' fabric glue to attach the opposing part to my vest.
It hasn't ever peeled up and once I press it into place it doesn't shift.
I did the same thing to add some extra stability for my knees as well. Just a small square of velcro attached to the knee of my flightsuit, and a piece on the back of the knee armor.