No, just the collar piece. I can't get my head through the vest with the collar attached so I just have to remove that part... The rest of the armour stays attached to the vest along with the back plate, jetpack and harness. I put the lot on in one go. 
Ah, good to know that it's not just me that melts while suiting up! 
I definitely need to look into helmet fans too!