Here's what I did to knock some of the 'sheen' off my vest. I misted it with taupe spray paint, then gray primer, then a very light misting of flat black. I finished it off with some Dull Cote and that did the trick. Be real careful with the black though. If you are too close, or don't shake the can well enough, it may 'spit' on your vest.
yeah I did the mist technique with lt. grey & primer, and had a bunch of broken up sand dust kinda stuff in the back of my truck(sandbags for the winter, damn rear wheel drive!) and dusted the vest up with that before the paint had dried. It worked pretty good I feel. I think chalk would work the same way. Good luck!