I'm sorry - it was only the soft part I had on mine as well when I got it! My mistake.
I had some industrial strips of velcro that I attached to the other side. Because of the weight of various objects in one of my pouches (chest lights, amp, etc.) it was pulling the sticky part of the velcro loose from the belt, so I ended up hot gluing it in place. That held for some time, but it loosened again, but then I used E6000 glue to keep it stuck.