Ummm, kind of unsure about this, but have you always had 110k vertices for clothing?
It takes about 0.5 ms for just 350 vertices on my system using 4 iterations and 1 substep (i7 13700k).
I’m not sure if the relation is linear or if this is the actual reason for the performance hit but might be worth looking into