Nasty edges during cloth simulation

I believe the issue was due to self penetration of the cloth mesh.

Because the creases are so close to each other, every area where the cloth paint exceeds the distance between the 2 creeses is prone to confuse the simulation - even with self collision enabled.

Instead of loosing your mind withe painting cloth in the engine…

First, well, you should pull the nvidial tools source and build the nvcloth tool from source to paint the apex cloth as it was originally intended to be painted.
Now, the whole process is painful, the software is decades old, and the source for it is gated by being approved as an nvidia dev, so it isnt for everyone.

Those who cannot/wont/dont want to can levarage Vector paint.
More specifically, use my blender plugin to visualize your cloth.