Thank you everyone for sharing your work around,
Here I found something that is working for me at the moment.
The physics seems to be related with the Temporal Sample count as soon I put it down to one the Clothing sim started to behave in a more realistic way.
Try playing around with the Temporal Sample count to have a better sim and if you want better quality in your image bump your Spatial sample count.
Is my first time being useful here, thank you to everyone this forum has been really helpful to me.