Cloth ignores Physics asset until i change the Colliders to a ridiculopus size

Hello everyone.
Lately ive been having this problem with Cloth to where it doesnt seem to be affected by the Physics Asset but when i change the size of the colissions to something really big, the cloth mesh starts to react to it.
Anyone knows something about this? any suggestions?

im also having this same issue in 4.26