Ok so apparently the default setting for chaos physics iterations is very low and not fit for a physics-centric project. Changing that solved almost everything for me and I feel very stupid rn.
Sadly this also means the performance gain I saw over Physx is fake.