How to get soccer net to dynamically react to a soccer ball?

So i made a soccer net (the frame is a separate object) and turned it into a cloth object with painted weights etc. My problem is no matter what I do the ball keeps going through the net or when I add a collision capsule into the physics asset it stops the ball completely.
I have the ball set to interact with the environment and a collision sphere on it. Not sure what Im doing wrong or missing, any input or advice would be greatly appreciated.