Okay, I figured out the solution for anyone else that might be struggling with this!
The issue was caused by a large capsule inside of the physics asset.
You just simply select it, delete it and your cloth object should work perfectly!
Here is a video that helped my solve this issue.
Look at around the 9 minute mark.