Ball physics problem

These are all flat surfaces, but the collisions and BSPs create lots of “lines”. And when the ball rolls over one of those lines or hits one, it bounces off of them, even though it is actually flat.
What could be the reason and a possible fix here?