Dynamic spring arm rotation

So I am making this roller ball with dynamic gravity. It works fine, but how do I make spring arm limits dynamic to my gravity direction? (I have gravity direction vector variable)
It works just fine on the ground, but when I get to wall/roof, it rotates weirdly as expected, here’s a video about it

Please someone help, I have tried to figure this out too long…