Ok, I figured it out 
I had Z offset on the spring arm to make camera go higher than the ball. When instead I should have Z offset on the camera itself.
That was causing the issue. After I reset spring arm coordinates, it stopped rolling around with the ball.
However, side effect was - when looking from top-down, camera was rolling over the ball, cutting off vision at the bottom of the screen:
So I attached a spring arm with zeroed coordinates to the ball, and then another one to this spring arm:
This fixed the issue with both rolling and camera behavior:
