When the impact force multiplier exceeds 1000 - the ball jerks and slowly rolls in the opposite direction.
If you strike directly or backwards, everything is fine. But it is necessary to turn sideways, or 45 degrees - the ball rolls either forward or backward.
im no expert, but Rotation from x vector does not consider the y vector i guess.
i would try to use radial Impulse component (add component -> add rad force comp) and instead of add impulse you use “fire impulse” where radial Impulse is your target