So here is a video showcasing my problem, hopefully it’s more obvious this time.
I modified my code given above just to change SetActorRotation to AddLocalRotation to the sphere component which I also changed to a cube to make it easier to visualize.
You can see that the problem is that the direction the spheres move from the players perspective changes at different locations. The spheres should appear to be moving the same direction relative to the player regardless of the starting position of the player.