You might get better results by adding force on Server through client-to-server RPC, and replicating the ball movement.