We will need more information on how exactly the “manager” actor ticks the movement. Are you sure this event happens on both client and the server?
The jitter is probably because the actors are moved only on the client and the server then replicates their initial position so they move back into place.