@Fjordhoj
Smooth Sync uses ownership to determine who is the authority of the position. It sounds like your client owns your actor. Set the server as the owner or have no owner to have the server determine the position. Then if you move it on the server, that position will be sent out to all clients.
Let me know how it goes or if you think I misunderstood.