I was able to find a solution that works effectively for this, I decided to apply the rotation on the client side first, then make the call to the server to then set the same rotation again. That way the client gets quick instant rotation allowing a smooth visual while the server is happy the position matches.