@GuacFrog I seem to be able to switch owners and switch who simulates physics no problem. I added an empty Actor in the Third Person Template, added a Cube to it and set the cube as the root component of the Actor, and also enabled replication of course. I added a SmoothSync component and some blueprint magic and an rpc to change ownership and physics.
Everything seems to work as expected. The only time I was able to get behaviour similar to what you are seeing is when I enabled “Component Replication” on my root primitive component (cube). That seemed to cause all sorts of weird twitching, though I’m not really clear why that would be, it’s definitely doing something bad. Do you have component replication enabled?