Choppy item movement replication in VR

Hello,

My problem is one of item movement replication in VR. When a player picks up an object, he will send out a message to every other player to stop simulating physics on that object, and then continually update the other players with that object’s new location through RPC calls. The player who picks up the object is called the owner. This system works well for 2 players, but when a second client joins the server (3 players total now), the owner will get a noticeable chop in the item they’re holding when they move it around. This is perplexing, since the owner is the one who updates other players of the object’s location, and thus he should not be noticing any lag what so ever, and yet he does. The new client who just joined does receive a message to stop simulating physics on the owner’s held object if the new client joined while the object was being held.

Does anyone have any thoughts on why this issue would occur, and make a great two player game become an unplayable three player game?

Thanks,