Controlling another clients components

So I am trying to make a physics object pickup system work through multiplayer, but clients aren’t as smooth as a want.

To fix this I would like to have the server tell the client who is picking up an item, then the client will give that pawn a physics handle and assign that handle the object and then turn off replication on that object. Because the pawns are replicated, it should look like the picked up object is in the same location across clients.

On dropping the object, turn replication back on.

My concern is that I can’t have a client can’t tell another pawn that is controlled by another player to add a physics handle component (since each client is doing this, it will almost look replicated, but with the benefit of being local). Does any one know another about something like this?

I tried passing the pawn that is doing the picking up from the server to client through multicast, but I can’t quite get it to work.