AttachToComponent and network relevancy

In my game world you have Actors that can be “picked up” with AttachToComponent by Characters. If this happens and the character who is picking up the Actor is outside your relevancy the Actor stays there as it does not know what to attach to. Only when the character returns to relevancy the pickupable Actor snaps in to his hands via a notify OnRep.

I suppose I could make every Character and pickupable Actor bAlwaysRelevant=true but this seems extreme. I imagine this problem has been tackled many times before in capture the flag like games. What is a good way to solve this problem?

Thanks in advance