Solution
SetHiddenInGame replicates at the Actor level, but it doesnt if you change it at the **component **level instead.
So you can get the client to do whatever you want if you use USceneComponent::SetHiddenInGame instead of the Actor equivalent.
Rama