Add more details here for anyone comes after:
NetworkPhysicsComponent has a method SetIsRelayingLocalInputs which says
Mark this as controlled through locally relayed inputs rather than controlled as a pawn through a player controller. Set if NetworkPhysicsComponent is implemented on an AActor instead of APawn and it’s currently being fed inputs, or if this is controlled by the server. NOTE: The actor for this NetworkPhysicsComponent also needs to be owned by the local client if this is used client-side.
So for any component that drived by NetworkPhysicsComponent, and not controlled from PC, you need to call this method.
and UModularVehicleBaseComponent::SetLocallyControlled calls it.
UE 5.6.