There are projects with a lot more concurrent players in a server than yours, and the controller and HMD replication is actually not the heaviest part of the replication (though more prone to being noticable since it does not have rollback).
If I were you I would make sure first that the update rate is reasonably set (30-45htz), and then secondly that your bandwidth settings for the engine are NOT STOCK. The stock settings are very very conservative and downright too low for most normal games now aday’s, let alone VR ones where there is a lot more communication passing through.
For the client rates: Default Engine.ini
[/Script/OnlineSubsystemUtils.IpNetDriver]
MaxClientRate=15000
MaxInternetClientRate=15000
[/Script/Engine.Player]
ConfiguredInternetSpeed=15000
ConfiguredLanSpeed=20000
For the server bandwidth: DefaultGame.ini
[/Script/Engine.GameNetworkManager]
TotalNetBandwidth=64000
MaxDynamicBandwidth=15000
MinDynamicBandwidth=4000
Those values are from the example template not the stock engine settings. You could use them as a starting point to tweak higher or lower based on the net consumption of your project.
If you are still hitting net saturation then you really need to run some network profiling and figure out if you have something else sending around a lot of data.