It turns out there was a very simple solution to this, which was to uncheck “Lock to HMD” on the camera. I honestly have no idea why this fixes the problem, but it does.
If you need the camera to still follow the HMD, you can then just set the position manually on tick by using the Get Tracked Device Position and Orientation and then setting the device ID to 0.
