Looking further into it, disabling running start with the AllowAsyncRunningStart command from the SteamVR debug menu seems to make the the issue disappear.
With running start:
Without it:
