Hi, I am making a mirror in vr and am using scene capture components to capture the scene. The basic flow of what I’m doing is getting the characters camera position, using that position to calculate the proper scene capture components location and rotation, then projecting it on the mirror. Initially, there was quite noticeable delay between moving the hmd and what was shown in the mirror. I was able to minimize most of the delay by setting an appropriate tick group on the mirror actor, but still have a very small (but still very noticeable) delay. Setting the camera’s setting lock to hmd to off removes this delay completely, but then the movement of the hmd is slightly jittery, almost like it’s lagging behind the actual physical movement of the device and feels very unnatural.
I’ve identified the following function in c++ that introduces this. XRCamera->SetupLateUpdate(ParentWorld, this, bLockToHmd == 0);
I could not figure out how to work around this problem and would appreciate any ideas that come your way!
The late update applies the camera pose from the next frame when the render thread begins rendering, so the pose you get in the game thread will always be at least one frame behind.
If I remember right, scene capture components render even before the XR plugins get the late update poses.
I’m not sure if there is any easy solution to this. The ones I can think of is either modify the scene capture component to support late updates and rendering (and doing the pose calculations there), or reprojecting the output texture somehow when sampling from it in the main scene. Both would likely require a lot of work.
Turning off late updates might also be an option (although not ideal). I think the worst of the lagging comes from the XR plugins assuming you always have it on and submitting the wrong rendering poses, making the VR runtime reproject the scene wrong. The plugins would have to be modified to account for it.
Yes, as I’ve mentioned turning the late update off solves the issue, but then the camera itself feels a bit jittery. Basically, every heartbeat can be seen as it manifests as a small jitter.
I’ve tried using the SetupLateUpdate on the capture scene components, but that seemed to have no effect at all. I’ve also tried using the camera’s OnUpdateTransform to position and capture from there, thinking those would only fire after the late update, but that did not seem to affect it at all.
I’ve tried using FLateUpdateManager class, but was not able to set it up correctly. Are you familiar?
SetupLateUpdate seems to be for the XR camera only, since it’s controlled by the XR plugin, and I think OnUpdateTransform runs in the game thread.
With FLateUpdateManager, it seems you would need to manually call Apply_RenderThread from the render thread at the right time to update the pose.
It’s been a while, so unfortunately I can’t fully remember how the rendering infrastructure is set up. It’s possible that the engine needs modifications to allow the scene capture to render in the correct place.
I’m quite out of ideas at this point. Been trying to solve this for a month plus with no luck. If you have any other pointers that I could look into, Id love to hear it. Cheers!