Lock to hmd and setup late update troubles? :(

Development Platform & Builds XR Development
,
1

Hi, I am making a mirror in vr and am using scene capture components to capture the scene. The basic flow of what I’m doing is getting the characters camera position, using that position to calculate the proper scene capture components location and rotation, then projecting it on the mirror. Initially, there was quite noticeable delay between moving the hmd and what was shown in the mirror. I was able to minimize most of the delay by setting an appropriate tick group on the mirror actor, but still have a very small (but still very noticeable) delay. Setting the camera’s setting lock to hmd to off removes this delay completely, but then the movement of the hmd is slightly jittery, almost like it’s lagging behind the actual physical movement of the device and feels very unnatural.

I’ve identified the following function in c++ that introduces this.
XRCamera->SetupLateUpdate(ParentWorld, this, bLockToHmd == 0);

I could not figure out how to work around this problem and would appreciate any ideas that come your way!

2

The late update applies the camera pose from the next frame when the render thread begins rendering, so the pose you get in the game thread will always be at least one frame behind.

If I remember right, scene capture components render even before the XR plugins get the late update poses.

I’m not sure if there is any easy solution to this. The ones I can think of is either modify the scene capture component to support late updates and rendering (and doing the pose calculations there), or reprojecting the output texture somehow when sampling from it in the main scene. Both would likely require a lot of work.

Turning off late updates might also be an option (although not ideal). I think the worst of the lagging comes from the XR plugins assuming you always have it on and submitting the wrong rendering poses, making the VR runtime reproject the scene wrong. The plugins would have to be modified to account for it.

3

Yes, as I’ve mentioned turning the late update off solves the issue, but then the camera itself feels a bit jittery. Basically, every heartbeat can be seen as it manifests as a small jitter.

I’ve tried using the SetupLateUpdate on the capture scene components, but that seemed to have no effect at all. I’ve also tried using the camera’s OnUpdateTransform to position and capture from there, thinking those would only fire after the late update, but that did not seem to affect it at all.

I’ve tried using FLateUpdateManager class, but was not able to set it up correctly. Are you familiar?

4

SetupLateUpdate seems to be for the XR camera only, since it’s controlled by the XR plugin, and I think OnUpdateTransform runs in the game thread.

With FLateUpdateManager, it seems you would need to manually call Apply_RenderThread from the render thread at the right time to update the pose.

It’s been a while, so unfortunately I can’t fully remember how the rendering infrastructure is set up. It’s possible that the engine needs modifications to allow the scene capture to render in the correct place.

5

I’m quite out of ideas at this point. Been trying to solve this for a month plus with no luck. If you have any other pointers that I could look into, Id love to hear it. Cheers!

6

Unfortunately I’m out of ideas. Hopefully someone with good knowledge of the renderer can help.

7

Hi there, I know time has passed but during a personal project I experienced the same problem. I was doing a portal in VR ( same issue) and I kind of solve it ( not perfect). I woul like to show mye insights to try help finding a final solution.

In some months I will actually write a blogspot about this, but here there is a little resume:


class XRBASE_API  FDefaultXRCamera : public IXRCamera, public FHMDSceneViewExtension
{

. . .

virtual void PreRenderViewFamily_RenderThread(FRDGBuilder& GraphBuilder, FSceneViewFamily& InViewFamily) override;
 
}

This last function is the latest one, as far as I am concerned, that calls for the late update ( Apparently late update is implemented in more than 1 place). I think Unreal calls in multi-threading all the FSceneExtension in groups. This means that if I write my logic in a FSceneExtension function that runs AFTER this one I will be able to access the correct pose of the camera.

I did not have time to explore more, but found 2 promissing functions:

/**
 * Called on render thread at the start of rendering, for each view, after PreRenderViewFamily_RenderThread call.
 */
virtual void PreRenderView_RenderThread(FRDGBuilder& GraphBuilder, FSceneView& InView) {}

Technically runs after the late update, but the extension for the hmd also implements this function, so this could be not exactly perfect depending on order execution ( really small time difference). The good thing is that it provides the view.

/**
 * Called on render thread right before Base Pass rendering. bDepthBufferIsPopulated is true if anything has been rendered to the depth buffer. This does not need to be a full depth prepass.
 */
virtual void PreRenderBasePass_RenderThread(FRDGBuilder& GraphBuilder, bool bDepthBufferIsPopulated) {}

This one looks like it really happens after the late update, but did not have the views and I did not explore.

In the end I used the `PreRenderView_RenderThread() as I had the view. My solution inside this function was to ask directly the current relative position of the hmd, offset of each eye and make the calculation myself:

void FPortalViewExtension::PreRenderView_RenderThread(FRDGBuilder& GraphBuilder, FSceneView& InView)
{
    . . .  
    GEngine->XRSystem->GetCurrentPose(HMDDeviceId, CamRotationLocal, CamLocationLocal);

    . . .
    GEngine->XRSystem->GetRelativeEyePose(HMDDeviceId,EyeIndex, offsetRotator, offsetLocation);

    FVector EyeLocation = CamRotation.RotateVector(offsetLocation) + CamLocation;

    FVector eyeSyncronizedLocation;
    PortalTools::XR::TeleportWorldLocationMirroredXR(Portal->OtherPortal, Portal, EyeLocation, eyeSyncronizedLocation);
    FQuat eyeSyncronizedRotation;
    PortalTools::XR::TeleportWorldRotationMirroredXR(Portal->OtherPortal, Portal, CamRotation, eyeSyncronizedRotation);
    . . .

This means that my approach is almost right. The differences between the position that the late update and my calculation uses is negligible, specially if the framrate stays good.

I hope it helps, let me know if you see a better approach or gives you new ideas, I leave also videos of how it was before and after. The video shows the game running and I tap into the headset to show better the results.

Stutter / shaking problem:

After solution (Arrow shows tap effect on debug text, but portal is fine)