Engine Mod to do post-HMD work (for VR portals) which option will be most favourable to Epic?

Development Programming & Scripting C++
, , ,
1

So, a bit of background, I have a marketplace item “Portals Blueprint” - which does not work in VR for several reasons (mostly the steroscopy is hard to grab from head and hmd rotation happens after all other work is done, but I need to do the portal caputure after HMD rotation).

I’ve taken some time to thourghly check over the SceneCapture2D (which I use for the portal system) and PlanarReflection (another type of capture, which works with VR)

Looking at the relevent rendering part of the code ( here https://github.com/EpicGames/UnrealE…ring.cpp#L3158 )
it seems regular scenecapture2d happens right before HMD rotation, and planarreflection right after.

Though the scenecapture2d hooks back to the component (momenterally), the planarreflection does not, instead using an engine module to position the camera module and handle stereo.

So, the upshot of this seems that I cannot implement VR supporting portals as an external module, and finally have a reason to write some engine code and submit a PR, but not sure which route to go down: [HR][/HR]

1. Add another tick group, in the rendering section, between HMD rotation and planar reflection
-pro: minimal change, extnesible to do other VR things
-con: ugly, slow, does not help with stereo issue (but that should still be resolveable…)

2. Extend SceneCapture2D to have a flag “bAfterHMD”, hook back to the component to give it a chance to alter the scenecaptures transform
-pro: fairly optimal, “cleanish”
-con: not actually very clean, still does not really solve the stereo capture issue

3. new SceneCaptureHMD class, add new code similar to planar reflection which will capture with correct stereo for equipped HMD, but store to a user-defined rendertarget (or two), perhaps have a “bAfterHMD” flag to optionally avoid inducing delay in HMD when not needed
-pro: fairly optimal, solves both rotation lag and stereo issue, keeps in style of existing code better
-con: involves writing quite a bit more engine code, not sure how useful it will be for things other than my portal system [HR][/HR]

Overall, I am not quite happy with any of these ideas, (3) is perhaps my preferred, but I have doubts about if Epic will approve, (1.) also sounds tempting, but not sure about the practicality of having a tick group called from the render thread, may be a mortal sin to do that after HMD rotation.

Any feedback or suggestions from Epic staff, Enging contribitors or anyone else is very welcome - I’d really like to get a good portal system working

Also looking into better portal physics, it seems physx allows for hooking the collisions with a custom callback, which should allow me to do physics better on an object partially intersecting a portal, but that is a topic for another post :slight_smile:

2

If you were to do #1 (Add new Tick Group PostHMDUpdate), wouldn’t you be able to also use that in a similar way as #3 (Write new SceneCaptureHMD class). This would also leave open door for people to write these sort of C++ extensions that need to take place after HMD rotation.

Or would it be possible to request for current HMD Transform in the latest available tick.

Or run the HMD update twice, once before a last ditch tick, but then update again to reduce its lag, but get a better position for the other tasks like the captures.

3

I think in older versions of the engine, there may have been a tick after the HMD rotation was read, but it was moved.
It looks to me like HMD rotation is done as late in the pipeline as possible intentionally, to minimise lag in the HMD. Only reflections and the actual rendering seem to happen after.

So, I don’t think it would be accepable to move the hmd transform sooner, better to do work needed after, and as minimal amount of work possible.

Requesting it a second time should get different results, which would still mean the information I need is desynced (though perhaps much less so).

If I do #1 then #3 should then probably be done as an external module. (I will try out #1, and see if it even has a chance of working, may not be practical)