NVidia 3DVision Integration

Hi, very new to UE. hoping there’s someone with a good knowledge of the UE4 rendering system who can help us.

We are in the process of migrating to UE4 and need to add Nvidia’s 3DVision stereoscopic 3D support. There seems to be a good amount of stereo support but nothing complete yet for frame sequential stereo or Nvidia’s stereo api.

We know how to set it up in a generic DX c++ app but obviously have no idea where to put/access everything in Unreal.

This post https://forums.unrealengine.com/showthread.php?4950-VR-for-projection-environments&highlight=stereoscopic has some good starting tips but I wanted to see if anyone could get more specific to speed us along.

The steps are as follows, any suggestions most appreciated:

  1. Firstly, on start up and before the D3D renderwindow is created (if stereo is requested) we need to initialise the NVapi stereo system something like this:

NvAPI_Status nvStatus = NvAPI_Initialize();
if (NVAPI_OK == nvStatus)
nvStatus = NvAPI_Stereo_SetDriverMode(NVAPI_STEREO_DRIVER_MODE_DIRECT);		
if (NVAPI_OK == nvStatus)
{
	NvAPI_ShortString nvapiStatusMessage;
	NvAPI_GetErrorMessage(nvStatus, nvapiStatusMessage);
	mIsNvapiInitialized = true;
	NvAPI_Status nvStatus = NvAPI_Stereo_SetDriverMode(NVAPI_STEREO_DRIVER_MODE_DIRECT);
	nvStatus = NvAPI_Stereo_IsEnabled(&mStereoEnabled);		
}

So the question is where does this best go?

  1. Next we need to link the D3D render window to the stereo driver like:

NvAPI_Status nvStatus = NvAPI_Stereo_CreateHandleFromIUnknown(IDirect3DDevice9*, &nvapiStereoHandle);
if (logErrorMessage(nvStatus))
nvStatus = NvAPI_Stereo_IsActivated(nvapiStereoHandle, bIsStereoOn);

The trick here is we need to access the D3D render context which was created by Unreal… question, where and how?

  1. Now once stereo is activated on the designated window, we need to specify the left, right or centre eye position before rendering the view as:

nvStatus = NvAPI_Stereo_SetActiveEye(nvapiStereoHandle,_bufferTarget);

where _bufferTarget can be either: NVAPI_STEREO_EYE_MONO, NVAPI_STEREO_EYE_LEFT or NVAPI_STEREO_EYE_RIGHT

However, this requires the view to be rendered twice, either as two separate cameras or by moving the camera’s position and view frustum between each render pass

The linked post has a good start:

What I wasn’t completely sure of by the way this is described is if this ultimately does a full scene update (physics etc.) for each eye, rather than triggering two render passes?

So wasn’t 100% sure if

was what that refers to?

Mainly I could do with knowing if FFakeStereoRenderingDevice is everything I need to build on to get dual render contexts which I can channel independently to back left/ back right on the fly?

  1. Finally, we need to manually set the view frustum on each camera as it is asymmetrical and will be calculated manually based on a head-tracked position

I think that was fully answered by:

but I thought I’d still throw it in in case anyone what’s to elaborate.

Huge thanks for any guidance!

Ally

bump

Anyone?

Thanks,
Ally

Double Bump*

100% interested. We work with big industry in CAVE environments and I have been desperate to be able to get UE4 into one. As it stands Unity3D is the easy engine of choice within universities (we use bespoke atm) but I can easily see UE4 panning this if we could get sequential stereoscopic support.

There is a huge market out there…

Hi Ally,

I am trying a similar thing here and found your post. Have you found where is the best / cleaner place to do it? Any progress?

Thanks,
jalley

Hello,

This might be an old thread, but I haven’t found the exact information anywhere so I’ll post my solution here, in case anyone still needs it.

DISCLAIMER: It works, but it is not necessarily the best way to do it. It required making some changes that didn’t fit so nicely in the existing code.

So here are some answers to Ally’s questions:

  1. The initialization has to go before the 3D device creation. I just put the calls in the FD3D11DynamicRHI::InitD3DDevice function. (Don’t forget to check the status of those).

NvAPI_Initialize();
NvAPI_Stereo_SetDriverMode(NVAPI_STEREO_DRIVER_MODE_DIRECT);


  1. The device is accessible globally (you might need to include some additional headers, depending on where you put this):


auto D3D11RHI = static_cast<FD3D11DynamicRHI*>(GDynamicRHI);
ID3D11Device* D3DDevice = D3D11RHI->GetDevice();

StereoHandle deviceHandle;
NvAPI_Stereo_CreateHandleFromIUnknown(D3DDevice, &deviceHandle);
NvAPI_Stereo_Activate(deviceHandle);


  1. If you take *FFakeStereoRenderingDevice *as an example, you can see that it already creates two views, one for each eye, and displays both of them side by side. To select which eye sees which, you have to select the eye before its view gets rendered. There are a few places where you can do that, but it HAS to be in the rendering thread. The driver crashes if you do it in the game thread. I suggest putting it in one of the *Process *functions.


if (StereoPass == eSSP_RIGHT_EYE)
{
	NvAPI_Stereo_SetActiveEye(deviceHandle, NVAPI_STEREO_EYE_RIGHT);
}
else if (StereoPass == eSSP_LEFT_EYE)
{
	NvAPI_Stereo_SetActiveEye(deviceHandle, NVAPI_STEREO_EYE_LEFT);
}


The other problem is that, when I try giving both views the same screen position, so that they overlap rather than only take half of it each, they are written to the same buffer, so only the left eye sees anything (and it’s a mix of both images). To work around that I offset the right-eye view so that it doesn’t overlap, as if it would be printed outside the screen, then offset it back to the right position just before it is drawn, in FRCPassPostProcessTonemap::Process. If anyone has a better way of doing it, I would be happy to know it. (This might have some effects that I don’t really understand).

  1. The glasses don’t track the head’s position, so I don’t do anything special to the projection matrix, except translating it a bit along x for each eye.

I put most of the code in a new class that implements IStereoDevice. This way the device handle is kept in the same place and only has to be created once in the program. (And it is accessible everywhere). The stereo enable/disable is also handled by that class.

Hope this helps. And if anyone has any improvements to suggest (especially for question 3) it would be appreciated.

Cheers,
Vincent

Picking up 3DVision

Hi Vincent, just saw this reply by chance - thanks!

We had to drop this project to work on others but are picking it back up now. We still need full sequential 3D support so over the next couple of months will be putting together support for this, starting from your notes is a great help.

btw - setting the view frustum correctly for each eye is really important even without head tacking otherwise you will get uncomfortable distortions in the resulting ‘fused’ 3D image (i.e. from a pure ‘x’ translation, or the similar ‘toe-in’ technique - can point anyone towards some resources if interested, or just try Google), not necessarily obvious at first but can be uncomfortable to look at and the 3D ‘feel’ is generally better with the correct off-axis frustum(s) in place.

Cheers,
Ally

I started a plugin a while back to do this working with the devs at Uppercut Games, but I’ve been too busy with other work to finish it - this is the result of one weekend of mad hacking to get a proof of concept up and running:

https://github.com/DarkStarSword/UnrealEngine/tree/3DVisionWIP

If anyone wants commit access to that let me know (or just clone it and send pull requests).

It currently alternates between the left and right eye each frame - it needs a bit of work to render both views each frame (shouldn’t be difficult, I was just under a tight deadline to get a working POC).

It integrates with the separation and convergence settings from 3D Vision (so the hotkeys work as expected), and uses a projection matrix that has a mathematically equivalent result to using nvidia’s stereo formula (because it’s one of the best, and the shaderhackers in the community are all familiar with the formula and how to use it to fix broken effects).

It currently uses the FOV calculations for VR - obviously this is wrong since we are not using VR and not overriding the FOV should be easy.

A lot of the code in the plugin is likely unnecessary - much of it is copied from the VR plugins (but again, had to get it working before I could think about polish).

I’m the top shaderhacker in the 3D Vision community (20+ games personally fixed including several engines previously thought to be “unfixable”, plus assistance with many others), and let me tell you that it is very important that this be able to work with 3DMigoto so the community doesn’t have to throw away a mostly working 3D game just to fix a broken crosshair (which we will if there’s no choice). This should just be a matter of making sure the stereo parameters (a way to determine the current eye, separation, convergence, or whatever stereo projection the game is using) are available in every shader, even ones that may not seem to need it (like the HUD) so the community can fix them. Since I’ve integrated with the 3D Vision separation + convergence settings those will be available in the stereo texture that 3DMigoto injects in t125, but the current eye will not as that depends on 3D Vision Automatic, so we need to make sure that is available from UE4. 3DMigoto is open source and I’m one of the two active developers on the project, so it’s no trouble if we need to extend 3DMigoto for this as well.

Hi everyone,
I used DarkStarSwords plugin and it worked for UE 4.9 , but as trying to make it work for UE 4.14 there were some errors due to version incompatibilities. I got though all but there is one major issue due to which i think 3d vision is not working.
IStereoRendering no longer includes the method FinishRenderingFrame_RenderThread(class FRHICommandListImmediate& RHICmdList) , in which left right eye views are set.
So my question is where does this code go now?



	if (custom_present.eye == 1.0f) {
		if (custom_present.stereo_handle)
			NvAPI_Stereo_SetActiveEye(custom_present.stereo_handle, NVAPI_STEREO_EYE_LEFT);
		custom_present.eye = -1.0f;
	} else {
		if (custom_present.stereo_handle)
			NvAPI_Stereo_SetActiveEye(custom_present.stereo_handle, NVAPI_STEREO_EYE_RIGHT);
		custom_present.eye = 1.0f;
	}


as this code needs to run every frame and i couldn’t find any other suitable method for this to put in.
Full code from line 319

any other solution for enabling 3d Vision in UE 4.14 with necessary code samples or links would be appreciated.
Thanks
(new to UE)

Hi, please visit the “iLocalPlayer” UE4 C++ Plugin: iLocalPlayer4UE in Code Plugins - UE Marketplace;

Functions of this plugin :1. Side-by-Side Stereo 3D implementing; 2. Asymmetric projection(or named asymmetric view frustum) implementing.

Hope this helps you.

Latest version of my plugin is here, which is pretty much complete (except for maybe some minor polish, and one known bug with the mouse cursor position - I have a hacked up fix for that on my other box): https://github.com/DarkStarSword/UnrealEngine/tree/3DVision-4.14.3

Don’t use the old WIP branch - that was only ever a proof of concept, this branch is the final thing.