Mixed Reality Status update

[emretanirgan]

Has anyone been able to get this running in 90 FPS? I was able to get everything running in 4.15.1, but the frame rate drops to 45 FPS, I think because of the SceneCapture component. I currently have a GTX 980 Ti, so I might try with a better GPU if anyone’s seeing 90 FPS. Or if anyone has suggestions for performance improvements using this, that’s also appreciated!

[anonymous_user_b7ed3889]

@mebalzer;700845

Sadly, even with your instructions it will not compile correctly. It is frustrating that you don’t find this out until the you get to the section for plug-ins, which is about 45 minutes in, but I still get the same error, which is the includes are not in the right order:

include “SteamVRPrivatePCH.h”
include “Classes/SteamVRFunctionLibrary.h”
include “SteamVRHMD.h”

Did you click “Rebuild” for compiling and are you sure that you copy all “// MIXED-REALITY” marked codelines? Some files have more than one modification.
Also try to put:

#include "Engine/TextureRenderTarget2D.h"

into the SteamVRRender.cpp file after your includes, and if that does not help, set the includes “in the right order”.

Good luck.

[anonymous_user_b7ed3889]

@emretanirgan;702284

Has anyone been able to get this running in 90 FPS? I was able to get everything running in 4.15.1, but the frame rate drops to 45 FPS, I think because of the SceneCapture component. I currently have a GTX 980 Ti, so I might try with a better GPU if anyone’s seeing 90 FPS. Or if anyone has suggestions for performance improvements using this, that’s also appreciated!

Without running OBS this works fine for me, except my problem here:
I think that also dependent from your project. To avoid the OBS performance drop, use a hdmi-capture-card.

[emretanirgan]

@MMS_Tester;702800

Without running OBS this works fine for me, except my problem here:
I think that also dependent from your project. To avoid the OBS performance drop, use a hdmi-capture-card.

What are your PC specs? I’m not even running OBS, and just running the basic mixed reality project that’s on the Github repo. Are you just doing VR Preview in the Editor, or running a packaged build in VR? Thanks for the help by the way! Will definitely share some results if I can get everything running smoothly.

[anonymous_user_b7ed3889]

@emretanirgan;702839

What are your PC specs? I’m not even running OBS, and just running the basic mixed reality project that’s on the Github repo. Are you just doing VR Preview in the Editor, or running a packaged build in VR? Thanks for the help by the way! Will definitely share some results if I can get everything running smoothly.

Using Win7 with an GTX 1070 and i7-6700 processor. So nothing high end. Its running in editor.

No problem.

Cheers

[anonymous_user_b7ed3889]

@MMS_Tester;699851

Do anyone have a idea why i got everything running in the sample project, but after i put everything in my project, the two rendered MR views are both the same?
I mean normaly one view (upper left) is green with the “Render CustomDepth Pass = true” marked objects included, but for me i see two times the same view with all actors (normaly only bottom right).

If someone have the same problem, i solved it now: After i set the Texture Target for the SceneCaptureComponent2D component to RT_VRCapture2 (for the CaptureComponent2D still RT_VRCapture) and the Capture Source to Final Color (for the CaptureComponent2D too) it works for me now.
Also set the Post Process Material to ForwardPost (like in the Sample).
A different setup than the Sample, but hey, now it works greatly.

[smbv1]

Is there any news about 4.16 Mixed reality option? I want to record myself on DSLR camera and attach ViveTracker for VirtualCamera and record me from both Real Life Action and VR action then do merge these realities together. Is it possible yet? In Trello there is already option which should be completed by this time already.

[anonymous_user_10ad849b]

Does anyone know a good way to calibrate the camera offset? Also where do I plug those in? Into the BP_VRCapture2D or it’s MotionController child?

[MeleeCampz]

Hey guys. Unfortunately I have to stop working on my Mixed Reality solution ;(
However for those still tinkering with the stuff posted in here, you may look at this plugin : GitHub - chaosgrid/ExtraCamWindow: A simple Unreal Engine 4 plugin that allows to have extra game wind
It allows to render a camera to a second game window (works in 4.15). If you could combine this with the source for the SteamVR-Plugin mod as well as my shader you could create s pretty sick plugin solution that just works out of the box

[smbv1]

This is what is scary, when you starting to develop project hoping everything going to be okay and after that no support, I hope Epic will implement Mixed Reality soon.

[fabito_rychter]

I am on the same boat… Can´t believe I will have to switch to Unity after all…

[Juangea]

Maybe some UE developer can chime in and throw some light over all this, mixed reality is a pretty big deal nowadays with VR.

Cheers!

[aussieburger]

It’s still on Epics new “future roadmap”: Trello
but no approx ETA given

[anonymous_user_914c992f]

@MMS_Tester;699840

I got it running in 4.15 after i copy all the code for my own (marked in code with //Mixed Reality) into my running engine code. Before this i got running an early implementation from here: GitHub - haowang1013/UE4-MixeReality: Some stuff for doing mixed reality recording in UE4
Its allmost the same code like this from the post here: https://github.com/richmondx/UE4- only that richmondx add some modification to get some forground calculations.

You only(!) need to copy the code from SteamVRFunctionLibrary.h (…\Engine\Plugins\Runtime\Steam\SteamVR\Source\SteamVR\Classes)
and SteamVRFunctionLibrary.cpp, SteamVRHMD.h and SteamVRRender.cpp (…\Engine\Plugins\Runtime\Steam\SteamVR\Source\SteamVR\Private)
and recompile everything completely new(!).

If you got some problems with this line (you will get problems):

RHICmdList.Clear(true, FLinearColor::Black, false, 0, false, 0, FIntRect());

you have to replace it with this line:

RHICmdList.ClearColorTexture(BackBuffer, FLinearColor::Black, FIntRect());

After compiling and seting up you scene with the blueprints (like BP_VRCapture2D)(Or first try the downloaded MR Sample), you have to set this:

[SteamVR.Settings]
WindowMirrorMode=3

in the Engine.ini file of your project.

Some optional stuff:
With the code from tweaq, you put it after “else if (WindowMirrorMode == 3){:” in the SteamVRRender.cpp,

RHICmdList.ClearColorTexture(BackBuffer, FLinearColor::Black, FIntRect());
			
			RendererModule->DrawRectangle(
				RHICmdList,
				0, ViewportHeight / 2,
				ViewportWidth / 2, ViewportHeight / 2,
				0.0f, 0.3f,
				0.4f, 0.4f,
				FIntPoint(ViewportWidth, ViewportHeight),
				FIntPoint(1, 1),
				*VertexShader,
				EDRF_Default);

you can also add the view from the vr headmount.
Also you need to remove RHICmdList.ClearColorTexture(BackBuffer, FLinearColor::Black, FIntRect()); in a later codeline.

And with the blueprint action from you got some other fancy stuff.

cheers

I was able to get everything running in 4.15.1. Thanks to, richmondx, , tweaq, and everyone else who posted here.

I built another small application that allows you to use the Kinect v2 to filter out the background so you don’t need a green screen It’s sort of limited to the accuracy of the Kinect but it’s still cool and helpful because its portable.
I’d be happy to share that if anyone is interested.

The issue I’m running into with the quad view method is that the resolution seems to be capped very low. Maximizing the window doesn’t increase the resolution either. I tried playing around with stuff like increaing r.screenPercentage and r.SetRes but it doesn’t work. Any help would be greatly appreciated.

[fabito_rychter]

Cool! Could you post a video?

I heard that you need to output to a 4k monitor for better resolution.

At this site: How To Live Stream Mixed Reality
I found this:

“It’s important you have a machine that is powerful enough to run VR content at 2K resolution and at a constant frame rate. The reason for 2K resolution is that due to the way mixed reality content is captured, the output resolution will be halved. With a standard 1080p monitor, the final output resolution would be 960 x 540, which is quite small. With 2K resolution (2560×1440), the final output resolution is 1280 x 720. If you want a 1080p stream, then you will need a 4K monitor.”

[anonymous_user_914c992f]

Cool! Could you post a video?

I heard that you need to output to a 4k monitor for better resolution.

At this site: How To Live Stream Mixed Reality
I found this:

“It’s important you have a machine that is powerful enough to run VR content at 2K resolution and at a constant frame rate. The reason for 2K resolution is that due to the way mixed reality content is captured, the output resolution will be halved. With a standard 1080p monitor, the final output resolution would be 960 x 540, which is quite small. With 2K resolution (2560×1440), the final output resolution is 1280 x 720. If you want a 1080p stream, then you will need a 4K monitor.”

Thanks for sharing that. I’ll post a video soon. I do have a 4k monitor. Plus 2 Titan X’s in SLI. Not quite sure what to do about the issue with the resolution

[anonymous_user_914c992f]

Hey guys. Unfortunately I have to stop working on my Mixed Reality solution ;(
However for those still tinkering with the stuff posted in here, you may look at this plugin : GitHub - chaosgrid/ExtraCamWindow: A simple Unreal Engine 4 plugin that allows to have extra game windows with their own camera viewpoints.
It allows to render a camera to a second game window (works in 4.15). If you could combine this with the source for the SteamVR-Plugin mod as well as my shader you could create s pretty sick plugin solution that just works out of the box

The plugin was very helpful we were able to go up to well beyond even HD with it

[tweaq]

Just wanted to post what we did at a local makerfaire a few weeks ago: ACL | Flickr
As for performance, we have not found a way to have it running with MR at 90fps. We’re running an i7, win10, with a 1080 running at 1440p. We even tried using a hdmi capture and capturing on a separate computer. As soon as the MR camera is active/rendering it drops the frame rate. Our “hack” was to have a shortcut to kill or create the camera. So you can at least run 90fps until you want mixed reality.

[anonymous_user_914c992f]

Cool! Could you post a video?

I heard that you need to output to a 4k monitor for better resolution.

At this site: How To Live Stream Mixed Reality
I found this:

“It’s important you have a machine that is powerful enough to run VR content at 2K resolution and at a constant frame rate. The reason for 2K resolution is that due to the way mixed reality content is captured, the output resolution will be halved. With a standard 1080p monitor, the final output resolution would be 960 x 540, which is quite small. With 2K resolution (2560×1440), the final output resolution is 1280 x 720. If you want a 1080p stream, then you will need a 4K monitor.”

Video link is below: Within Prana on Vimeo

Performance still seems to be a bit of an issue but there should be a solution in place for that pretty soon.

[smbv1]

Hello everyone, I’m looking for any solution to make my VR demo into beautiful video tour. I was waiting for UE mixed reality in April, but they have removed trello board with this announce. Who have working tutorial for 4.16 to make this working? There is bits informations everywhere how to do it, but they all use some external techniques. Are they still actual? Who was able to solve real DSLR camera and Virtual Camera? I can do it offline, I mean, I can shot video on my cell-phone and at the same time I want to be able to rotate camera around me? Maybe I should use Vive-Tracker as a Vrtirual-Camera assistant? Who have clear information about this? Let’s do it at least offline, so we can all make VR demos from third view!