There are a few released applications now with support for mixed reality with unreal. Mostly using different techniques. Some are in engine compositing, some are external and using OBS. The setup we (Chroma Coda, with The Music Room) used at CES and NAMM is using OBS to composite. I’ve got a system for loading standard unity style camera config files too. Biggest issue I haven’t worked out how to fix is getting the clear colour on the near and far clip to be green instead of black! The team from Grab Games have solved this though and their system is pretty snazzy. Our system at the moment also doesn’t allow for the camera view to be behind any foreground partial transparency, but that’s fine for us.
I think all we want Unreal was able to do mixed reality like Unity “out of the box”; I don’t know how to do the decomposition of the render in layers (background, foreground….), if I need a special branch of UE or a plugin and how to create the virtual camera that mimics the real moving camera; this is the reason we need some advice or a more deep tutorial. I don’t want to switch to Unity only for doing mixed reality, and Unreal has de power to do virtual sets; please Epic, include it in next release, and if someone can explain all the process (with a project if possible) it would be super useful. Lot of thanks to all the community!
I’ll try to work on it in a week or so, I’m a little bit caught up in other stuff right now.
Thanks DeeMel, I’ll be waiting patiently …
Here you go guys: https://github.com/richmondx/UE4-
I have it working (and i’m a n00b). You have to compile UE from the github source using those files. Then using OBS to to composite the webcam footage and the in game footages.
I need help changing one of the 4 views. Right now it shows the greenscreened stuff, the background and 2 blank screens. I would like to display the HMD in one them.
Again I got it work, and I’m hacking together blueprints and learning on the fly.
Hot dang, I figured it out.
Add this before the if(vrCameraRenderTarget) in the SteamVRRender.cpp:
RHICmdList.ClearColorTexture(BackBuffer, FLinearColor::Black, FIntRect());
RendererModule->DrawRectangle(
RHICmdList,
0, ViewportHeight / 2,
ViewportWidth / 2, ViewportHeight / 2,
0.0f, 0.3f,
0.4f, 0.4f,
FIntPoint(ViewportWidth, ViewportHeight),
FIntPoint(1, 1),
*VertexShader,
EDRF_Default);
And comment out the other RHICmdList in the windowmode=3 section.
That will give you the foreground and background as expected and the full widescreen HMD in the 3rd quadrant. Now you can use OBS to capture and overlay for mixed reality, and get a view from the POV of the person playing
Hey tweaq, thanks for your work!
I am currently experimenting with your solution. But as far as i can tell, it only supports manually selecting objects to appear in the foreground, which is kind of torublesome.
I rewrote the “ForwardPost”-Material. Now it decides for each pixel, based on the position relative to the headset and the virtual camera, if it’s supposed to be foreground or background. This works pretty well for me and allows to only cull parts of an object if it intersects the seperation plane between forground and background. I took over your idea to allow objects to always appear in the foreground by using the costom depth pass.
Here is the shader code for those who might be interested:
Best wishes,
That is just GREAT. I was JUST thinking about this. I haven’t spent a whole lot of time with the foreground/background objects and was wondering if it was object based. Thanks a lot! And MOST of the work goes to richmondx at github.
*Edit: Just tried it out. I THINK i recreated it correctly (probably not). Mine is close, but the “green screen” doesn’t rotate with the camera. I’m setting the HeadSetPosition in the BP_VRcatpure2D using the HMD position. But, dang fine work any how.
Hey guys, I’m also interested in getting Mixed Reality working in UE4, so I’ll be diving into the stuff you guys posted.
I’ll keep you updated.
We are super interested in this guys, we’re working on making mixed reality accessible to everyone and Unreal is the last step to add to our stack! We have a lot of developer asking how they can do MR with unreal and we’re obviously super excited to point them to any good direction: liv.tv
In the README/Documentation it says to use UE 4.12.5. Does it also work with UE 4.15? Some of the latest additions to this threas are young enough to be using UE 4.15, hence the question.
I believe the richmondx just updated the git from someone else and didn’t change the documentation. The version posted works in 4.14.3 for sure. Not sure about 4.15
Your Liv Cube looks great. I’ve been thing about something similar for a while. Kind of like the pop out green screens, but in cube form. I signed up. It would be perfect for setting up our NASA public outreach demos.
Btw for anyone having problems, helped me out. When setting the headset position variable use the actor location and NOT the headset location node. It’s local position inside the play area
Could you guys do it after all? Can you post videos? What are the limitations so far?
Yes it does work. Here is a screen shot of the gameplay
The “screen” is pushed forward a little, so it cuts the headstraps. So when you are looking away from the “camera” it cuts out the headset also, but leaves the straps.
We are hoping to composite the footage on a separate computer. Having the bp_capture2d in the scene takes a hit on performance. I added a shortcut to spawn/kill it when needed. The “green screen” in the top left doesn’t show green all the time. As in, if you looked away from the headset it would show the same as the background (bottom right). Which isn’t REALLY a problem, as there would be nothing to cut out anyway.
Please! We need the tutorial too!
Cheers.
Go here and download everything. Go here and download the 4.14 source. I haven’t tried using 4.15, but I suspect it will work too. If you haven’t already, you will have to sign up for github and go through epics signup stuff to get access. Unzip the unreal source to someplace. (could be the same install folder as other versions. I recommend renaming it to something like 4.14_MR). Unzip the mixed reality and copy the “Engine” folder to where you unziped unreal source. It should overwrite a few files. Optional Add this before the if(vrCameraRenderTarget) in the SteamVRRender.cpp
RHICmdList.ClearColorTexture(BackBuffer, FLinearColor::Black, FIntRect());
RendererModule->DrawRectangle(
RHICmdList,
0, ViewportHeight / 2,
ViewportWidth / 2, ViewportHeight / 2,
0.0f, 0.3f,
0.4f, 0.4f,
FIntPoint(ViewportWidth, ViewportHeight),
FIntPoint(1, 1),
*VertexShader,
EDRF_Default);
I don’t remember exactly what it does now, but I THINK it adds a HMD view to the quad view.
Anyway, from here you can read the readme for the unreal source for details. The gist is: install visual studio 2015, run Setup.bat in the root of the source folder. After that finishes, run GenerateProjectFiles.bat. Open Ue4.sln. Set it to development editor and win64, then Build project. After that finishes, there is something about running a filepath exe or something, I can’t remember and its not important. But you should have a UE4Editor.exe in Engine>Binaries>Win64. Run that and open the sample project. From the sample you can see how stuff is setup and connected. You can migrate the files to your own project. I would also recommend recreating ForwardPost material as it works MUCH better.
Hey tweaq, i’ve got a question to this mixed reality addon. Just to get sure: The addon makes it possible to use 3 htc vive motion controller?
The 3th controller can be attached to a real world camera. After that you simulate a virtual camera attached to 3th controller and record the output of this virtual camera with OBS and the real world with the real world camera.
After that you combine the two records (after filtering green) and voila, you got the mixed reality thing. Right?
EDIT: When i try to compile the engine (4.14) with the included addon, some errors appear:
Error C1083 … “E:\GitLab\UnrealEngine-4.14\Engine\Plugins\Runtime\Steam\SteamVR\Intermediate\Build\Win64\UE4Editor\Development\SteamVRController\Module.SteamVRController.cpp”: No such file or directory UE4 E:\Mixed Reality\UE 4-14\UnrealEngine-4.14.3-release\Engine\Intermediate\ProjectFiles\c1xx
…
Error Failed to produce item: E:\Mixed Reality\UE 4-14\UnrealEngine-4.14.3-release\Engine\Plugins\Runtime\Steam\SteamVR\Binaries\Win64\UE4Editor-SteamVR.pdb UE4 E:\Mixed Reality\UE 4-14\UnrealEngine-4.14.3-release\Engine\Intermediate\ProjectFiles\ERROR
Error MSB3075
How can i solve these?
In a nutshell, yes. To use a 3rd motion controller you either have to connect it via usb or use a reflashed steam adapter. You will get a quad view like in my image. You screen capture the top left and bottom right and composite those in OBS with the webcam footage.
I’m not sure on that error, but did you run the Setup.bat and then the GenerateProjectFiles.bat? You also may need to install the c++ “stuff” for visual studio.