Announcement

Collapse
No announcement yet.

Mixed Reality Status update

Collapse
X
  • Filter
  • Time
  • Show
Clear All
new posts

    #46
    Originally posted by tweaq View Post
    Here you go guys: https://github.com/richmondx/UE4-MixedReality
    I have it working (and i'm a n00b). You have to compile UE from the github source using those files. Then using OBS to to composite the webcam footage and the in game footages.
    I need help changing one of the 4 views. Right now it shows the greenscreened stuff, the background and 2 blank screens. I would like to display the HMD in one them.
    Again I got it work, and I'm hacking together blueprints and learning on the fly.
    Hot dang, I figured it out.
    Add this before the if(vrCameraRenderTarget) in the SteamVRRender.cpp:
    Code:
    		RHICmdList.ClearColorTexture(BackBuffer, FLinearColor::Black, FIntRect());
    			
    			RendererModule->DrawRectangle(
    				RHICmdList,
    				0, ViewportHeight / 2,
    				ViewportWidth / 2, ViewportHeight / 2,
    				0.0f, 0.3f,
    				0.4f, 0.4f,
    				FIntPoint(ViewportWidth, ViewportHeight),
    				FIntPoint(1, 1),
    				*VertexShader,
    				EDRF_Default);
    And comment out the other RHICmdList in the windowmode=3 section.

    That will give you the foreground and background as expected and the full widescreen HMD in the 3rd quadrant. Now you can use OBS to capture and overlay for mixed reality, and get a view from the POV of the person playing

    Comment


      #47
      Hey tweaq, thanks for your work!
      I am currently experimenting with your solution. But as far as i can tell, it only supports manually selecting objects to appear in the foreground, which is kind of torublesome.
      I rewrote the "ForwardPost"-Material. Now it decides for each pixel, based on the position relative to the headset and the virtual camera, if it's supposed to be foreground or background. This works pretty well for me and allows to only cull parts of an object if it intersects the seperation plane between forground and background. I took over your idea to allow objects to always appear in the foreground by using the costom depth pass.

      Here is the shader code for those who might be interested: Click image for larger version

Name:	MixedRealityShader.png
Views:	1
Size:	382.2 KB
ID:	1123921

      Best wishes,
      MeleeCampz

      Comment


        #48
        That is just GREAT. I was JUST thinking about this. I haven't spent a whole lot of time with the foreground/background objects and was wondering if it was object based. Thanks a lot! And MOST of the work goes to richmondx at github.

        *Edit: Just tried it out. I THINK i recreated it correctly (probably not). Mine is close, but the "green screen" doesn't rotate with the camera. I'm setting the HeadSetPosition in the BP_VRcatpure2D using the HMD position. But, dang fine work any how.
        Last edited by tweaq; 03-06-2017, 10:36 AM.

        Comment


          #49
          Hey guys, I'm also interested in getting Mixed Reality working in UE4, so I'll be diving into the stuff you guys posted.

          I'll keep you updated.

          Comment


            #50
            We are super interested in this guys, we're working on making mixed reality accessible to everyone and Unreal is the last step to add to our stack! We have a lot of developer asking how they can do MR with unreal and we're obviously super excited to point them to any good direction: liv.tv

            Comment


              #51
              In the README/Documentation it says to use UE 4.12.5. Does it also work with UE 4.15? Some of the latest additions to this threas are young enough to be using UE 4.15, hence the question.

              Comment


                #52
                I believe the richmondx just updated the git from someone else and didn't change the documentation. The version posted works in 4.14.3 for sure. Not sure about 4.15

                Comment


                  #53
                  Your Liv Cube looks great. I've been thing about something similar for a while. Kind of like the pop out green screens, but in cube form. I signed up. It would be perfect for setting up our NASA public outreach demos.

                  Comment


                    #54
                    Btw for anyone having problems, MeleeCampz helped me out. When setting the headset position variable use the actor location and NOT the headset location node. It's local position inside the play area

                    Comment


                      #55
                      Could you guys do it after all? Can you post videos? What are the limitations so far?

                      Comment


                        #56
                        Yes it does work. Here is a screen shot of the gameplayClick image for larger version

Name:	ue4_MR.jpg
Views:	1
Size:	210.6 KB
ID:	1126003

                        The "screen" is pushed forward a little, so it cuts the headstraps. So when you are looking away from the "camera" it cuts out the headset also, but leaves the straps.
                        We are hoping to composite the footage on a separate computer. Having the bp_capture2d in the scene takes a hit on performance. I added a shortcut to spawn/kill it when needed. The "green screen" in the top left doesn't show green all the time. As in, if you looked away from the headset it would show the same as the background (bottom right). Which isn't REALLY a problem, as there would be nothing to cut out anyway.

                        Comment


                          #57
                          Please! We need the tutorial too!

                          Cheers.

                          Comment


                            #58
                            Go here and download everything. Go here and download the 4.14 source. I haven't tried using 4.15, but I suspect it will work too. If you haven't already, you will have to sign up for github and go through epics signup stuff to get access. Unzip the unreal source to someplace. (could be the same install folder as other versions. I recommend renaming it to something like 4.14_MR). Unzip the mixed reality and copy the "Engine" folder to where you unziped unreal source. It should overwrite a few files. *Optional* Add this before the if(vrCameraRenderTarget) in the SteamVRRender.cpp
                            Code:
                            		RHICmdList.ClearColorTexture(BackBuffer, FLinearColor::Black, FIntRect());
                            			
                            			RendererModule->DrawRectangle(
                            				RHICmdList,
                            				0, ViewportHeight / 2,
                            				ViewportWidth / 2, ViewportHeight / 2,
                            				0.0f, 0.3f,
                            				0.4f, 0.4f,
                            				FIntPoint(ViewportWidth, ViewportHeight),
                            				FIntPoint(1, 1),
                            				*VertexShader,
                            				EDRF_Default);
                            I don't remember exactly what it does now, but I THINK it adds a HMD view to the quad view.

                            Anyway, from here you can read the readme for the unreal source for details. The gist is: install visual studio 2015, run Setup.bat in the root of the source folder. After that finishes, run GenerateProjectFiles.bat. Open Ue4.sln. Set it to development editor and win64, then Build project. After that finishes, there is something about running a filepath exe or something, I can't remember and its not important. But you should have a UE4Editor.exe in Engine>Binaries>Win64. Run that and open the sample project. From the sample you can see how stuff is setup and connected. You can migrate the files to your own project. I would also recommend recreating MeleeCampz ForwardPost material as it works MUCH better.
                            Last edited by tweaq; 04-06-2017, 10:06 AM.

                            Comment


                              #59
                              Originally posted by tweaq View Post
                              Yes it does work. Here is a screen shot of the gameplay[ATTACH=CONFIG]136019[/ATTACH]

                              The "screen" is pushed forward a little, so it cuts the headstraps. So when you are looking away from the "camera" it cuts out the headset also, but leaves the straps.
                              We are hoping to composite the footage on a separate computer. Having the bp_capture2d in the scene takes a hit on performance. I added a shortcut to spawn/kill it when needed. The "green screen" in the top left doesn't show green all the time. As in, if you looked away from the headset it would show the same as the background (bottom right). Which isn't REALLY a problem, as there would be nothing to cut out anyway.
                              Hey tweaq, i've got a question to this mixed reality addon. Just to get sure: The addon makes it possible to use 3 htc vive motion controller?
                              The 3th controller can be attached to a real world camera. After that you simulate a virtual camera attached to 3th controller and record the output of this virtual camera with OBS and the real world with the real world camera.
                              After that you combine the two records (after filtering green) and voila, you got the mixed reality thing. Right?

                              EDIT: When i try to compile the engine (4.14) with the included addon, some errors appear:

                              Error C1083 ... "E:\GitLab\UnrealEngine-4.14\Engine\Plugins\Runtime\Steam\SteamVR\Intermediate\Build\Win64\UE4Editor\Development\SteamVRController\Module.SteamVRController.cpp": No such file or directory UE4 E:\Mixed Reality\UE 4-14\UnrealEngine-4.14.3-release\Engine\Intermediate\ProjectFiles\c1xx

                              ...

                              Error Failed to produce item: E:\Mixed Reality\UE 4-14\UnrealEngine-4.14.3-release\Engine\Plugins\Runtime\Steam\SteamVR\Binaries\Win64\UE4Editor-SteamVR.pdb UE4 E:\Mixed Reality\UE 4-14\UnrealEngine-4.14.3-release\Engine\Intermediate\ProjectFiles\ERROR

                              Error MSB3075


                              How can i solve these?

                              Comment


                                #60
                                In a nutshell, yes. To use a 3rd motion controller you either have to connect it via usb or use a reflashed steam adapter. You will get a quad view like in my image. You screen capture the top left and bottom right and composite those in OBS with the webcam footage.

                                I'm not sure on that error, but did you run the Setup.bat and then the GenerateProjectFiles.bat? You also may need to install the c++ "stuff" for visual studio.

                                Comment

                                Working...
                                X