Mixed Reality and Spectator View

So there have been a few other threads previously on here about Mixed Reality and Spectator camera modes for virtual reality but they have kind of died and not really gotten much of a response so I thought I would start a new 1.

With the SteamVR plugin for Unity at launch they are now able to do mixed reality video recording in real time on a greenscreen. This would be a great feature to include in the Unreal plugin, is there any indication from Epic if this will be looked at or should we implement our own functionality for this? I have been working on integrating it with the Kinect for background removal rather than a greenscreen too but some more native integration would be greatly appreciated.

The next part is on the VR spectator mode.

Unreal will bring up the screen mirror when you are running a program on the Vive which is great. It’s also an unwarped view which is exactly what you want to put up on the TV to see what other people are playing.

Is there any way to control this camera at all? With several Unity releases this capability is built into them. Job Simulator, Fantastic Contraption and Tilt Brush all have the ability to place a camera manually in the scene or have multiple different modes such as rotating the camera around. Obviously this stuff is fairly easy to build in with a camera but currently I am unaware of how to hijack the camera view to be a custom camera placed in the scene rather than the headset view without creating a multiplayer game and running the camera as the 2nd player.

Any thoughts or updates from Epic would be awesome.

Giving this 1 a bump as I think it’s a pretty important feature for demonstrating virtual reality

Going to bump again.

Surprised nobody else has been asking about this since currently the Unity games like Fantastic Contraption and Job Simulator are doing great for Mixed Reality videos and live streaming to twitch etc but no UE4 games have the capability, or just nobody has figured out how to do it yet.

+1 This would definitely be awesome to have!

I spent a considerable amount of time trying to make this work for my own projects. I switched priorities entirely until UE4 develops some method of 2nd camera control similar to Unity.f Hopefully we can see this soon!

I believe at the demos, they were using modified cameras which took advantage of lighthouse tracking, hence allowing the position data to be synchronized with the game. I don’t think it will be doable without that.

Here’s a step-by-step how they did it - sounds pretty easy and straight forward with that unity plugin - would be great to have the same for UE4! :slight_smile:

Yep the Steam VR plugin for Unity allows that camera control quite easily, that’s the bit that seems to be missing with Unreal

All they were doing in most of the other non Valve demo’s was use an extra controller attached to a camera, that gets you the moving camera. But the piece we are lacking is the secondary screen camera control and non hmd visible components such as an extra menu or UI for camera control

I am very interested in this as well.
I have been trying to make it work doing a multi player game where player 2 is the camera but it is alos proving quite difficult .

The multiplayer method I can do fine. Just setup a multiplayer game and 1st player in gets the VR pawn and the 2nd player just gets a camera pawn. Im trying to avoid doing this since it seems like unnecessary overhead.

I am trying to project a mixed reality video to a display outside of an event so that people can see what is going on inside.

Don’t want to be showing the player view because it’s too jerky and not great for giving people an idea of things.

Also an added benefit of this is that they can see a different angle as well, so if something is happening behind the player it gives them an interesting perspective on it.

What I was trying to do is have 2 players , each one with it own Vive and each own computer. player one is the player and player 2 is the camera man.
This doesnt seem to work straight away.
Is this what you have managed to make it work?
Could you point me towards how you did it?
Thanks

As in you want 2 players both in virtual reality?

That is just a regular multiplayer game. You will just want to assign a different pawn/ player controller to player 2. Nothing really different about that 1 compared to any other multiplayer game.

Yes but not exactly .
2 players where one is the regular player that sees everything through the HMD and the other player in the other computer with the other Vive is the camera man, so what sees is through a camera attached to one of the controllers, no to the HMD.
I know that on paper it sounds easy and maybe it is, but I cant make it work.

I’m in the same boat as Irigoyen. We have our project set up to run multiplayer with 2 players. However, the headset rotation dictates the rotation of both players.

Still don’t quite get what you mean. The second player has another vive controller to move the camera around but not the headset? So he looks at the controllerCam through the monitor rather than a headset?

That is essentially what I am trying to do but I want to be able to do it as a single player game rather than having to do multiplayer and replicate all the events etc.

For the multiplayer you just setup different pawns and/or player controllers and assign them to the different players.

Currently with Unreal it doesn’t even seem to be possible to use the vive controllers without using the headset so that complicates matters further with these ideas though.

Although I could be wrong. Would be great if somebody from Epic could chime in and explain this?

Yes, my logic behind that was that if the second player used a second computer there is no overhead in the Player 1 computer for rendering the " camera man " Player 2 Camera.

So I think we are trying to do the same thing but 2 different ways.

Ideally everything would happen in one computer but since I failed with that I was looking at alternatives and multi user seemed to be a good one.

I also hope someone from epic could give us some advice.

Well you can definitely do the second player stuff fine, it’s the using the controller to move the camera around that seems to be the tricky part at the moment.

It’s worth mentioning that you can play 2 characters simultaneously from the same UE4 on the same computer. Before you choose play in VR, hit the dropdown and select the number of players. Of course, this will still make the headset apply to both characters.

This is a key issue for us. The goal is to make mixed reality demos, as that is the only effective way to make VR trailers. What I need to be able to do is place a camera in the game world and render it to the standard screen while running the Vive. Using multiplayer with two instances running is very fiddly. This would also be handy for the social screen on PS VR.