Download

How To: Manual Camera Control Setup with HMD & Motion Controllers

Hey guys, there is a really tricky problem to deal with in VR: How do you prevent the player from putting their head through a colliding object?

I came up with a two part solution and it goes against the Epic VR documentation, but I feel my setup is far superior. I’m going to walk you guys through how to replicate my success.

  1. I am using the 4.13 Source build of the engine. You MUST download the latest source and compile the engine for this to work because we’re going to be making a couple small modifications to the engine source in order for this to work. Before we get into that, let me explain what’s going on. Normally, you would go into your Camera Component and find a checkbox which says “Lock to HMD” and you’d make sure that is checked. This would make sure that any HMD **translations **and **rotations **are applied to the camera, and it also applies a rendering late update. The important thing to note is that in the standard documentation, this will cause the Camera Component Relative Location to change by the HMD location, but the scene component your camera is attached to is NOT CHANGED. Okay? So this means people can put their head through a wall. In order to prevent people from putting their head through a wall, we’d have to do a sweep test between the cameras current location and its previous location to figure out if there’s anything colliding. We don’t want to sweep along the camera component location delta, we want to sweep along the actor location delta.

Conceptually, I have a disembodied head which contains all of the VR specific implementation code. I call this my “VR Head”. It will always be centered on the players head, and if you call “Get Actor Location” on this head, you’ll get the players head in world space coordinates! This is super simplified, because now you don’t have to derive this location from the actor location + camera relative offset. Anyways, lets get to the engine source!

  1. Open up “CameraComponent.h” in \Engine\Source\Runtime\Engine\Classes\Camera\CameraComponent.h

  2. Look for the following code (line 59)



/** True if the camera's orientation and position should be locked to the HMD */
UPROPERTY(EditAnywhere, BlueprintReadWrite, Category = CameraSettings)
uint32 bLockToHmd : 1;


And then add the following two code blocks:



/** True if ONLY the camera's orientation should be locked to the HMD. Position must be manually set. */
UPROPERTY(EditAnywhere, BlueprintReadWrite, Category = CameraSettings)
uint32 bLockRotToHmd : 1;

/*This will use a late update to on the render thread. Use this for VR if you aren't locking the HMD.*/
UPROPERTY(EditAnywhere, BlueprintReadWrite, Category = CameraSettings)
uint32 bUseLateUpdate : 1;


  1. Next, open up “CameraComponent.cpp” in \Engine\Source\Runtime\Engine\Private\Camera\CameraComponent.cpp

  2. Look for the “GetCameraView(float DeltaTime, FMinimalViewInfo& DesiredView)” method (Line 216). We’re going to be making some small modifications to this. We’re going to add support for locking the HMD rotation and support for a late render update. Change the top portion of the method code to look like this:



if (bUseLateUpdate && GEngine->HMDDevice.IsValid() && GEngine->HMDDevice->IsHeadTrackingAllowed())
	{
		ResetRelativeTransform();
		const FTransform ParentWorld = GetComponentToWorld();
		GEngine->HMDDevice->SetupLateUpdate(ParentWorld, this);
	}

	if ((bLockToHmd || bLockRotToHmd) && GEngine->HMDDevice.IsValid() && GEngine->HMDDevice->IsHeadTrackingAllowed())
	{
		ResetRelativeTransform();
		const FTransform ParentWorld = GetComponentToWorld();
		GEngine->HMDDevice->SetupLateUpdate(ParentWorld, this);

		FQuat Orientation;
		FVector Position;

		if (GEngine->HMDDevice->UpdatePlayerCamera(Orientation, Position))
		{
			if (bLockToHmd)
			{
				SetRelativeTransform(FTransform(Orientation, Position));
			}
			else if (bLockRotToHmd)
			{
				SetRelativeTransform(FTransform(Orientation, FVector::ZeroVector));
			}
		}

	}


  1. Compile the engine and go get some coffee. It’s gonna be about ~30-45 minutes.
    What did we just do?
    We enabled ourselves to use a late rendering update, even if we’re manually setting the rotation and location of the camera. In the past, disabling “Lock to HMD” would also skip the late update, so if you manually modified the camera position / rotation, you’d get some very slight distortions due to the lack of this late update.
    We can also lock the camera to the HMD rotation, but manually change the HMD position. If we have this option enabled, we’re automatically using the late update, so it’s not necessary to enable/disable the late update checkbox because it’s going to run.

  2. Open up your game and create a brand new PAWN based blueprint. It is super important to use the pawn instead of the character! The root component of the pawn is just a scene component, whereas the character blueprint’s root component is a capsule component.

  3. We’re going to be adding in our components now! This is going to be a bit different from the Epic documentation, but I think it is a better setup.
    8a) Create a scene component, rename it to “Base” and set it as the root component.
    8b) Create two more scene components. Call them “Tripod” and “MC_Base”. They should be parented to the “Base” component.
    8c) Within the tripod scene component, add a camera component. Make sure that “Lock to HMD” is false, and “Lock Rot to HMD” is set to true.
    8d) Add two motion controller components and parent them to the “MC_Base” scene component. Make sure you set one of these components to the right motion controller.
    8e) optional: Add two static mesh components and parent each one to the motion controller components. If you’re using the Oculus Touch Controller meshes, make sure that you multiply the right controller mesh scale Y value by -1.

After you’re done, it should look like this:
dc8bd199840f1b417a8ff1bc117f61a6cdeb3831.png

  1. Okay, now it’s time to create our blueprint logic. This is deceptively simple, but it took me a lot of trial and error to come up with this.
    9a) Create a Vector variable called “HMDDelta”. This will be our change in HMD device position since the last frame!
    9b) Create a second vector variable called “LastHMDPos”. This will be the last position of the HMD device at the last frame. We can’t get a delta value without knowing where we were last frame.

Here’s what your Begin Play should look like:
f8545f37849025001c8975f39731dd2f9d9e1c41.png

In the first node, we’re setting our Last HMD Position to the current hardware position when the game started.
In the second node, we’re going to be offsetting our MC_Mount relative location by the distance between the HMD and its play area origin (PAO). If you skip this step, your motion controllers will be offset! We have to do this because we’re doing everything manually.

  1. Now, we need to do our “Tick” event. This is where we’ll be manually moving our actor by the change in HMD location over time. Let’s look at the blueprint first:
    cfa69233bd2acb88a3f4d471ac2d9fff4b01944c.png

10a) We’re calculating our HMD delta. This is the change in HMD position since the last frame. This is super important because we’re manually changing stuff!
10b) Now, instead of moving the Camera Component (as the documentation does), we’re changing the actor location by the HMD Delta amount!
10c) We can’t forget about changing the motion controller offset! We do this by changing the relative location by the change in the HMD delta. This will keep our hand positions correct even when players move their head around.
10d) Finally, we have to save our current HMD location into the “Last HMDPos” value so that we can repeat the process in the next frame update.

If it helps you to visualize what’s happening here, you can draw a bunch of debug spheres on each of the scene components. You’ll notice that that actor location moves instead of the camera component. If we attach a collision sphere to the actor, we can check “Sweep” on the “Set Actor Location” component in [10b]. Now, players can’t put their head through a virtual wall, yet move around in their play area using room scale VR :slight_smile:

Although its not covered in this how-to, you’re now ready to mount the head on top of a character body! Since we’re controlling the HMD position manually by setting the actor position, we can manually control where exactly the head is in world space by calling “Set Actor Location” on it. So, the head can drive a character body position, and the character body position can also drive the head position!

Also: I tested this on both the Vive motion controllers and Oculus motion controllers. It works perfectly with both.

Very cool work, looking forward to trying this out myself. This is of course one of those immediate loss of immersion conditions that is nearly universal in VR. If you’ve tried Onward (a brilliant game), the dev has done at least some things to try and deal with out of bounds players but it’s a pretty nasty kludge compared to what, hopefully, I think this implementation gives you. I’m really only starting up again with UE4 after a long absence, but it seems similar to mordentral’s work with physics-based objects in VR–do a sweep t make sure you don’t pass the object through another object?

Anyway, thanks for putting this up, will try it soon.

Does this solve the artificial locomotion problem? What I mean is, if I move the character artificially will the capsule component behave like it should if a character component was used (collides with both dynamic and static geometry). I have tried everything prior and found that adding a capsule to the camera and artificially moving the character you will not get collisions with static geometry.

Also, is it possible to use a character movement component with your system?

I am trying to do exactly what you describe above and also have full artificial locomotion, with full collision, that works regardless of where the player is standing in their play space.

Yes, this is a big part in solving the room scale locomotion problem. The problem is that players could walk through colliding geometry in VR because there is nothing in their play space blocking them from walking through it. If you’re just constantly setting the camera position to the HMD position (as most people currently do), the player can clip through any colliding geometry. So, I discovered that the best thing to do is to break the HMD controls from the character controls by creating two completely separate actors. The HMD is the “VR_Head”, which contains all VR hardware interactions. The head is then mounted onto the head position of a character and constantly set to the character head position. In order to move the character, I measure the HMD position delta and apply that delta as movement input into the character movement component. The character receives that movement input and moves exactly how far the player moved. So, the head isn’t moving, but it’s sending movement commands to the body, which creates the illusion of movement. Now, because the body is moving, we get all of the in-game collision effects, including being blocked by static geometry. When the player tries to physically walk through a virtual wall, the character is blocked and the camera is blocked as well.

You can also have a disembodied head. In this case, you can still have collisions. You’ll just have to put in a colliding component onto the VR head pawn and then do a sweep from the last head position to the current head position and if there are no collisions, you can update the head position by the HMD position delta. This would work well enough for spectators.

Will try this out. Sounds very promising.

Hi,

because Im going really mad about telling the virtuel camera what to do, I will try this out, because it does exactly what I didnt accomplish yet. I want a complete independant movement controll.The Problem always was, that I could not shut off the Movement Input from the HMD. So I just have one question. What do you mean about “… compile the Engine…” ?? You mean, I have to download the source code, and compile it via Visual Studio!!!

Thank you so much for this post.
Best

Yes, you have to download the engine source code and compile it. Here is the documentation on how to do that:
https://docs.unrealengine.com/latest/INT/Programming/Development/BuildingUnrealEngine/

Pro tip: Before you compile the engine, make the engine modifications I listed above. This will save you about 1 hour of time. Otherwise, you have to compile the engine, wait an hour, modify the engine, compile it again, wait another hour.

will this work with 4.18?

Yes, though the camera code for 4.18 has been slightly modified, so the engine source code modification is going to be slightly different from what’s posted above. I implemented the mod on my version of 4.18 and have no problems.

Thank you for this awesome post - i have been struggling with problem in VR for a while now.

I’ve tried to modify the source code as you have described in 4.20 - but i seem to get some errors regarding the lines that includes “GEngine->HMDDevice” as it says:

“Class UEngine” has no member “HMDDevice”

I am sure there is a simple way to modify it, i just haven’t seemed to find it yet. I was hoping you might have an idea what i might need to do?

The most recent versions have abstracted the concept of HMD to also include AR devices alongside VR ones. You should use GEngine->XRSystem.

I haven’t looked at the engine code in 4.20 yet, but if the engine still doesn’t support locking only the HMD rotation to the camera, the implementation principle is going to be roughly the same as I outlined above. I know Epic has refactored a lot of the backend implementation code to be more hardware agnostic (reflected in their “XRSystem” rename), so if you’re trying to mod the engine in 4.20, you’ll need to adapt to their refactoring. Here is the modified code I’m using in 4.18, which is probably not going to be too much different from 4.20:

C:\UE418\Engine\Source\Runtime\Engine\Classes\Camera\CameraComponent.h (Line 68)



    /** True if the camera's orientation and position should be locked to the HMD */
    UPROPERTY(EditAnywhere, BlueprintReadWrite, Category = CameraSettings)
    uint32 bLockToHmd : 1;

    /** True if ONLY the camera's orientation should be locked to the HMD. Position must be manually set. */
    UPROPERTY(EditAnywhere, BlueprintReadWrite, Category = CameraSettings)
        uint32 bLockRotToHmd : 1;

    /*This will use a late update to on the render thread. Use this for VR if you aren't locking the HMD.*/
    UPROPERTY(EditAnywhere, BlueprintReadWrite, Category = CameraSettings)
        uint32 bUseLateUpdate : 1;


C:\UE418\Engine\Source\Runtime\Engine\Private\Camera\CameraComponent.cpp (Line 237)



void UCameraComponent::GetCameraView(float DeltaTime, FMinimalViewInfo& DesiredView)
{
    /*if (bUseLateUpdate && GEngine->XRSystem.IsValid() && GEngine->XRSystem->IsHeadTrackingAllowed())
    {
        ResetRelativeTransform();
        const FTransform ParentWorld = GetComponentToWorld();

    }*/

    if ((bLockToHmd || bLockRotToHmd) && GEngine->XRSystem.IsValid() && GetWorld()->WorldType != EWorldType::Editor )
    {
        IXRTrackingSystem* XRSystem = GEngine->XRSystem.Get();

        auto XRCamera = XRSystem->GetXRCamera();
        if (XRSystem->IsHeadTrackingAllowed() && XRCamera.IsValid())
        {
            const FTransform ParentWorld = CalcNewComponentToWorld(FTransform());

            XRCamera->SetupLateUpdate(ParentWorld, this);

            FQuat Orientation;
            FVector Position;
            if (XRCamera->UpdatePlayerCamera(Orientation, Position))
            {
                if (bLockToHmd)
                {
                    SetRelativeTransform(FTransform(Orientation, Position));
                }
                else if (bLockRotToHmd)
                {
                    SetRelativeTransform(FTransform(Orientation, FVector::ZeroVector));
                }
            }
            else
            {
                ResetRelativeTransform();
            }

            XRCamera->OverrideFOV(this->FieldOfView);
        }
    }
...


Some day when I get github figured out, I’ll submit this as a pull request so that the change is integrated into future engine releases and doesn’t require mods. Alternatively, if anyone else wants to get the ‘engine contributor’ tag, you can submit it yourself and get credit for it. I don’t really care, as long as it gets in there.

It doesn’t require an engine change…

I have something kind of like this in my plugin, you just need to derive from the camera and override that function. Its niche enough that it likely wouldn’t make a good pull request, and is pretty easy to implement with a subclass.

hello, what’s the name of your plugin? can u share the link?

www.vreue4.com

but that specific override is in this source file where I back the camera to zero on X/Y again if requested.

https://bitbucket.org/mordentral/vrexpansionplugin/src/Master/VRExpansionPlugin/Source/VRExpansionPlugin/Private/ReplicatedVRCameraComponent.cpp