VR motion controllers don't update after SetBasePosition is used to offset the origin

I’ve been dealing with some differences between Steam VR and Oculus. To start off my Pawn is a sphere collision, with a scene node, and a camera attached to that. They are all at FVector (0,0,0) relatively and in world space. When starting in Steam VR my camera would get moved up by the distance of the real world floor and the HMD. I had tried using

GEngine->XRSystem->SetTrackingOrigin(EHMDTrackingOrigin::Floor); and GEngine->XRSystem->SetTrackingOrigin(EHMDTrackingOrigin::Eye);

but this doesn’t stop it from happening. So I decided to try manually offsetting the base use this block of code

//This section is very fragile.  The base location of the origin is an inverse offset.  So we first set it to the inverse of the sphere collision
        StartLoc = PlayerRef->GetActorLocation();
        GEngine->XRSystem->SetBasePosition(FVector( -StartLoc.X, -StartLoc.Y, -StartLoc.Z));
        //Then we set it to the inverse of the vector from the  sphere collision to the camera which Steam VR forced upwards and out
        StartLoc = PlayerRef->View->GetComponentLocation() - PlayerRef->Sphere_Collision->GetComponentLocation();
        GEngine->XRSystem->SetBasePosition(FVector( -StartLoc.X, -StartLoc.Y, StartLoc.Z));

Along the way I learned that setbaseposition is an inverse offset. Once I learned this I was able to get the three components of my pawn all at zero both in world space and in relative space. From there the camera tracks the movements of the HMD of course and applies it as expected.

The problem now is that after this one time initial update my touch controllers did not follow. They are supposed to be attached to the scene node between the sphere collision and the camera. However they are now behind me and above me by a body’s length. The distance is pretty obviously the exact distance that the SetBasePosition was just moved. To make things more confusing they appear to have accumulated a large relative offset. So while the Sphere Collision, Scene Node, and Camera are all FVector (0,0,0) in relative space and world, the touch controllers have inherited an offset relative to the previous pre Set Base Position location.

Obviously I’ve tried just setting those relative transform back to zero but this does nothing. I can see change in the position by turning on the “DisbaleLowLatencyUpdate” but it still doesn’t seem to fix the problem and I’d rather figure out why this components are hanging back. Has anyone else experienced this?