I see the problem. The issue is that Control Rotation is basically ignored when you are in VR mode and being replaced by the HMD one (+ delta). I am trying to come up with a simple solution that wouldn’t require any changes in existing interfaces (meaning, can be integrated into upcoming 4.11 (and backport into 4.10, hopefully); otherwise you’ll have to wait till 4.12, which is pretty long time to wait).
I’ll let you know once I have anything usable.