Rendering UI for Desktop and VR

I am running a multiplayer game for both VR and desktop.
I have decided to not create a client and server setup and run both as parallel playercontrollers on the same instance for multiple reasons that stem from the project specifications.

I have managed to have the VRPawn and a third player character working together as a POC,
The VRPawn is rendering the camera into the viewport, and the desktop player has an overlay widget UI with a render target rendering from the third person player.

The Issue:
The UI widget appears on the left eye of the VR headset.

Is there a way to not render UI into the VR headset?
Or rather the have a player controller specific UI?
Note that the solution must also work for a packaged project.

Here are some of my blueprints: