UMG HUD Widget Gamepad D-Pad or keyboard arrow keys navigation only

I am working on a HUD widget that will be open during normal game-play. Because it is always open but will need to be interact-able I need to split my controls. What I would like is the widget controls to be gamepad D-Pad or keyboard arrow keys only, the gamepad left stick and WASD stay on controlling character, and the mouse and gamepad right stick stay controlling camera.

I have tried setting input mode game and ui, but that creates 2 issues.

  1. The controls are doubling up. The D-Pad and arrow keys do work, but the WASD/Mouse/Gamepad Sticks interact with the character and the UI at the same time.
  2. When the focus is set to the widget (In order to interact with it with arrows/D-Pad) my other buttons don’t affect the character properly until i click with the mouse to refocus.

What is the best way to set this up?