As the name says, FInputModeUIOnly is meant for UI input. Input components and their associated PlayerInput manager are not considered UI input but game input. With that in mind, there are many ways around this.
If you want to handle closing the UI through an input component binding, you can use FInputModeGameAndUI and flip a flag that disables the inputs you don’t want while the UI is active.
You can also use different input components for game-only input and mixed input, queue them both by overriding APlayerController::BuildInputStack, and omit the game-only input while the UI is displayed.
You could also have the UI handle game-related inputs while the UI is active, so they don’t make it to PlayerInput. Unhandled UI key events are passed (bubbled) up the widget hierarchy, all the way to the viewport which will then hand the event over to PlayerInput if Game input is enabled. Handling the event in UI will prevent that from happening, but it is a clumsy way of doing it for fairly obvious reasons.
You can also use FInputModeUIOnly and handle everything through the UI. One of your UI widgets would have to handle OnKeyDown events and ask the player controller to dismiss the UI once the close UI key is pressed.
If you’re feeling really fancy, you could even create a UI input component and manually handle it in UI input.
Just dig around the input system a bit, it’s all there.