Thank you, buddy! This really helped!
Just want to share what worked for me: unchecking “Supports Activation Focus” in the widget’s class defaults Details pane. This allows me to control the Game Mode like normal on a per-widget basis, while still leaving “Enable Default Input Config” checked in the Common Input Settings, so it can still handle everything else.