Solved it!
I previously tried to set only the canvases of the menus to EVisibility::SelfHitTestInvisible, but that didn’t do it. Setting the whole menu instead of canvas worked like a charm. Interestingly it’s buttons still work if menu is set to EVisibility::SelfHitTestInvisible, so visibility seems not to be propagated through hierarchy.
Thank you!