I have the same problem, and I found out this is caused by having RenderDoc installed. Uninstalling RenderDoc will make the overlay go away, but of course you then also loose RenderDoc.
Though there is documentation about using RenderDoc with Unreal 5 here: Using RenderDoc with Unreal Engine | Unreal Engine 5.0 Documentation
I assume Unreal somehow enables RenderDoc by default through its in-application api: In-application API — RenderDoc documentation
In the Unreal 5 documentation it mentions a RenderDoc plugin, which is not enabled by default, but I just found out that you can disable the overlay by enabling the plugin, instead of uninstalling RenderDoc. You get a RenderDoc button in Unreal Edlitor too then!