It turns out that you have to delete YourProject/Saved/Config/WindowsEditor/Engine.ini in order for your project-level TemplateMapInfos to be recognized.
Try that if your template map isn’t showing up in the “New Level” window.
In UE5, there’s a new way to do this now.
In Project Settings->Project->Maps & Modes, there’s a
‘Editor Template Map Overrides’ where you can edit an array of possible map templates.
Comment from the config field:
/** Map templates that should show up in the new level dialog. This will completely override the default maps chosen by the default editor */
Edit that array to point to your map templates, save, restart editor, and you’re good to go.