In previous versions of windows you would right click on your unreal project and you could open said project with a specific version of Unreal. That is no longer the case.
In order to do this you know have to click “Show More Options” at the bottom of the drop down. Now you will be presented with the option to switch unreal versions.
I got it to work. The bit of information that was missing in the original post is that you have to select the project in the Windows Explorer, and then use “Show More Options” in the context menu there.
I was looking for it in the Library section of the Epic Games Launcher…