I figured out my particular issue and it’s highly uncommon, but incase someone is in the same boat:
At some point, I added C:\Users\<user>\AppData\Roaming\Unreal Engine\UnrealBuildTool\BuildConfiguration.xml and hardcoded the compiler to VisualStudio2017. I did this because of some silly configuration I was figuring out with developing code plugins.
Deleting the file worked, and probably is the correct path forward when developing on both UE4 and UE5 targets.