This is kinda obscure, but in case anyone is interested. Turning off the following boolean gives the required behavior.
Edit -> Editor Preferences -> Miscellaneous -> Hot Reload -> Automatically Compile Newly Added C++ Classes
I guess Epic Games thinks of everything huh. 