As the title asked, how does the Unreal Engine get away with loading the Core module DLL at runtime but also compiling against classes like FString without having Core around at compile time (because it’s dynamically loaded)? The Core could be compiled both static and dynamic, but linking to as static would defeat the whole purpose of loading the Core module at runtime.
FString is part of the Core module.
It’s under /Runtime/Core/Public/Containers/UnrealString.h
But that doesn’t answer your question. 
How other modules manage to compile while using the core module, that’s what the UnrealBuildTool is there for and why Build.cs and Target files are required for each modules.
I’m not the best at explaining so I’ll just point you to this article: Understanding Unreal Build Tool – Eric Lemes' Blog
In short, the UnrealBuildTool is the first thing compiled by the MSBuild and from there the Unreal Engine uses its own tools for compiling.