So I did some digging and figured it out. When you pass a name into EditorLevelUtils::CreateNewLevel, it’ll pass that through to FEditorFileUtils::SaveLevel. If that function can’t deduce the package name, it’ll fall back to the supplied name and presume that it’s been saved before. This alternative code-path, for some reason, doesn’t actually save it properly. If you call FEditorFileUtils::SaveLevel directly, once you get the level back from EditorLevelUtils, you can provide no default name and it will instead be able to deduce the package name, and will thus treat it as if it’s actually new and in need of proper saving.
UWorld* editorWorld = m_editorEngine->GetEditorWorldContext().World();
FString preppedName = L"/Game/Levels/" + a_levelName;
ULevel* level = EditorLevelUtils::CreateNewLevel(editorWorld, false, ULevelStreamingKismet::StaticClass(), preppedName);
/* manually save after creation once the level has a name, so we don't have to specify one, and thus we
still get the proper file saving path (as if the user was prompted). */