Preloading objects using subsystems

Hello!

I have a general question if there is a design for preloading assets that the game wants to use at a later stage.

What we have done is that we have Subsystems that can preload assets which will be used a lot during runtime. (e.g. Items, NPC Data, etc).

The issue with having the subsystems load the asset is that when you start PIE, there are a lot of FlushAsyncLoading that happens. This creates big stalls if lots of objects have been requested to load.

If we request loading during UWorldSubsystem::Initialize, they will get flushed when the GameMode is loaded.

  • UGameInstance::StartPlayInEditorGameInstance -> UWorld::SetGameMode -> UGameInstance::CreateGameModeForURL

If we request loading during UWorldSubsystem::OnWorldBeginPlay, they will get flushed when LoadMap happens during TickWorldTravel.

  • UEditorEngine::Tick -> UEngine::TickWorldTravel -> UEngine::LoadMap

Are there any other suggestions to preloading multiple objects during the start of a session, or is the intended way to only load objects on demand?

Best regards,

Oliver

Hello!

The approach we take in Fortnite and Lyra is specific to multiplayer, but it’s essentially “a little bit after BeginPlay”. I’ll reference Lyra code:

  • In Lyra, we have a concept of “experiences (youtube link)” which is basically a runtime decided game mode, with some Game Feature Plugins to load.
  • Dedicated servers load a map already, without knowing the experience data asset to load, which may be decided later by the matchmaking server. The dedicated servers load a map already, to “warm up” the server and minimize the wait time from a matchmaking match, to players joining. When match info becomes known to the server, ALyraGameMode::OnMatchAssignmentGiven runs on the server.
  • This sets the experience data asset on a GameState component: ULyraExperienceManagerComponent::SetCurrentExperience.
    • Server-side, this starts async preloading assets because now, some Game Features Plugins (GFP) will activate. Which GFPs to activate is defined by the (already loaded) experience data asset.
    • Client-side, loading begins once a reference to the correct experience data asset is replicated down. See ULyraExperienceManagerComponent::OnRep_CurrentExperience.

In a single0player setting or simpler multiplayer setting without prewarmed servers, you could say we preload at BeginPlay. I’m puzzled that when you request async loads in UWorldSubsystem::OnWorldBeginPlay, they get flushed. Can you provide a callstack of that FlushAsyncLoading happening? An Unreal Insights capture that shows the FlushAsyncLoading is fine too. I’d like to rule out that some non-engine code is triggering the FlushAsyncLoading.

Another thought: what method of map travel are you using? Is it perhaps the WorldSubsystem for the transition level (empty level) that’s requesting the async load, and those requests get flushed when loading the destination level?

Hey, thanks for the reply!

I should have clarified that we do have a multiplayer environment with a dedicated server that is pre-warmed. But our main issue comes from the dedicated server loading in PIE. (The issue is on pre-warmed servers too, but not as noticeable).

Here is the callstack for flushing after UWorldSubsystem::OnWorldBeginPlay. We’re doing NetMode -> Play As Client.

This callstack happens when the clients PieContext is ticking.

FlushAsyncLoading(TArrayView<…>) AsyncPackageLoader.cpp:314
FlushAsyncLoading(int) AsyncPackageLoader.cpp:296
UEngine::LoadMap(FWorldContext &, FURL, UPendingNetGame *, FString &) UnrealEngine.cpp:15664
UEngine::TickWorldTravel(FWorldContext &, float) UnrealEngine.cpp:15536
UEditorEngine::Tick(float, bool) EditorEngine.cpp:2115
UUnrealEdEngine::Tick(float, bool) UnrealEdEngine.cpp:547
FEngineLoop::Tick() LaunchEngineLoop.cpp:5905
[Inlined] EngineTick() Launch.cpp:69
GuardedMain(const wchar_t *) Launch.cpp:195
LaunchWindowsStartup(HINSTANCE__ *, HINSTANCE__ *, char *, int, const wchar_t *) LaunchWindows.cpp:266
WinMain(HINSTANCE__ *, HINSTANCE__ *, char *, int) LaunchWindows.cpp:317

“But our main issue comes from the dedicated server loading in PIE.”

Aha, that clarifies my confusion. I was confused about why LoadMap is happening after OnWorldBeginPlay, but in PIE with server and client under a single process, I think the following is happening:

  • The server duplicates the world for PIE (no LoadMap needed).
  • The server calls OnWorldBeginPlay. You request some async loads.
  • The client’s PIE context is created. LoadMap is called for the client, which triggers FlushAsyncLoading. When running under a single process, this also flushes your server’s async load requests.
  • The client calls OnWorldBeginPlay.

The problem happens when the server and client run under a single process, because the client’s world life cycle is interfering with the server’s.

If you want to work around this, you can consider letting the UWorldSubsystems defer the async load requests until all PIE clients have executed their LoadMap (and the FlushAsyncLoading() calls from those). It would look a bit like this:

#include "UObject/UObjectGlobals.h"
 
void UMyWorldSubsystem::OnWorldBeginPlay(UWorld& World)
{
	Super::OnWorldBeginPlay(World);
 
#if !WITH_EDITOR
	// Not an editor build, load immediately
	AsyncLoadAssets();
#else
	if (World.WorldType == EWorldType::PIE)
	{
		// Setup PostLoadMapWithWorld triggering of async loads
		FCoreUObjectDelegates::PostLoadMapWithWorld.AddUObject(this, &UMyWorldSubsystem::PostLoadMapWithWorld);
	}
	else
	{
		// Not a PIE world, load immediately
		AsyncLoadAssets();
	}
#endif
}
 
#if WITH_EDITOR
void UMyWorldSubsystem::OnWorldEndPlay(UWorld& World)
{
	Super::OnWorldEndPlay(World);
 
	// Clean PostLoadMapWithWorld triggering of async loads
	if (World.WorldType == EWorldType::PIE)
	{
		FCoreUObjectDelegates::PostLoadMapWithWorld.RemoveAll(this);
	}
}
 
void UMyWorldSubsystem::PostLoadMapWithWorld(UWorld* LoadedWorld)
{
	if (/*Check if LoadedWorld is the last PIE client*/)
	{
		AsyncLoadAssets();
	}
}
#endif

This is just a barebones code sample, needs to be thoroughly tested for all configurations (PIE settings, and build types), but hopefully you get the idea and find this a useful direction to investigate more in. Two changes that are still needed:

1) You need to handle the single client standalone starting PIE case since that skips PostLoadMapWithWorld. Note that checking world NetMode == NM_Standalone isn’t enough, since you can start PIE with multiple clients standalone.

2) You have to check whether LoadedWorld is the latest PIE client, since when starting PIE with 2 clients, PostLoadMapWithWorld will be called for each client, in this order:

  • Server calls OnWorldBeginPlay
  • Client1 calls LoadMap -> FlushAsyncLoading, OnWorldBeginPlay, PostLoadMapWithWorld
  • Client2 calls LoadMap -> FlushAsyncLoading, OnWorldBeginPlay, PostLoadMapWithWorld

You only want everyone to react on the last PostLoadMapWithWorld.

In both cases, what can help is getting GetDefault<ULevelEditorPlaySettings>() to get your PIE settings. It gets a bit messy in runtime modules, because those play settings are in an editor module. Be sure to:

  • Encapsulate code in #if WITH_EDITOR
  • Add to your Build.cs:
if (Target.bBuildEditor)
{
	PublicDependencyModuleNames.Add("UnrealEd");
}

I’m thinking of using something like World->GetOutermost()->GetPIEInstanceID(), which is, for example, 3 if your PIE settings are server + 3 clients under a single process. (server = 0, player 3 = 3).

It’s a big write-up for an incomplete solution, but hopefully it makes sense. This is surprisingly difficult, too. If “hackier” solutions are fine, then a one-frame delay on PIE might be a much simpler solution. :slight_smile:

Thanks for the rundown!

The flow that you explained is what I can see in editor as well, which is unfortunate :sweat_smile:

We’ve got plenty of different systems loading, so I think we’ll need to experiment and see what will work best. We have got a custom StreamableManager so I think it could be wise to queue up async loading requests there until we know that it is safe to perform async loading. In the end out biggest flaw is that we’re loading way too much.

Best regards,

Oliver

“We have got a custom StreamableManager so I think it could be wise to queue up async loading requests there until we know that it is safe to perform async loading.”

Sounds great to me. That would be a nice way to keep the complex deferred async loading logic in one place, instead of in multiple systems!

“The flow that you explained is what I can see in editor as well, which is unfortunate :sweat_smile:

It is indeed unfortunate. In my testing, I noticed that UEditorEngine::OnAllPIEInstancesStarted() also fires prematurely, which otherwise would have been a great place to put this logic. I’ll ask what the team thinks about fixing that, or adding something like “OnAllPIEInstancesMapLoaded()” that runs after all the PIE-startup triggered FlushAsyncLoading() calls.

“I’ll ask what the team thinks about fixing that, or adding something like “OnAllPIEInstancesMapLoaded()” that runs after all the PIE-startup triggered FlushAsyncLoading() calls.”

It would be excellent if there was something easy to hook up to!

Good to know, I’ll give you an update after our internal discussion!

Hey again, after internal discussions here are the conclusions:

People were averse to introducing more PIE-specific delegates/virtual functions. Making the existing virtual function OnAllPIEInstancesMapLoaded() execute later, people were open to that idea.

I’ve created UE-361591. Making the change is non-trivial, especially to help in your use case where you want the PIE clients to not just be connected, but also want them all to have finished loading the gameplay map. A client’s map load finishing, even when running under the same single process, can take an arbitary amount of engine ticks. I’m not sure when we would get to addressing UE-361591, since we’ll be treated it fairly low priority.

For a fool-proof method to know when clients have connected, you could explore the following options:

  • Bind to FCoreUObjectDelegates::PostLoadMapWithWorld to specifically count PIE instances that have loaded the gameplay map. You would have code specifically for PIE startup.
  • Await some server RPC that can only be sent if the client has loaded the map, like APlayerController::AcknowledgePossession. Count until the expected number of clients have acknowledged possession to start off async load requests.

Hopefully when we get to addressing UE-361591, there will be simpler options.

Thanks for getting back to me!

We’ll explore our options, but I’ve personally tested out FCoreUObjectDelegates::PostLoadMapWithWorld and it seemed to work good enough for a first proof of concept.

Allt the best!

Oliver

Happy to hear that! I’ll close this thread then.