Recently I’ve noticed our game sometimes gets severe virutal memory spikes while playing it in editor. It seemed inconsistent at first, but it would regularly reappear so I took it upon myself to try to see what’s causing the problem.
We are using forked UE 5.3 so I’ve cloned 5.3.2 from github, compiled it in debug and loaded ‘Valley of the Ancients’ to try to figure out if there’s an error on our end or in the UE build. Soon enough I’ve had 100% reproduction rate for that same problem.
This is VOTA regularly on my PC (32 GB RAM, 59.5 GB available virtual memory):
And this is after a spike:
RAM and virtual memory get stuck at this point and editor reactivation is required.
When I had a consistent repro, I’ve used Memory Insights to get a better look at what’s causing the problem. Spike was easily noticeable so I was able to pinpoint where the issue actually starts:
I’ll just quickly return to the leak reproduction to explain what’s happening. I’m starting VOTA on the ‘AncientWorld’ map. ‘AncientBattle’ plugin was loaded on editor startup so it deactivates on PIE play since it’s not yet relevant. After a few quick interactions it’s possible to enter the next map. This deactivates ‘HoverDrone’ plugin and again loads ‘AncientBattle’. Both of these are game feature plugins designed to keep the game content modularized, all fine and well - game is still normally playable. Stopping PIE doesn’t seem to unload this plugin and everything still seems fine, but starting a second session first deactivates ‘AncientBattle’ and this is the point of no return, RAM usage skyrockets and game becomes borderline unplayable.
Issue seem to stem from here, found at ‘PackageTools.cpp’, line 555, this was implement with commit 36234e0 labeled “Fix potential assert/crash when renaming packages” (I’m not trying to pass blame on anyone here, I’m just interested in the reason behind the leak and how it could potentially be fixed):
// Calling ::ResetLoaders now will force any bulkdata objects still attached to the FLinkerLoad to load
// their payloads into memory. If we dont call this now, then the version that will be called during
// garbage collection will cause the bulkdata objects to be invalidated rather than loading the payloads
// into memory.
// This might seem odd, but if the package we are unloading is being renamed, then the inner UObjects will
// be moved to the newly named package rather than being garbage collected and so we need to make sure that
// their bulkdata objects remain valid, otherwise renamed packages will not save correctly and cease to function.
ResetLoaders(TArray<UObject*>(PackagesToUnload.Array()));
Core of the problem is found at ‘LinkerLoad.cpp’, line 5706:
void FLinkerLoad::LoadAndDetachAllBulkData()
{
#if WITH_EDITOR
// Detach all lazy loaders.
const bool bEnsureAllBulkDataIsLoaded = true;
DetachAllBulkData(bEnsureAllBulkDataIsLoaded);
#endif
}
Not to leave anything hanging, here’s a full stack:
It seems that when plugin is deactivated all its data is loaded into memory. Accompanying comment tries to explain logic behind loading payloads, but I don’t understand why it’s exactly necessary or how does it correlate to package renaming (is this something that happens on package reloads?). What I know is that commenting it out stops invoking massive memory spikes without any apparent (but probably unavoidable) problems.
If someone has any ideas or comments on how unreal packaging works, or anything adjacent to the issue - I’ll really appreciate your input. And who knows, maybe along the way we’ll fix this edgecase Unreal Editor nuisance. 