Hi,
We have observed an issue when gc.AllowIncrementalReachability is enabled, especially with some other flags (the config can be found in the steps). It seems the UWorld & ULevel associated with the streamed in/out levels are never garbage collected with incremental reachability enabled, turning it off will immediately destroy them on the next GC.
One of the problems is that this forces anything that is reachable from the UWorld to be marked reachable, so the ULevel and anything that references, the problem was noticed when some fairly large data we stored in the ULevel::AssetUserData seemed to never get destroyed.
The actors and such of the cell seem to be cleared in FLevelStreamingGCHelper::PrepareStreamedOutLevelForGC so those are not reached during the analysis and get garbage collected, but not ex. the AssetUserData. We did solve this for now by manually clearing that link from ULevel::AssetUserData in the callback ULevel::OnCleanupLevel to allow it to be garbage collected, but the issue of the remaining UWorld & ULevel Objects still remain.
I may have tracked the offender down to UEngine::PerformGarbageCollectionAndCleanupActors, specifically the ForEachObjectOfClass(UWorld::StaticClass()), this seems to internally mark all the UWorlds as reachable, just commenting that to test seems to allow the objects to be garbage collected.
Any help and insight into this issue would be appreciated.
Steps to Reproduce
- Create a new blank project
- Add the provided sections (below) to DefaultEngine.ini or toggle them on manually
- Create a new level with the world partitioned landscape template
- Fly around the level to stream in and out cells as much as possible (also enable gc.ContinuousIncrementalGC=1)
- Another option is to manually override streaming range (with wp.Runtime.OverrideRuntimeLoadingRange) to a low value, and then a high value and repeat this to continuously load and unload a lot of cells
- Run a command like: obj list class=World occasionally
- Observe that there is an ever-increasing amount of such objects (UWorld & ULevel), the ones that should be garbage collected (and not reused) have been renamed to TrashedPackage_* but are never destroyed.
I ran this test with gc.ContinuousIncrementalGC=1 to make sure it was trying to run the incremental GC as often as possible but these objects were never destroyed.
Triggering a non-incremental GC will destroy these objects, or if I disable gc.AllowIncrementalReachability they will also be destroyed.
Add this config to DefaultEngine.ini:
[/Script/Engine.StreamingSettings]
s.ForceGCAfterLevelStreamedOut=0 ; Prevent the non-incremental GC from triggering constantly
[/Script/Engine.GarbageCollectionSettings]
gc.GarbageEliminationEnabled=False
gc.AllowIncrementalReachability=1
Hi,
Incremental GC is experimental and not supported. It’s only used for single-threaded servers internally under special circumstances.
Cheers, Johan Torp
Hi Johan,
We have been running with incremental GC for a bit now and except for this issue it seems to have been working well, and helped us alleviate a largre chunk of the hitches we have been experiencing. Is there any other issues you know about with the incremental GC to be wary of?
The documentation page reason for the experimental label to me does not seem to give the same level of caution, so maybe an update there would be nice (https://dev.epicgames.com/documentation/en-us/unreal-engine/incremental-garbage-collection-in-unreal-engine).
Cheers,
Robin Krokfors
Hi Robin,
I’ll try and update the docs but the main concern about incremental reachability is thread safety, specifically the fact that GIsIncrementalReachabilityPending is not guarded against race conditions. This may lead to some objects not being marked as reachable before GC finishes scanning objects.
We rely on GIsIncrementalReachabilityPending in TObjectPtr where we mark objects as reachable when they are assigned to a TObjectPtr variable. It’s possible that a worker thread may cache the old value of GIsIncrementalReachabilityPending which will then be changed on the game thread when GC starts and a TObjectPtr on said worker thread will fail to mark an object as reachable.
As for TrashedPackage leaks, I’d suggest trying to catch what marks these objects as reachable, maybe there’s a TObjectPtr which points to these objects and even though is not exposed to GC through a UProperty, it’s being constantly written to / copied and triggers MarkAsReachable(const UObjectBase* Obj)
Hi Robert!
Thanks for this information, great to have some more insight!
Just some more information after adding some logging + stack dump in MarkObjectItemAsReachable(FUObjectItem* ObjectItem) for objects that had TrashedPackage in the name, I got these two callstacks for UWorld objects (in editor):
UnrealEditor-CoreUObject.dll!MarkObjectItemAsReachable<0>() [E:\UE-Stock\UE561\UnrealEngine\Engine\Source\Runtime\CoreUObject\Private\UObject\GarbageCollection.cpp:6364]
UnrealEditor-CoreUObject.dll!UObjectBase::MarkAsReachable() [E:\UE-Stock\UE561\UnrealEngine\Engine\Source\Runtime\CoreUObject\Private\UObject\GarbageCollection.cpp:6414]
UnrealEditor-CoreUObject.dll!ForEachObjectOfClass() [E:\UE-Stock\UE561\UnrealEngine\Engine\Source\Runtime\CoreUObject\Private\UObject\UObjectHash.cpp:1888]
UnrealEditor-Engine.dll!UEngine::PerformGarbageCollectionAndCleanupActors() [E:\UE-Stock\UE561\UnrealEngine\Engine\Source\Runtime\Engine\Private\UnrealEngine.cpp:2040]
UnrealEditor-Engine.dll!UEngine::ConditionalCollectGarbage() [E:\UE-Stock\UE561\UnrealEngine\Engine\Source\Runtime\Engine\Private\UnrealEngine.cpp:1981]
UnrealEditor-Engine.dll!UWorld::Tick() [E:\UE-Stock\UE561\UnrealEngine\Engine\Source\Runtime\Engine\Private\LevelTick.cpp:1728]
UnrealEditor-UnrealEd.dll!UEditorEngine::Tick() [E:\UE-Stock\UE561\UnrealEngine\Engine\Source\Editor\UnrealEd\Private\EditorEngine.cpp:1941]
UnrealEditor-UnrealEd.dll!UUnrealEdEngine::Tick() [E:\UE-Stock\UE561\UnrealEngine\Engine\Source\Editor\UnrealEd\Private\UnrealEdEngine.cpp:533]
UnrealEditor.exe!FEngineLoop::Tick() [E:\UE-Stock\UE561\UnrealEngine\Engine\Source\Runtime\Launch\Private\LaunchEngineLoop.cpp:5625]
UnrealEditor.exe!GuardedMain() [E:\UE-Stock\UE561\UnrealEngine\Engine\Source\Runtime\Launch\Private\Launch.cpp:187]
UnrealEditor.exe!LaunchWindowsStartup() [E:\UE-Stock\UE561\UnrealEngine\Engine\Source\Runtime\Launch\Private\Windows\LaunchWindows.cpp:271]
UnrealEditor.exe!WinMain() [E:\UE-Stock\UE561\UnrealEngine\Engine\Source\Runtime\Launch\Private\Windows\LaunchWindows.cpp:339]
UnrealEditor-CoreUObject.dll!MarkObjectItemAsReachable<0>() [E:\UE-Stock\UE561\UnrealEngine\Engine\Source\Runtime\CoreUObject\Private\UObject\GarbageCollection.cpp:6364]
UnrealEditor-CoreUObject.dll!UObjectBase::MarkAsReachable() [E:\UE-Stock\UE561\UnrealEngine\Engine\Source\Runtime\CoreUObject\Private\UObject\GarbageCollection.cpp:6414]
UnrealEditor-CoreUObject.dll!ForEachObjectOfClass() [E:\UE-Stock\UE561\UnrealEngine\Engine\Source\Runtime\CoreUObject\Private\UObject\UObjectHash.cpp:1888]
UnrealEditor-UnrealEd.dll!UEditorEngine::Tick() [E:\UE-Stock\UE561\UnrealEngine\Engine\Source\Editor\UnrealEd\Private\EditorEngine.cpp:2300]
UnrealEditor-UnrealEd.dll!UUnrealEdEngine::Tick() [E:\UE-Stock\UE561\UnrealEngine\Engine\Source\Editor\UnrealEd\Private\UnrealEdEngine.cpp:533]
UnrealEditor.exe!FEngineLoop::Tick() [E:\UE-Stock\UE561\UnrealEngine\Engine\Source\Runtime\Launch\Private\LaunchEngineLoop.cpp:5625]
UnrealEditor.exe!GuardedMain() [E:\UE-Stock\UE561\UnrealEngine\Engine\Source\Runtime\Launch\Private\Launch.cpp:187]
UnrealEditor.exe!LaunchWindowsStartup() [E:\UE-Stock\UE561\UnrealEngine\Engine\Source\Runtime\Launch\Private\Windows\LaunchWindows.cpp:271]
UnrealEditor.exe!WinMain() [E:\UE-Stock\UE561\UnrealEngine\Engine\Source\Runtime\Launch\Private\Windows\LaunchWindows.cpp:339]
To test I added EObjectFlags::RF_MirroredGarbage as an exclusion flag on both of these ForEachObjectOfClass(UWorld::StaticClass()) to prevent the TrashedPackages from being iterated over, this seems to have solved the issue.
Not sure what the proper solution would be.
Cheers,
Robin Krokfors