Garbage collection crash.

Two-faced · March 11, 2020, 10:50am

From time to time we have a crash while changing dedicated servers:

It’s a simple OpenLevel with IP:Port level name.

Any ideas, guys?

Two-faced · March 11, 2020, 11:14am

More info:
Assertion failed: ObjectItem->IsUnreachable()

ExtraLifeMatt · March 11, 2020, 1:35pm

Looks like an LowEntryHttpRequest is somehow being marked for GC (forcefully being set to MarkAsPendingKill), but it’s actually still being held on by a UPROPERTY somewhere.

Two-faced · March 11, 2020, 3:45pm

@ExtraLifeMatt think so? But we are on index 1078. Trying to access object that is already dead.

Two-faced · March 11, 2020, 3:48pm

Another GC crash. Kinda similar. At some point engine is trying to access object that is already dead.

Two-faced · March 11, 2020, 4:18pm

Our GC settings:
[/Script/Engine.GarbageCollectionSettings]
gc.MaxObjectsNotConsideredByGC=10000
gc.SizeOfPermanentObjectPool=6000000
gc.CreateGCClusters=False
gc.NumRetriesBeforeForcingGC=4
gc.MaxObjectsInEditor=46229396
gc.MaxObjectsInGame=16777216
gc.AllowParallelGC=True
gc.IncrementalBeginDestroyEnabled=False

Two-faced · March 11, 2020, 5:02pm

And another one. With gc.AllowParallelGC=False


Unhandled Exception: EXCEPTION_ACCESS_VIOLATION 0xd6580400

UE4Editor_Engine!FSkeletalMeshLODModel::~FSkeletalMeshLODModel()
UE4Editor_Engine!FSkeletalMeshModel::`scalar deleting destructor'()
UE4Editor_Engine!USkeletalMesh::~USkeletalMesh() [d:\build\++ue4\sync\engine\source\runtime\engine\private\skeletalmesh.cpp:397]
UE4Editor_Engine!USkeletalMesh::`vector deleting destructor'()
UE4Editor_CoreUObject!FAsyncPurge::TickDestroyObjects() [d:\build\++ue4\sync\engine\source\runtime\coreuobject\private\uobject\garbagecollection.cpp:295]
UE4Editor_CoreUObject!IncrementalDestroyGarbage() [d:\build\++ue4\sync\engine\source\runtime\coreuobject\private\uobject\garbagecollection.cpp:1634]
UE4Editor_CoreUObject!IncrementalPurgeGarbage() [d:\build\++ue4\sync\engine\source\runtime\coreuobject\private\uobject\garbagecollection.cpp:1367]
UE4Editor_CoreUObject!CollectGarbageInternal() [d:\build\++ue4\sync\engine\source\runtime\coreuobject\private\uobject\garbagecollection.cpp:1903]
UE4Editor_CoreUObject!CollectGarbage() [d:\build\++ue4\sync\engine\source\runtime\coreuobject\private\uobject\garbagecollection.cpp:1982]
UE4Editor_Engine!UEngine::TrimMemory() [d:\build\++ue4\sync\engine\source\runtime\engine\private\unrealengine.cpp:12811]
UE4Editor_Engine!UEngine::LoadMap() [d:\build\++ue4\sync\engine\source\runtime\engine\private\unrealengine.cpp:12446]
UE4Editor_Engine!UEngine::TickWorldTravel() [d:\build\++ue4\sync\engine\source\runtime\engine\private\unrealengine.cpp:12212]
UE4Editor_Engine!UGameEngine::Tick() [d:\build\++ue4\sync\engine\source\runtime\engine\private\gameengine.cpp:1655]
UE4Editor!FEngineLoop::Tick() [d:\build\++ue4\sync\engine\source\runtime\launch\private\launchengineloop.cpp:4485]
UE4Editor!GuardedMain() [d:\build\++ue4\sync\engine\source\runtime\launch\private\launch.cpp:173]
UE4Editor!GuardedMainWrapper() [d:\build\++ue4\sync\engine\source\runtime\launch\private\windows\launchwindows.cpp:134]
UE4Editor!WinMain() [d:\build\++ue4\sync\engine\source\runtime\launch\private\windows\launchwindows.cpp:263]
UE4Editor!__scrt_common_main_seh() [d:\agent\_work\2\s\src\vctools\crt\vcstartup\src\startup\exe_common.inl:288]
kernel32
ntdll

Two-faced · March 12, 2020, 9:28am

Out of scope of free support?

ExtraLifeMatt · March 12, 2020, 12:38pm

More like it’s hard to tell the cause from just random callstacks. Try and reproduce it on a level that doesn’t use your game code, start disabling systems one by one, etc. You can also search the open bug list in the UE4 bug tracker to see if this is already known (or grab the preview of 4.25 and see if this still occurs there as well).

That last callstack in particular is strange since I don’t see how you can have MappedRegion valid, but MappedHandle not be given that MappedRegion is only initialized after MappedHandle is known to be valid.

Two-faced · March 12, 2020, 4:05pm

It’s sad that usual fixing algorithms are very slow at this point. Each iteration is like 1 hour, cause I have to wait until it crashes to prove.

I wish I had at least the direction. It could be our 10 streaming levels, or StreamableManager.RequestAsyncLoad that we are using a lot, or corrupted blueprints, or http deferred requests, or bad texture management, or just huge amount of objects persisting in GC array. Anything. I could spend months but I have 2 days.

Two-faced · March 13, 2020, 9:53am

Same with
[/Script/Engine.GarbageCollectionSettings]
gc.MaxObjectsNotConsideredByGC=10000
gc.SizeOfPermanentObjectPool=6000000
gc.CreateGCClusters=False
gc.NumRetriesBeforeForcingGC=4
gc.MaxObjectsInEditor=46229396
gc.MaxObjectsInGame=16777216
gc.AllowParallelGC=True
gc.IncrementalBeginDestroyEnabled=False
gc.MultithreadedDestructionEnabled=True
gc.FlushStreamingOnGC=True

Two-faced · March 13, 2020, 10:51am

Two-faced · March 13, 2020, 12:34pm

Two-faced · March 16, 2020, 1:22pm

Could be highy related: Unreal Engine Issues and Bug Tracker (UE-46063). If so I can provide any information you need.

Darkloud1989 · March 19, 2020, 12:05pm

This may or may not be related, but I moved a project from 4.23 to 4.24, and started getting garbage collection crashes and stuttering until I turned off UDP and TCO Transport… I know this isn’t a solution for you, and may be unrelated, but I thought I would mention it. Some uncommon issue with 4.24