It could totally be a memstomp, though ASAN has not surfaced anything suspicious yet.
Each server logs this on startup: LogStreaming: Display: AsyncLoading2 - Created: Event Driven Loader: false, Async Loading Thread: false, Async Post Load: false, and another later: LogStreaming: Display: AsyncLoading2 - Thread Started: false, IsInitialLoad: false. I’ve not seen any FRunnableThreads named FAsyncLoadingThread in cores or when inspecting a running server with lldb, I assume this means no ALT and we’re timeslicing async loading on the GameThread?
Most of the core GameThread stacks are from WorldPartition kicking a flush:
* frame #0: 0x0000000004f8cfc0 Server-Linux-ShippingFGenericPlatformMisc::RaiseException(ExceptionCode=1) at GenericPlatformMisc.cpp:738:16
frame #1: 0x0000000005115301 Server-Linux-ShippingFOutputDevice::LogfImpl(this=..., Fmt=u"%s") at OutputDevice.cpp:81:2 frame #2: 0x0000000005063016 Server-Linux-ShippingUE::Logging::Private::BasicFatalLog(Category=…, Log=…) at StructuredLog.cpp:1104:2
frame #3: 0x00000000052e55d5 Server-Linux-ShippingFAsyncLoadingThread2::FlushLoading(this=..., RequestIDs=TConstArrayView<int32> @ 0x00005cb8f19f7190) at AsyncLoading2.cpp:9991:7 frame #4: 0x00000000052ff400 Server-Linux-ShippingFlushAsyncLoading(RequestIds=TConstArrayView @ 0x00005cb8e4b40e10) at AsyncPackageLoader.cpp:327:19
frame #5: 0x000000000903b959 Server-Linux-ShippingUWorld::FlushAsyncLevelRequests(this=...) at World.cpp:4375:3 frame #6: 0x00000000090356a2 Server-Linux-ShippingUWorld::FlushLevelStreaming(EFlushLevelStreamingType) [inlined] UWorld::FlushLevelStreaming(EFlushLevelStreamingType)::$_0::operator()(this=…) const at World.cpp:4686:4
frame #7: 0x0000000009035691 Server-Linux-ShippingUWorld::FlushLevelStreaming(this=..., FlushType=Full) at World.cpp:4693:2 frame #8: 0x000000000903b07d Server-Linux-ShippingUWorld::BlockTillLevelStreamingCompleted(this=…) at World.cpp:4250:3
frame #9: 0x00000000090a91d6 Server-Linux-ShippingTBaseUObjectMethodDelegateInstance<false, UWorldPartition, void (), FDefaultDelegateUserPolicy>::ExecuteIfSafe() const [inlined] decltype(UE::Core::Private::DereferenceIfNecessary<UWorldPartition>(Forward<UWorldPartition*&>(fp0), &fp0).*fp()) Invoke<void (UWorldPartition::*)(), UWorldPartition*&, UWorldPartition>(PtrMemFun=..., Target=...) at Invoke.h:66:9 frame #10: 0x00000000090a91c5 Server-Linux-ShippingTBaseUObjectMethodDelegateInstance<false, UWorldPartition, void (), FDefaultDelegateUserPolicy>::ExecuteIfSafe() const [inlined] decltype(auto) UE::Core::Private::Tuple::TTupleBase<TIntegerSequence>::ApplyAfter<void (UWorldPartition::* const&)(), UWorldPartition*&>(this=…, Func=…, Args=…) const & at Tuple.h:317:143
frame #11: 0x00000000090a91bd Server-Linux-ShippingTBaseUObjectMethodDelegateInstance<false, UWorldPartition, void (), FDefaultDelegateUserPolicy>::ExecuteIfSafe(this=...) const at DelegateInstancesImpl.h:667:24 frame #12: 0x00000000090d39f9 Server-Linux-ShippingAWorldSettings::NotifyBeginPlay() [inlined] void TMulticastDelegateBase::Broadcast<IBaseDelegateInstance<void (), FDefaultDelegateUserPolicy>>(this=…) const at MulticastDelegateBase.h:257:128
frame #13: 0x00000000090d39b2 Server-Linux-ShippingAWorldSettings::NotifyBeginPlay() [inlined] TMulticastDelegate<void (), FDefaultDelegateUserPolicy>::Broadcast(this=...) const at DelegateSignatureImpl.inl:1079:26 frame #14: 0x00000000090d39b2 Server-Linux-ShippingAWorldSettings::NotifyBeginPlay(this=…) at WorldSettings.cpp:331:30
frame #15: 0x00000000087b515c Server-Linux-ShippingAGameStateBase::HandleBeginPlay(this=...) at GameStateBase.cpp:227:22 frame #16: 0x000000000903e213 Server-Linux-ShippingUWorld::BeginPlay(this=…) at World.cpp:5439:13
frame #17: 0x0000000008f8c2ca Server-Linux-ShippingUEngine::LoadMap(this=..., WorldContext=..., URL=..., Pending=..., Error=...) at UnrealEngine.cpp:16011:24 frame #18: 0x0000000008f89335 Server-Linux-ShippingUEngine::Browse(this=…, WorldContext=…, URL=FURL [Content removed] Error=…) at UnrealEngine.cpp:15154:10
frame #19: 0x0000000008f8a463 Server-Linux-ShippingUEngine::TickWorldTravel(this=..., Context=..., DeltaSeconds=...) at UnrealEngine.cpp:15317:33 frame #20: 0x0000000008755ded Server-Linux-ShippingUGameEngine::Tick(this=…, DeltaSeconds=…, bIdleMode=false) at GameEngine.cpp:1773:4
frame #21: 0x0000000009439169 Server-Linux-ShippingFEngineLoop::Tick(this=...) at LaunchEngineLoop.cpp:5863:12 frame #22: 0x000000000943b5ca Server-Linux-ShippingGuardedMain(char16_t const*) [inlined] EngineTick() at Launch.cpp:69:14`If full stacks from all threads in the core would be useful please let me know where I can upload/post files - they are too large to paste here.
Thanks,
-blake