Crash in ~TLockFreeFixedSizeAllocator() - check(GetNumUsed()==0) in dedicated server

Development Programming & Scripting Multiplayer & Networking
, ,
1

If in a dedicated server we call FPlatformMisc::RequestExit(false) (in the game thread or in a delegate not running in the game thread), in ~TLockFreeFixedSizeAllocator, GetNumUsed returns different to 0 and the server crashes in check(GetNumUsed() == 0);

This seems to be a leak and investigating (adding some leak tracker code in that class) we got this allocate callstack that never gets freed:

[0] FUnixPlatformStackWalk::CaptureStackBackTrace(unsigned long long*, unsigned int, void*) (Engine/Source/./Runtime/Core/Private/Unix/UnixPlatformStackWalk.cpp:806)

[1] TLockFreeFixedSizeAllocator<65536, 64, FThreadSafeCounter>::Allocate(int) (Engine/Source/Runtime/Core/Public/Containers/LockFreeFixedSizeAllocator.h:339)

[2] FPageAllocator::Alloc(int) (Engine/Source/./Runtime/Core/Private/Misc/MemStack.cpp:242)

[3] TLinearAllocatorBase<FDefaultBlockAllocationTag, (ELinearAllocatorThreadPolicy)0>::AllocateBlock(TLinearAllocatorBase<FDefaultBlockAllocationTag, (ELinearAllocatorThreadPolicy)0>::FBlockHeader*&) (Engine/Source/Runtime/Core/Public/Experimental/ConcurrentLinearAllocator.h:321)

[4] TLinearAllocatorBase<FDefaultBlockAllocationTag, (ELinearAllocatorThreadPolicy)0>::Malloc(unsigned long, unsigned int) (Engine/Source/Runtime/Core/Public/Experimental/ConcurrentLinearAllocator.h:424)

[5] FRenderCommandList::Create(ERenderCommandListFlags, FMemStackBase::EPageSize) (Engine/Source/Runtime/RenderCore/Public/RenderingThread.h:845)

[6] UWorld::SendAllEndOfFrameUpdatesInternal(UWorld::EEndOfFrameUpdateLocation) (Engine/Source/./Runtime/Engine/Private/LevelTick.cpp:1224)

[7] UWorld::SendAllEndOfFrameUpdates() (Engine/Source/./Runtime/Engine/Private/LevelTick.cpp:1350)

[8] UEngine::SendWorldEndOfFrameUpdates() (Engine/Source/./Runtime/Engine/Private/UnrealEngine.cpp:1759)

[9] UEngine::PreGarbageCollect() (Engine/Source/Runtime/Core/Public/ProfilingDebugging/CpuProfilerTrace.h:236)

[10] TBaseStaticDelegateInstance<void (), FDefaultDelegateUserPolicy>::ExecuteIfSafe() const (Engine/Source/Runtime/Core/Public/Delegates/DelegateInstancesImpl.h:827)

[11] void TMulticastDelegateBase<FDefaultDelegateUserPolicy>::Broadcast<IBaseDelegateInstance<void (), FDefaultDelegateUserPolicy> >() const (Engine/Source/Runtime/Core/Public/Delegates/MulticastDelegateBase.h:301)

[12] void UE::GC::PreCollectGarbageImpl<false>(EObjectFlags) (Engine/Source/Runtime/Core/Public/ProfilingDebugging/CpuProfilerTrace.h:236)

[13] UE::GC::FReachabilityAnalysisState::PerformReachabilityAnalysisAndConditionallyPurgeGarbage(bool) (Engine/Source/./Runtime/CoreUObject/Private/UObject/GarbageCollection.cpp:5872)

[14] TryCollectGarbage(EObjectFlags, bool) (Engine/Source/./Runtime/CoreUObject/Private/UObject/GarbageCollection.cpp:5527)

[15] UEngine::PerformGarbageCollectionAndCleanupActors() (Engine/Source/./Runtime/Engine/Private/UnrealEngine.cpp:2034)

[16] UEngine::ConditionalCollectGarbage() (Engine/Source/./Runtime/Engine/Private/UnrealEngine.cpp:1982)

[17] UWorld::Tick(ELevelTick, float) (Engine/Source/Runtime/Core/Public/ProfilingDebugging/CpuProfilerTrace.h:236)

[18] UGameEngine::Tick(float, bool) (Engine/Source/./Runtime/Engine/Private/GameEngine.cpp:1883)

[19] FEngineLoop::Tick() (Engine/Source/Runtime/Core/Public/ProfilingDebugging/CpuProfilerTrace.h:236)

[20] GuardedMain(char16_t const*) (Engine/Source/./Runtime/Launch/Private/Launch.cpp:190)

[21] CommonUnixMain(int, char**, int (*)(char16_t const*), void (*)()) (Engine/Source/./Runtime/Unix/UnixCommonStartup/Private/UnixCommonStartup.cpp:323)

The interesting line is: UWorld::SendAllEndOfFrameUpdatesInternal(UWorld::EEndOfFrameUpdateLocation) - LevelTick.cpp:1224, which in our version is:

FRenderCommandList* RootRenderCommandList = FRenderCommandList::Create(ERenderCommandListFlags::CloseOnSubmit);

It seems this is never properly removed.

What is the best approach to avoid this happening?

We have implemented this quick workaround, that works, but we are unsure it is the right fix for this situation:

FRenderCommandList* RootRenderCommandList = nullptr;

TOptional<FRenderCommandList::FRecordScope> RecordScope;

if (!IsRunningDedicatedServer())

{

RootRenderCommandList = FRenderCommandList::Create(ERenderCommandListFlags::CloseOnSubmit);

RecordScope.Emplace(RootRenderCommandList, FRenderCommandList::EStopRecordingAction::Submit);

}

Thank you,

Jose

[Attachment Removed]

2

Steps to Reproduce

  • Run a dedicated server solution.
  • In the code, with some trigger, call FPlatformMisc::RequestExit(false)
  • Observe how in ~TLockFreeFixedSizeAllocator() GetNumUsed() is different to 0 and the server crashes.

We have observed this by calling FPlatformMisc::RequestExit in a delegate (in our case from a graceful termination) or by calling it in the main thread.

[Attachment Removed]

3

Hello,

I am wondering if anyone at Epic has looked into this yet please?

Thank you!

Jose

[Attachment Removed]

4

Hi Jose,

Sorry for the delay. I’ve had some trouble trying to repro the crash you described. Specifically, the UWorld::SendAllEndOfFrameUpdatesInternal function is never called in a dedicated server build in the examples I’ve tested, which would mean the FRenderCommandList object is never created, and therefore never causes a leak and an assert on shutdown. I’ve also tried hacking the AActor::Tick function to force a call to MarkRenderStateDirty, but the calls are all gated by bRenderStateCreated being false on the components, which would make sense in a server build that does not need render state.

Can you confirm what components are actually causing you to hit the UWorld::SendAllEndOfFrameUpdatesInternal path?

Cheers,

Luke

[Attachment Removed]

5

I’m not sure what exactly is going on there or why it’s not something we are seeing internally, but that workaround should be fine for the server. The only thing is that you need to make the scope take the outer loop as well. It looks like it contains the scope to that if block. You could do this with a TOptional for example. EDIT: Nevermind, I totally misread the code! You’re already doing that.

[Attachment Removed]

6

Hello,

Thank you [mention removed]​ and [mention removed]​ for your answers and very sorry for the delay.

We are in the middle of a refactor on some features and I have not been able to check in more detail your question Luke, but we have not changed anything that could have affected bRenderStateCreated in the components, and the FRenderCommandList object is explicitly created in UWorld::SendAllEndOfFrameUpdatesInternal in this line:

FRenderCommandList* RootRenderCommandList = FRenderCommandList::Create(ERenderCommandListFlags::CloseOnSubmit);

And that, in 5.7.3 appears without any protection to not be created in the server.

Not sure if this FRenderCommandList being the one you are referring to ?

As soon as I have more time I will try to see which components cause UWorld::SendAllEndOfFrameUpdatesInternal. How do you suggest this to be done?

Thank you,

Jose

[Attachment Removed]

7

The server path skips a bunch of rendering code. The issue is probably due to some mismatch in reference counting in the render command list implementation which is somehow only affecting your specific setup. I would just skip the render command list entirely on the server.

[Attachment Removed]