stompmalloc best practices

Development Programming & Scripting
, ,
1

We are facing a crash that has the following callstack :

[ 00 ] RtlpWaitOnCriticalSection ( ntdll.dll )

[ 01 ] RtlpEnterCriticalSectionContended ( ntdll.dll )

[ 02 ] RtlEnterCriticalSection ( ntdll.dll )

[ 03 ] EnterCriticalSection(Windows::CRITICAL_SECTION *) ( MinimalWindowsApi.h:238 )

[ 04 ] FWindowsCriticalSection::Lock() ( WindowsCriticalSection.h:44 )

[ 05 ] FScopeLock::{ctor}(FWindowsCriticalSection *) ( ScopeLock.h:39 )

[ 06 ] FD3D12BaseShaderResource::RemoveRenameListener(FD3D12ShaderResourceRenameListener *) ( D3D12Resources.h:893 )

[ 07 ] FD3D12View::~FD3D12View() ( D3D12View.cpp:242 )

[ 08 ] FD3D12ShaderResourceView_RHI::`scalar deleting destructor’(unsigned int) ( LastFlagClient-Win64-Shipping.exe )

[ 09 ] FRHIResource::DeleteResources(TArray<FRHIResource *,TSizedDefaultAllocator<32> > const &) ( RHIResources.cpp:71 )

[ 10 ] FRHICommandListExecutor::FSubmitState::Submit(FRHICommandListExecutor::FSubmitState::FSubmitArgs const &) ( RHICommandList.cpp:1056 )

[ 11 ] UE::Core::Private::Function::TFunctionRefBase<UE::Core::Private::Function::TFunctionStorage<0>,void __cdecl(void)>::operator()() ( Function.h:470 )

[ 12 ] FRHICommandListExecutor::FTaskPipe::Execute(FRHICommandListExecutor::FTaskPipe::FTask *,TRefCountPtr<FBaseGraphTask> const &) ( RHICommandList.cpp:627 )

[…]

Reading about RtlEnterCriticalSection, I don’t see how it could crash unless it’s not properly initialized but FScopeLock will make sure it is initialized so that triggered my assumption that we do get memory stomping or a race condition.

I’ve tried to run with stompmalloc but it makes the game so slow that it’s unplayable. We only repro the crash rarely so that doesn’t become an option.

Is there a way to use stompmalloc only for certain allocators? How would I make that happen?

I am also planning to run with -enableasan.

2

Steps to Reproduce

3

Hi,

When you use -StompMalloc, the engine replaces the global allocator (where most of the allocations goes) with the StompAllocator instance at boot time. That’s how it’s designed and since most allocations are going through the base allocator, you are paying the global price and it’s slow. At the same time, since most of the allocations goes there, you also more likely to capture a stomp. Using -enableAsan with the game is likely going to be slow too.

See: D:\UE_5.5\Engine\Source\Runtime\Core\Private\Windows\WindowsPlatformMemory.cpp for setting up StompMalloc in:

FMalloc* FWindowsPlatformMemory::BaseAllocator()If the crash always happen in FD3D12View::~FD3D12View with the same call stack, the corrupted memory is very likely the member ResourceInfo.BaseResource unless FD3D12View is also corrupted (or already destroyed). If you have a core dump, you should be able to tell which one look corrupted. Did you consider manually instrumenting this? I’m not a rendering expert, but I would study a bit how the member is assigned and I’ll probably try adding a couple of data breakpoint or conditional breakpoint to check if somethings doesn’t corrupt it or change it (race condition). You can add buffer before/after the critical section member in FD3D12BaseShaderResource add check if something stomp those buffer (with data breakpoints). Something like that.

Hope this help.

Regards,

Patrick