RHI Crash on Packaged World Partition Streaming

Development Rendering
, , , , , , ,
1

I’m in a tough spot and could really use the help of any RenderThread gurus. I have spent the last few weeks trying to debug a hard in-game client crash many users have been reporting with no luck.

My interpretation of the logs is that the RHIThread / FRDGBuilder_Execute is attempting to render something that doesn’t exist during partition streaming, and crashes. When I try to read the minidump or AttachToProcess, a significant number of variables return null or unreadable.

Crash View (Users):

1
1331×199 7.39 KB

Crash Log Message (Abbreviated):

LogRHI: Error: Breadcrumbs 'RHIThread'
LogRHI: Error: Breadcrumbs 'Parallel'
Context 1/2
	00 FRDGBuilder_Execute
...
LogThreadingWindows: Error: Runnable thread Background Worker #14 crashed.
[2024.11.30-10.19.31:480][823]LogWindows: Error: === Critical error: ===
[2024.11.30-10.19.31:481][823]LogWindows: Error: 
[2024.11.30-10.19.31:481][823]LogWindows: Error: Fatal error!
[2024.11.30-10.19.31:482][823]LogWindows: Error: 
[2024.11.30-10.19.31:482][823]LogWindows: Error: Unhandled Exception: EXCEPTION_ACCESS_VIOLATION reading address 0x00000000000000df
...

Steps Taken to Investigate:

  • Created over 50+ different build variants that have various assets removed and settings changed/reset, hoping to find a ‘problem’ actor or setting.
    • Testing locally using load level 127.0.0.1
  • Analyzing output logs, Attaching Visual Studio debugger to client process to see breakpoints

Notes from Investigation:

  • The crash is inconsistent and there are no definitive steps to causing one, but there is a certain pattern that does trigger it (moving between two points on the world map that are quite far apart)
    • The Crash does not occur in-engine
  • The crash does not seem to be intrinsically linked to any particular actor. If I delete enough total actors, the crash does not happen. But through trial-and-error, there is no particular one that I can see that causes it
  • In some builds I am able to trigger this warning LogD3D12RHI: Warning: Descriptor cache ran out of sub allocated descriptor blocks! Moving to Context local View heap strategy.
    • I have tried increasing the heap size but no resolution. If I increase it too much I cannot open the uproject (assertion that heap size less than maxresource of something)
  • In some builds I am able to trigger this warning LogNetPackageMap: Warning: Using None instead of replicated reference to LandscapeHeightfieldCollisionComponent /Game/Maps/Maps_UE5/L_MVP/_Generated_/B9KF9Z4A1SFWE2F9XE0UNM9JF.L_MVP:PersistentLevel.LandscapeStreamingProxy_0I6FR9EO3797IE9SK52SR17M5_254_5_4_0.LandscapeHeightfieldCollisionComponent_139 because the level it's in has not been made visible
    • Unsure if this is relevant
  • When testing a ‘debug’ config build, a crash is not reported and a minidump is not created. The program simply closes
  • The dedicated server is fine, simply reports that the client disconnects due to network timeout when it crashes

Technical Game Setup / Flow:

  • Engine Version 5.4.4
  • This is a dedicated client/server game
    • One small ‘lobby’ level (single player), one large dedicated server open world map
  • On begin game, load into lobby. Authenticate and initiate server travel
  • Load into main level (dedicated server). Walk around to crash.
    • If character loads in a certain location, crash ensues

Environment Profiles:

  • Windows (10 & 11)
  • Medium to High-End Systems (GPUs & CPUs, 32+GB Ram)

Full Callstack:

LogThreadingWindows: Error: Runnable thread Background Worker #16 crashed.
LogWindows: Error: === Critical error: ===
LogWindows: Error: 
LogWindows: Error: Fatal error!
LogWindows: Error: 
LogWindows: Error: Unhandled Exception: EXCEPTION_ACCESS_VIOLATION reading address 0x00000000000000df
LogWindows: Error: 
LogWindows: Error: [Callstack] 0x00007ff73de72eac BlakesGameProjectClient-Win64-Test.exe!FD3D12DescriptorCache::SetRootConstantBuffers() [C:\Blakes_Game\UE5_4_4\Engine\Source\Runtime\D3D12RHI\Private\D3D12DescriptorCache.cpp:670]
LogWindows: Error: [Callstack] 0x00007ff73deadc47 BlakesGameProjectClient-Win64-Test.exe!FD3D12StateCache::ApplyState() [C:\Blakes_Game\UE5_4_4\Engine\Source\Runtime\D3D12RHI\Private\D3D12StateCache.cpp:588]
LogWindows: Error: [Callstack] 0x00007ff73de74381 BlakesGameProjectClient-Win64-Test.exe!FD3D12CommandContext::SetupDispatch() [C:\Blakes_Game\UE5_4_4\Engine\Source\Runtime\D3D12RHI\Private\D3D12Commands.cpp:108]
LogWindows: Error: [Callstack] 0x00007ff73de6bb1d BlakesGameProjectClient-Win64-Test.exe!FD3D12CommandContext::RHIDispatchIndirectComputeShader() [C:\Blakes_Game\UE5_4_4\Engine\Source\Runtime\D3D12RHI\Private\D3D12Commands.cpp:145]
LogWindows: Error: [Callstack] 0x00007ff73d50c4d5 BlakesGameProjectClient-Win64-Test.exe!FRHICommand<FRHICommandDispatchIndirectComputeShader,FRHICommandDispatchIndirectComputeShaderString1716>::ExecuteAndDestruct() [C:\Blakes_Game\UE5_4_4\Engine\Source\Runtime\RHI\Public\RHICommandList.h:1295]
LogWindows: Error: [Callstack] 0x00007ff73dff740b BlakesGameProjectClient-Win64-Test.exe!FRHICommandListBase::Execute() [C:\Blakes_Game\UE5_4_4\Engine\Source\Runtime\RHI\Private\RHICommandList.cpp:472]
LogWindows: Error: [Callstack] 0x00007ff73dff11db BlakesGameProjectClient-Win64-Test.exe!`FRHICommandListImmediate::QueueAsyncCommandListSubmit'::`15'::<lambda_2>::operator()() [C:\Blakes_Game\UE5_4_4\Engine\Source\Runtime\RHI\Private\RHICommandList.cpp:612]
LogWindows: Error: [Callstack] 0x00007ff73bff2ff1 BlakesGameProjectClient-Win64-Test.exe!TGraphTask<TFunctionGraphTaskImpl<void __cdecl(void),0> >::ExecuteTask() [C:\Blakes_Game\UE5_4_4\Engine\Source\Runtime\Core\Public\Async\TaskGraphInterfaces.h:1235]
LogWindows: Error: [Callstack] 0x00007ff73bfea70c BlakesGameProjectClient-Win64-Test.exe!`LowLevelTasks::FTask::Init<`FTaskGraphCompatibilityImplementation::QueueTask'::`5'::<lambda_1> >'::`5'::<lambda_1>::operator()() [C:\Blakes_Game\UE5_4_4\Engine\Source\Runtime\Core\Public\Async\Fundamental\Task.h:499]
LogWindows: Error: [Callstack] 0x00007ff73bfedd63 BlakesGameProjectClient-Win64-Test.exe!LowLevelTasks::TTaskDelegate<LowLevelTasks::FTask * __cdecl(bool),48>::TTaskDelegateImpl<`LowLevelTasks::FTask::Init<`FTaskGraphCompatibilityImplementation::QueueTask'::`5'::<lambda_1> >'::`5'::<lambda_1>,0>::CallAndMove() [C:\Blakes_Game\UE5_4_4\Engine\Source\Runtime\Core\Public\Async\Fundamental\TaskDelegate.h:171]
LogWindows: Error: [Callstack] 0x00007ff73bff3fc5 BlakesGameProjectClient-Win64-Test.exe!LowLevelTasks::FTask::ExecuteTask() [C:\Blakes_Game\UE5_4_4\Engine\Source\Runtime\Core\Public\Async\Fundamental\Task.h:627]
LogWindows: Error: [Callstack] 0x00007ff73bff3d62 BlakesGameProjectClient-Win64-Test.exe!LowLevelTasks::FScheduler::ExecuteTask() [C:\Blakes_Game\UE5_4_4\Engine\Source\Runtime\Core\Private\Async\Fundamental\Scheduler.cpp:155]
LogWindows: Error: [Callstack] 0x00007ff73c009a06 BlakesGameProjectClient-Win64-Test.exe!LowLevelTasks::FScheduler::WorkerMain() [C:\Blakes_Game\UE5_4_4\Engine\Source\Runtime\Core\Private\Async\Fundamental\Scheduler.cpp:397]
LogWindows: Error: [Callstack] 0x00007ff73bfed970 BlakesGameProjectClient-Win64-Test.exe!UE::Core::Private::Function::TFunctionRefCaller<`LowLevelTasks::FScheduler::CreateWorker'::`2'::<lambda_1>,void __cdecl(void)>::Call() [C:\Blakes_Game\UE5_4_4\Engine\Source\Runtime\Core\Public\Templates\Function.h:406]
LogWindows: Error: [Callstack] 0x00007ff73c092404 BlakesGameProjectClient-Win64-Test.exe!FThreadImpl::Run() [C:\Blakes_Game\UE5_4_4\Engine\Source\Runtime\Core\Private\HAL\Thread.cpp:68]
LogWindows: Error: [Callstack] 0x00007ff73c224477 BlakesGameProjectClient-Win64-Test.exe!FRunnableThreadWin::Run() [C:\Blakes_Game\UE5_4_4\Engine\Source\Runtime\Core\Private\Windows\WindowsRunnableThread.cpp:149]
LogWindows: Error: [Callstack] 0x00007ff73c21c37a BlakesGameProjectClient-Win64-Test.exe!FRunnableThreadWin::GuardedRun() [C:\Blakes_Game\UE5_4_4\Engine\Source\Runtime\Core\Private\Windows\WindowsRunnableThread.cpp:79]
LogWindows: Error: [Callstack] 0x00007ffe232d7374 KERNEL32.DLL!UnknownFunction []
LogWindows: Error: 
LogWindows: Error: Crash in runnable thread Background Worker #16
LogExit: Executing StaticShutdownAfterError
LogWindows: FPlatformMisc::RequestExit(1, FRunnableThreadWin::GuardedRun.ExceptionHandler)
LogWindows: FPlatformMisc::RequestExitWithStatus(1, 3, FRunnableThreadWin::GuardedRun.ExceptionHandler)
LogCore: Engine exit requested (reason: Win RequestExit)
...

Further Detail:
I have created a public read-only GDrive folder with additional screenshots, with the raw logs and minidumps. Please feel free to peruse - PublicDebugCrashDiagnosing - Google Drive

Thanks for your time :slight_smile: