Crazy crash happening in 5.3: "LogRHI: Error: Breadcrumbs 'RHIThread'"

Development Programming & Scripting Multiplayer & Networking
,
1

Hi guys,

I just updated my project (multiplayer game running on a dedicated server) from 5.2 to 5.3, and am now getting the crash below during gameplay (on the client side). I have no idea how to debug this…Seems related to something in DirectX 12? Any ideas what this could be?

[2024.07.04-12.34.09:512][434]LogRHI: Error: Breadcrumbs 'RHIThread'

[2024.07.04-12.34.09:512][434]LogRHI: Error: Breadcrumbs 'Parallel'
Context 1/2
	00 Scene
	01 Scene
	02 LumenSceneLighting
	03 Radiosity
	04 BasePass

[2024.07.04-12.34.09:512][434]LogWindows: Could not start crash report client using ../../../Engine/Binaries/Win64/CrashReportClient-Win64-Debug.exe
[2024.07.04-12.34.09:512][434]LogMemory: Platform Memory Stats for Windows
[2024.07.04-12.34.09:512][434]LogMemory: Process Physical Memory: 13025.76 MB used, 14215.39 MB peak
[2024.07.04-12.34.09:512][434]LogMemory: Process Virtual Memory: 19184.47 MB used, 20073.73 MB peak
[2024.07.04-12.34.09:512][434]LogMemory: Physical Memory: 28499.36 MB used,  36957.70 MB free, 65457.06 MB total
[2024.07.04-12.34.09:512][434]LogMemory: Virtual Memory: 41190.10 MB used,  38602.96 MB free, 79793.06 MB total

[2024.07.04-12.34.09:512][434]LogThreadingWindows: Error: Runnable thread Background Worker #12 crashed.
[2024.07.04-12.34.09:512][434]LogWindows: Error: === Critical error: ===
[2024.07.04-12.34.09:512][434]LogWindows: Error: 
[2024.07.04-12.34.09:512][434]LogWindows: Error: Fatal error!
[2024.07.04-12.34.09:512][434]LogWindows: Error: 
[2024.07.04-12.34.09:512][434]LogWindows: Error: Unhandled Exception: EXCEPTION_ACCESS_VIOLATION reading address 0xffffffffffffffff
[2024.07.04-12.34.09:512][434]LogWindows: Error: 
[2024.07.04-12.34.09:512][434]LogWindows: Error: [Callstack] 0x00007ff6d8584f2b Shooter.exe!BindUniformBuffer() [A:\UnrealSource5p3\UnrealEngine\Engine\Source\Runtime\D3D12RHI\Private\D3D12Commands.cpp:85]
[2024.07.04-12.34.09:512][434]LogWindows: Error: [Callstack] 0x00007ff6d85b2952 Shooter.exe!SetShaderParametersOnContext() [A:\UnrealSource5p3\UnrealEngine\Engine\Source\Runtime\D3D12RHI\Private\D3D12Commands.cpp:1153]
[2024.07.04-12.34.09:512][434]LogWindows: Error: [Callstack] 0x00007ff6d85a3105 Shooter.exe!FD3D12CommandContext::RHISetShaderParameters() [A:\UnrealSource5p3\UnrealEngine\Engine\Source\Runtime\D3D12RHI\Private\D3D12Commands.cpp:1168]
[2024.07.04-12.34.09:512][434]LogWindows: Error: [Callstack] 0x00007ff6d88a6742 Shooter.exe!FRHICommandSetShaderParameters<FRHIGraphicsShader>::Execute() [A:\UnrealSource5p3\UnrealEngine\Engine\Source\Runtime\RHI\Public\RHICommandListCommandExecutes.inl:135]
[2024.07.04-12.34.09:512][434]LogWindows: Error: [Callstack] 0x00007ff6d736695b Shooter.exe!FRHICommand<FRHICommandSetShaderParameters<FRHIGraphicsShader>,FRHICommandSetShaderParametersString1416>::ExecuteAndDestruct() [A:\UnrealSource5p3\UnrealEngine\Engine\Source\Runtime\RHI\Public\RHICommandList.h:1247]
[2024.07.04-12.34.09:512][434]LogWindows: Error: [Callstack] 0x00007ff6d88a860d Shooter.exe!FRHICommandListBase::Execute() [A:\UnrealSource5p3\UnrealEngine\Engine\Source\Runtime\RHI\Private\RHICommandList.cpp:456]
[2024.07.04-12.34.09:512][434]LogWindows: Error: [Callstack] 0x00007ff6d8894179 Shooter.exe!`FRHICommandListImmediate::QueueAsyncCommandListSubmit'::`27'::<lambda_2>::operator()() [A:\UnrealSource5p3\UnrealEngine\Engine\Source\Runtime\RHI\Private\RHICommandList.cpp:596]
[2024.07.04-12.34.09:512][434]LogWindows: Error: [Callstack] 0x00007ff6d45d1de3 Shooter.exe!TGraphTask<TFunctionGraphTaskImpl<void __cdecl(void),0> >::ExecuteTask() [A:\UnrealSource5p3\UnrealEngine\Engine\Source\Runtime\Core\Public\Async\TaskGraphInterfaces.h:1266]
[2024.07.04-12.34.09:512][434]LogWindows: Error: [Callstack] 0x00007ff6d4533408 Shooter.exe!`LowLevelTasks::FTask::Init<`FTaskGraphCompatibilityImplementation::QueueTask'::`5'::<lambda_1> >'::`11'::<lambda_1>::operator()() [A:\UnrealSource5p3\UnrealEngine\Engine\Source\Runtime\Core\Public\Async\Fundamental\Task.h:499]
[2024.07.04-12.34.09:512][434]LogWindows: Error: [Callstack] 0x00007ff6d453aa53 Shooter.exe!LowLevelTasks::TTaskDelegate<LowLevelTasks::FTask * __cdecl(bool),48>::TTaskDelegateImpl<`LowLevelTasks::FTask::Init<`FTaskGraphCompatibilityImplementation::QueueTask'::`5'::<lambda_1> >'::`11'::<lambda_1>,0>::CallAndMove() [A:\UnrealSource5p3\UnrealEngine\Engine\Source\Runtime\Core\Public\Async\Fundamental\TaskDelegate.h:171]
[2024.07.04-12.34.09:512][434]LogWindows: Error: [Callstack] 0x00007ff6d4548e2e Shooter.exe!LowLevelTasks::FTask::ExecuteTask() [A:\UnrealSource5p3\UnrealEngine\Engine\Source\Runtime\Core\Public\Async\Fundamental\Task.h:627]
[2024.07.04-12.34.09:512][434]LogWindows: Error: [Callstack] 0x00007ff6d4548ba7 Shooter.exe!LowLevelTasks::FScheduler::ExecuteTask() [A:\UnrealSource5p3\UnrealEngine\Engine\Source\Runtime\Core\Private\Async\Fundamental\Scheduler.cpp:149]
[2024.07.04-12.34.09:512][434]LogWindows: Error: [Callstack] 0x00007ff6d452bda2 Shooter.exe!LowLevelTasks::FScheduler::TryExecuteTaskFrom<LowLevelTasks::TLocalQueueRegistry<1024>::TLocalQueue,&LowLevelTasks::TLocalQueueRegistry<1024>::TLocalQueue::DequeueGlobal,0>() [A:\UnrealSource5p3\UnrealEngine\Engine\Source\Runtime\Core\Private\Async\Fundamental\Scheduler.cpp:350]
[2024.07.04-12.34.09:512][434]LogWindows: Error: [Callstack] 0x00007ff6d456e032 Shooter.exe!LowLevelTasks::FScheduler::WorkerMain() [A:\UnrealSource5p3\UnrealEngine\Engine\Source\Runtime\Core\Private\Async\Fundamental\Scheduler.cpp:378]
[2024.07.04-12.34.09:512][434]LogWindows: Error: [Callstack] 0x00007ff6d4538730 Shooter.exe!UE::Core::Private::Function::TFunctionRefCaller<`LowLevelTasks::FScheduler::CreateWorker'::`2'::<lambda_1>,void __cdecl(void)>::Call() [A:\UnrealSource5p3\UnrealEngine\Engine\Source\Runtime\Core\Public\Templates\Function.h:480]
[2024.07.04-12.34.09:512][434]LogWindows: Error: [Callstack] 0x00007ff6d4711d93 Shooter.exe!FThreadImpl::Run() [A:\UnrealSource5p3\UnrealEngine\Engine\Source\Runtime\Core\Private\HAL\Thread.cpp:69]
[2024.07.04-12.34.09:512][434]LogWindows: Error: [Callstack] 0x00007ff6d4b13262 Shooter.exe!FRunnableThreadWin::Run() [A:\UnrealSource5p3\UnrealEngine\Engine\Source\Runtime\Core\Private\Windows\WindowsRunnableThread.cpp:149]
[2024.07.04-12.34.09:512][434]LogWindows: Error: [Callstack] 0x00007ff6d4b06ca7 Shooter.exe!FRunnableThreadWin::GuardedRun() [A:\UnrealSource5p3\UnrealEngine\Engine\Source\Runtime\Core\Private\Windows\WindowsRunnableThread.cpp:79]
[2024.07.04-12.34.09:512][434]LogWindows: Error: [Callstack] 0x00007ffd1e927344 KERNEL32.DLL!UnknownFunction []
[2024.07.04-12.34.09:512][434]LogWindows: Error: 
[2024.07.04-12.34.09:512][434]LogWindows: Error: Crash in runnable thread Background Worker #12
[2024.07.04-12.34.09:514][434]LogExit: Executing StaticShutdownAfterError
[2024.07.04-12.34.09:516][434]LogWindows: FPlatformMisc::RequestExit(1, FRunnableThreadWin::GuardedRun.ExceptionHandler)
[2024.07.04-12.34.09:516][434]LogWindows: FPlatformMisc::RequestExitWithStatus(1, 3, FRunnableThreadWin::GuardedRun.ExceptionHandler)
[2024.07.04-12.34.09:516][434]LogCore: Engine exit requested (reason: Win RequestExit)
2

Either you are using a plugin that uses FRunnable threads that aren’t closed properly (would need patching) or you are having problems with your shader cache.

You can either try deleting the DerivedDataCache folder and if that doesn’t work then try this shader fix from steam for unreal games

Nvidia shader cache cleaning:

Open the nvidia control panel app, global settings, disable directx shader cache.
RESTART your computer cause nvidia is stuck in the 90’s still.
go to this location in explorer:
%localappdata%\NVIDIA
there’s a DX and a GL cache folder, delete them, they’ll be recreated the next time they’re needed.
Turn the global shader cache back on in the nvidia control panel app.

Additionally, Windows has a shader cache for directx as well that you should clear.
Start, type: cleanup
click the Disk Cleanup app or press enter since it’ll be at the top and selected.
click ok for the C: drive
uncheck everything but the directx shader cache (unless you know what they are) and hit the OK button.

For AMD:
Global graphics settings, open the advanced section at the bottom, click the button at the bottom to clear the shader cache.
done.

3

Thanks for the reply.
I’ll try your suggestions.
Is it normal that my DerivedDataCache folder contains only one folder (called VT), and that’s it’s empty?

4

As you can see it can either be engine data cache or local project data cache. Contents can vary depending on the complexity of the project.

5

The NVIDIA folder in AppData contains these folders…
I guess I delete the GLCache folder only?

image

6

I think you can set it to disabled in the nvidia control panel

7

But in the instructions you sent me, it states to first disable the shader cache in the control panel, and then delete the GL and DX folders in AppData.

8

It’s from a steam forum so I can’t clarify too much on the info. Only passing it on.

9

Ok, I tried your suggestions, but the crash still happens with the same error.

It’s weird because the game starts off fine for the first minute. It crashes about 1-2 minutes after gameplay starts. What’s even weirder is that sometimes there is no crash at all…So it’s intermittent. I wasn’t getting this issue in 5.2, so must me something with 5.3. I tried updating my graphics drivers, and no changes.

10

Another thing I noticed is that the crash only happens when playing through the dedicated server! It doesn’t seem to happen when I play locally…What a mess :sweat:

11

Are you overclocking your GPU in any way?

12

I don’t think so. I’m not sure how this would be done.

13

msi afterburner can overclock your gpu. Just wondering if something is causing instability in your system that could cause the crash

14

No, I’m not using an afterburner.
But that fact that the crash is happening when playing on the dedicated server, and not locally - doesn’t that mean it’s something network related?
Note that the crash logs I provided in my original post are the client logs, not the server logs. Client is the one crashing.

15

The delay in the crash could point towards a variable being garbage collected leading to a null pointer on the server. Are all of your important variables marked with the UPROPERTY macro?

Dedicated server doesn’t really run graphics either. It uses a simplified server model.

16

That’s a good point.
I’ve marked all the pointers with a UPROPERTY. Should I also mark all of the variables (like bools, floats. etc…) with UPROPERTY?

17

I know that they are critical for references and if you want to expose a variable to blueprints but I’m not sure if pure vars can get axed.

18

I wasn’t getting this issue in 5.2 So maybe some garbage collection algo changed in 5.3?
One thing I can test is setting the RHI to directX 11. If that works, then the issue is something else.

19

You could try changing a line in the target files
under the line

DefaultBuildSettings = BuildSettingsVersion.V5;

add in

IncludeOrderVersion = EngineIncludeOrderVersion.Unreal5_3;

it will enforce the correct build order for 5.3 (otherwise it can build with 5.2 that can be diffferent)

20

Another good point.
Yes will try this as well!