How Best to Diagnose Sporadic PSO/Shader Compilation Crashes

anonymous-edc · August 13, 2025, 12:57am

Members of our team sporadically encounter PSO/shader compilation crashes in packaged builds. Are there any recommendations for diagnosing these issues?

[2025.08.13-00.31.02:404][413]LogRHI: Display: Encountered a new graphics PSO: 2207490882
[2025.08.13-00.31.02:404][413]LogRHI: Display: Encountered a new compute PSO: 1219710009
[2025.08.13-00.31.02:404][413]LogRHI: Error: Failed to create compute pipeline with hash 394CB3480054C7A1F034E4AA10B2B956231597B1.
[2025.08.13-00.31.02:404][413]LogRHI: Error: Shader: <unknown>
[2025.08.13-00.31.02:404][413]LogOutputDevice: Warning: 
 
Script Stack (0 frames) :
 
[2025.08.13-00.31.02:404][413]LogWindows: Error: appError called: Fatal error: [File:C:\WS\TF\built-in-0\TFPkg-main\TF\Engine\Source\Runtime\RHI\Private\PipelineStateCache.cpp] [Line: 546] 
Shader compilation failures are Fatal.

anonymous-edc · August 13, 2025, 12:57am

Steps to Reproduce

TimStullich · August 13, 2025, 11:37am

Hi Bill,

Have you tried running your packaged game with r.PSOPrecache.Validation=2? This option should give you more detailed log information on which PSOs are missed and sent off to be compiled. Once we can find out which compute PSO is failing to compile, we can look into patching this crash. Please let me know if you have any further questions.

anonymous-edc · August 19, 2025, 11:59pm

Thank you [mention removed], that’s helpful! In our next PT that we encounter this, I’ll work with the impacted people to see what we can uncover using this.

TimStullich · August 20, 2025, 7:44am

You are welcome. Once you have some more information from the logs, feel free to contact us again, and we can take another look.

anonymous-edc · September 3, 2025, 3:41pm

Thank you [mention removed], we did get the issue again on one of our dev machines, with r.PSOPrecache.Validation=2 enabled.

This is tough to diagnose for a couple of reasons - it seems to occur sporadically, and more for some folks than others. I’m unaware of what in UE’s shader compilation pipeline could be failing, and why. Any info is greatly appreciated.

Here’s an excerpt from the log:

[2025.09.03-15.36.23:583][309]LogD3D12RHI: Error: Failed to create pipeline state with combined hash D4FA420B5205DEEE, error 8007000e.

[2025.09.03-15.36.23:584][309]LogD3D12RHI: Warning: Failed to create graphics PSO with combined hash 0xD4FA420B5205DEEE:

AlphaToCoverageEnable = 0

IndependentBlendEnable = 1

RenderTarget[0] = { 1, 0, 0x5, 0x6, 0x1, 0x1, 0x6, 0x1, 0x0, 0xF }

SampleMask = 0xFFFFFFFF

FillMode = 3

CullMode = 1

FrontCounterClockwise = 1

DepthBias = 0

DepthBiasClamp = 0.000000

SlopeScaledDepthBias = 0.000000

DepthClipEnable = 1

MultisampleEnable = 1

AntialiasedLineEnable = 0

ForcedSampleCount = 0

ConservativeRaster = 0

DepthEnable = 1

DepthWriteMask = 0

DepthFunc = 7

StencilEnable = 1

StencilReadMask = 0xFF

StencilWriteMask = 0xF4

FrontFace = { 1, 1, 1, 8 }

BackFace = { 1, 1, 1, 8 }

InputLayout.NumElements = 2

InputLayout[0] = { “ATTRIBUTE”, 0, 0x6, 0, 0, 0x0, 0 }

InputLayout[1] = { “ATTRIBUTE”, 13, 0x2A, 1, 0, 0x1, 1 }

IBStripCutValue = 0x0

PrimitiveTopologyType = 0x3

NumRenderTargets = 1

RTVFormats[0] = 0xA

DSVFormat = 0x14

SampleDesc = { 1, 0 }

NodeMask = 0x1

Flags = 0x0

[2025.09.03-15.36.23:584][309]LogRHI: Error: Failed to create graphics pipeline, hashes: Vertex: 87E76DC6C67FE97A700E258C7ECF050EBEBC934E, Pixel: D74836589AC6BEEB1C5B1BB853EF276B3A8D0E64, Pipeline: DB924629FD7F67C4ED4C1AEC30F2B67011942697.

[2025.09.03-15.36.23:584][309]LogRHI: Error: Vertex: <unknown>

[2025.09.03-15.36.23:584][309]LogRHI: Error: Pixel: <unknown>

[2025.09.03-15.36.23:584][309]LogRHI: Error: Render Targets: (8)

[2025.09.03-15.36.23:584][309]LogRHI: Error: 0xa

[2025.09.03-15.36.23:584][309]LogRHI: Error: 0x0

[2025.09.03-15.36.23:584][309]LogRHI: Error: Depth Stencil Format:

[2025.09.03-15.36.23:584][309]LogRHI: Error: 0xb

[2025.09.03-15.36.26:424][480]LogRHI: Display: Encountered a new graphics PSO: 903167674

[2025.09.03-15.36.26:424][480]LogRHI: Display: Encountered a new graphics PSO: 3974658396

[2025.09.03-15.36.26:424][480]LogRHI: Error: Failed to create graphics pipeline, hashes: Vertex: 3833C0DE62E36A294D4DA0D3455E644DEF91B891, Pixel: 4C0CF36163AC09C2B1E6E3947A44771D568B7515, Pipeline: B80AD0635304F6E96DF90D6D41598B163B1F7A7A.

[2025.09.03-15.36.26:424][480]LogRHI: Error: Vertex: <unknown>

[2025.09.03-15.36.26:424][480]LogRHI: Error: Pixel: <unknown>

[2025.09.03-15.36.26:424][480]LogRHI: Error: Render Targets: (8)

[2025.09.03-15.36.26:424][480]LogRHI: Error: 0xa

[2025.09.03-15.36.26:424][480]LogRHI: Error: 0x0

[2025.09.03-15.36.26:424][480]LogRHI: Error: Depth Stencil Format:

[2025.09.03-15.36.26:424][480]LogRHI: Error: 0xb

[2025.09.03-15.36.26:424][480]LogOutputDevice: Warning:

Script Stack (0 frames) :

[2025.09.03-15.36.26:424][480]LogWindows: Error: appError called: Fatal error: [File:C:\WS\TF\built-in-0\TFPkg-main\TF\Engine\Source\Runtime\RHI\Private\PipelineStateCache.cpp] [Line: 528]

Shader compilation failures are Fatal.

[2025.09.03-15.36.26:424][480]LogWindows: Windows GetLastError: The operation completed successfully. (0)

[2025.09.03-15.37.20:517][480]LogWwiseMonitor: Error: Voice Starvation

[2025.09.03-15.37.20:517][480]LogRHI: Error: Breadcrumbs ‘Parallel’

- ParallelDraw (Index: 0, Num: 1)

- RenderTranslucency

- Translucency

- Scene

- FRDGBuilder::Execute

- Frame 15479

anonymous-edc · September 3, 2025, 5:25pm

Callstack as well:

 	[External Code]	
>	[Inline Frame] InvincibleVS.exe!ReportAssert(const wchar_t *) Line 1844	C++
 	InvincibleVS.exe!FWindowsErrorOutputDevice::Serialize(const wchar_t * Msg, ELogVerbosity::Type Verbosity, const FName & Category) Line 84	C++
 	InvincibleVS.exe!FOutputDevice::LogfImpl(const wchar_t * Fmt, ...) Line 81	C++
 	InvincibleVS.exe!UE::Logging::Private::BasicFatalLog(const FLogCategoryBase & Category, const UE::Logging::Private::FStaticBasicLogRecord * Log, ...) Line 1106	C++
 	InvincibleVS.exe!HandlePipelineCreationFailure(const FGraphicsPipelineStateInitializer & Init) Line 530	C++
 	InvincibleVS.exe!FCompilePipelineStateTask::CompilePSO(const FGraphicsPipelineStateInitializer::EPSOPrecacheCompileType * OptionalPriorityOverride) Line 3019	C++
 	[Inline Frame] InvincibleVS.exe!FCompilePipelineStateTask::DoTask(ENamedThreads::Type) Line 2896	C++
 	InvincibleVS.exe!TGraphTask<FCompilePipelineStateTask>::ExecuteTask() Line 635	C++
 	[Inline Frame] InvincibleVS.exe!UE::Trace::FChannel::operator|(const UE::Trace::FChannel &) Line 29	C++
 	[Inline Frame] InvincibleVS.exe!TaskTrace::FTaskTimingEventScope::{ctor}(unsigned __int64) Line 301	C++
 	InvincibleVS.exe!UE::Tasks::Private::FTaskBase::TryExecuteTask() Line 505	C++
 	[Inline Frame] InvincibleVS.exe!UE::Tasks::Private::FTaskBase::Init::__l2::<lambda>() Line 185	C++
 	[Inline Frame] InvincibleVS.exe!LowLevelTasks::FTask::Init::__l13::<lambda>(const bool) Line 499	C++
 	[Inline Frame] InvincibleVS.exe!Invoke(LowLevelTasks::FTask::Init::__l13::void <lambda>(const TArray<FString,TSizedDefaultAllocator<32>> &, FOutputDevice &) &) Line 47	C++
 	[Inline Frame] InvincibleVS.exe!LowLevelTasks::TTaskDelegate<LowLevelTasks::FTask * __cdecl(bool),48>::TTaskDelegateImpl<`LowLevelTasks::FTask::Init<`UE::Tasks::Private::FTaskBase::Init'::`2'::void <lambda>(const TArray<FString,TSizedDefaultAllocator<32>> &, FOutputDevice &)>'::`13'::void <lambda>(const TArray<FString,TSizedDefaultAllocator<32>> &, FOutputDevice &),0>::Call(void *) Line 162	C++
 	InvincibleVS.exe!LowLevelTasks::TTaskDelegate<LowLevelTasks::FTask * __cdecl(bool),48>::TTaskDelegateImpl<`LowLevelTasks::FTask::Init<`UE::Tasks::Private::FTaskBase::Init'::`2'::void <lambda>(const TArray<FString,TSizedDefaultAllocator<32>> &, FOutputDevice &)>'::`13'::void <lambda>(const TArray<FString,TSizedDefaultAllocator<32>> &, FOutputDevice &),0>::CallAndMove(LowLevelTasks::TTaskDelegate<LowLevelTasks::FTask * __cdecl(bool),48> & Destination, void * InlineData, unsigned int DestInlineSize, bool <Params_0>) Line 171	C++
 	[Inline Frame] InvincibleVS.exe!LowLevelTasks::TTaskDelegate<LowLevelTasks::FTask * __cdecl(bool),48>::CallAndMove(LowLevelTasks::TTaskDelegate<LowLevelTasks::FTask * __cdecl(bool),48> &) Line 308	C++
 	InvincibleVS.exe!LowLevelTasks::FTask::ExecuteTask() Line 627	C++
 	[Inline Frame] InvincibleVS.exe!FCpuProfilerTrace::FEventScope::{ctor}(unsigned int &) Line 153	C++
 	InvincibleVS.exe!LowLevelTasks::FScheduler::ExecuteTask(LowLevelTasks::FTask * InTask) Line 244	C++
 	[Inline Frame] InvincibleVS.exe!LowLevelTasks::FScheduler::TryExecuteTaskFrom(LowLevelTasks::Private::FWaitEvent *) Line 457	C++
 	InvincibleVS.exe!LowLevelTasks::FScheduler::WorkerLoop(LowLevelTasks::Private::FWaitEvent * WorkerEvent, LowLevelTasks::Private::TLocalQueueRegistry<1024,1024>::TLocalQueue * WorkerLocalQueue, unsigned int WaitCycles, bool bPermitBackgroundWork) Line 514	C++
 	[Inline Frame] InvincibleVS.exe!LowLevelTasks::FScheduler::WorkerMain(LowLevelTasks::Private::FWaitEvent * WorkerEvent, LowLevelTasks::Private::TLocalQueueRegistry<1024,1024>::TLocalQueue * WorkerLocalQueue, unsigned int WaitCycles, bool bPermitBackgroundWork) Line 571	C++
 	InvincibleVS.exe!LowLevelTasks::FScheduler::CreateWorker::__l2::<lambda>() Line 75	C++
 	[Inline Frame] InvincibleVS.exe!UE::Core::Private::Function::TFunctionRefBase<UE::Core::Private::Function::TFunctionStorage<1>,void __cdecl(void)>::operator()() Line 470	C++

TimStullich · September 5, 2025, 8:51am

Hi Bill,

Thanks for the detailed info. It does seem like there is a compilation issue, but the logs don’t mention exactly why this is happening, except for the error code on the first line:

[2025.09.03-15.36.23:583][309]LogD3D12RHI: Error: Failed to create pipeline state with combined hash D4FA420B5205DEEE, error 8007000eFrom what I have seen, this tends to indicate that you have run out of system memory. This reminded me that we had an issue in 5.5 with PSO compilation, where we did not properly release compiled PSOs after compilation. We fixed this in 5.6, but we advised people to apply a patch manually since we could not hot-fix this in 5.5. I wrote a tech note about it here: https://dev.epicgames.com/community/learning/knowledge\-base/DBOL/tech\-note\-fix\-for\-pso\-management\-issue\-on\-nvidia\-hardware\-in\-unreal\-engine\-5\-5\. Have you already integrated this patch into your own build? It could be worth trying this fix to see if this gets rid of your crashes. Let me know what you think

anonymous-edc · September 5, 2025, 3:39pm

Ooh, I wasn’t aware, this is super helpful. I’ll see about integrating this patch asap. We’re on 5.5.4. Thanks Tim!

TimStullich · September 8, 2025, 8:24am

You are welcome! I hope the patch fixes things up for you, but if you still get these crashes post-integration, reach out again, and we can take another look.