Hello Alex,
Yes, the test has finished, and it appears that Lumen Reflections did not significantly impact GPU crashes.
This time, we also have the driver version and date included in the crash statistics. The variability is quite wide, and it shows that even the most recent drivers still experience many GPU crashes.
The most unusual and at the same time the most common case occurs on app startup. We also see this in our regular autotests in the form of GPU hangs.
I ran an experiment where I progressively disabled different parts of the rendering that occur before the 5-7th frame, when the crash happens. In the end, almost nothing was running on the GPU, yet GPU hangs were still occurring.
I disabled the following:
- FDeferredShadingSceneRenderer::Render
- FSlateRHIRenderer::DrawWindow_RenderThread
- FScene::Update
- USE_BUFFER_POOL_ALLOCATOR & USE_TEXTURE_POOL_ALLOCATOR
- FD3D12DynamicRHI::RHIUpdateTexture3D
- FMediaTextureResource::InitRHI & FMediaTextureResource::Render
So from these RHI Breadcrumbs:
2026-04-23 02:37:19: [ 7]LogD3D12RHI: Warning: GPU timeout: A payload (0x000002606A426A60) on the [0x0000026069276840, 3D] queue has not completed after 5.000472 seconds.
2026-04-23 02:37:19: [ 7]LogD3D12RHI: Error: GPU crash detected:
2026-04-23 02:37:19: - Device 0 OK (no device removed reason)
2026-04-23 02:37:19:
2026-04-23 02:37:19: [ 7]LogRHI: Error: Active GPU breadcrumbs:
2026-04-23 02:37:19:
2026-04-23 02:37:19: Device 0, Pipeline Graphics: (In: 0x80000290, Out: 0x8000028f)
2026-04-23 02:37:19: (ID: 0x80000256) [ Active] Frame 4
2026-04-23 02:37:19: (ID: 0x800002a2) [ Active] FRDGBuilder::Execute
2026-04-23 02:37:19: (ID: 0x8000025e) [ Active] Scene
2026-04-23 02:37:19: (ID: 0x8000028e) [ Active] PostProcessing
2026-04-23 02:37:19: (ID: 0x8000028f) [ Active] ThirdParty FDLSSSceneViewFamilyUpscaler 2560x1440 -> 2560x1440
2026-04-23 02:37:19: (ID: 0x80000290) [ Active] DLSS
2026-04-23 02:37:19: (ID: 0x800002a9) [Not Started] FRDGBuilder::Execute
2026-04-23 02:37:19: (ID: 0x800002a6) [Not Started] SlateUI Title = Blitz (64-bit Development PCD3D_SM6)
2026-04-23 02:37:19: (ID: 0x800002a7) [Not Started] DrawDebugCanvas
2026-04-23 02:37:19: (ID: 0x800002a8) [Not Started] CanvasFlush
2026-04-23 02:37:19: (ID: 0x800002ae) [Not Started] BufferPoolCopyOps
2026-04-23 02:37:19: (ID: 0x800002af) [Not Started] TexturePoolCopyOps
2026-04-23 02:37:19: (ID: 0x800002ac) [Not Started] Frame 5
2026-04-23 02:37:19: (ID: 0x800002ad) [Not Started] WorldTick
2026-04-23 02:37:19: (ID: 0x800002b0) [Not Started] SendAllEndOfFrameUpdates
2026-04-23 02:37:19: (ID: 0x800002b1) [Not Started] RayTracingGeometry
2026-04-23 02:37:19: (ID: 0x800002f9) [Not Started] FRDGBuilder::Execute
2026-04-23 02:37:19: (ID: 0x800002b5) [Not Started] Scene
2026-04-23 02:37:19: (ID: 0x800002b7) [Not Started] FXSystemPreRender
2026-04-23 02:37:19: (ID: 0x800002fa) [Not Started] GPUParticles_PreRender
2026-04-23 02:37:19: (ID: 0x800002b8) [Not Started] GPUSceneUpdate
2026-04-23 02:37:19: (ID: 0x800002ba) [Not Started] BuildRenderingCommandsDeferred(Culling=On)
2026-04-23 02:37:19: (ID: 0x800002bb) [Not Started] PrePass DDM_AllOpaqueNoVelocity (Forced by DBuffer)
2026-04-23 02:37:19: (ID: 0x800002bd) [Not Started] HZB
2026-04-23 02:37:19: (ID: 0x800002be) [Not Started] BuildHZB(ViewId=0)
2026-04-23 02:37:19: (ID: 0x800002bf) [Not Started] BuildHZB
2026-04-23 02:37:19: (ID: 0x800002c0) [Not Started] SortLights
2026-04-23 02:37:19: (ID: 0x800002c1) [Not Started] ComputeLightGrid
2026-04-23 02:37:19: (ID: 0x800002c2) [Not Started] CullLights 40x23x32 NumLights 0 NumCaptures 0
2026-04-23 02:37:19: (ID: 0x800002c3) [Not Started] LightFunctionAtlasGeneration
2026-04-23 02:37:19: (ID: 0x800002c5) [Not Started] BasePass
2026-04-23 02:37:19: (ID: 0x800002c6) [Not Started] EditorPrimitives
2026-04-23 02:37:19: (ID: 0x800002c7) [Not Started] ShadowDepths
2026-04-23 02:37:19: (ID: 0x800002c8) [Not Started] BuildRenderingCommandsDeferred(Culling=On)
2026-04-23 02:37:19: (ID: 0x800002ca) [Not Started] RenderDeferredLighting
2026-04-23 02:37:19: (ID: 0x800002cc) [Not Started] InitTranslucencyLightingVolumeTextures
2026-04-23 02:37:19: (ID: 0x800002d4) [Not Started] FilterTranslucentVolume 64x64x64 Cascades:2
2026-04-23 02:37:19: (ID: 0x800002d6) [Not Started] ReflectionIndirect
2026-04-23 02:37:19: (ID: 0x800002d7) [Not Started] ScreenSpaceReflections(Quality=2)
2026-04-23 02:37:19: (ID: 0x800002d8) [Not Started] ScreenSpaceReflections
2026-04-23 02:37:19: (ID: 0x800002d9) [Not Started] ReflectionEnvironment
2026-04-23 02:37:19: (ID: 0x800002dc) [Not Started] PostRenderOpsFX
2026-04-23 02:37:19: (ID: 0x800002dd) [Not Started] FXSystemPostRenderOpaque
2026-04-23 02:37:19: (ID: 0x800002fb) [Not Started] GPUParticles_PostRenderOpaque
2026-04-23 02:37:19: (ID: 0x800002fc) [Not Started] NiagaraUpdateCollisionGroupsMap
2026-04-23 02:37:19: (ID: 0x800002de) [Not Started] Translucency
2026-04-23 02:37:19: (ID: 0x800002e0) [Not Started] RenderTranslucency
2026-04-23 02:37:19: (ID: 0x800002e5) [Not Started] PostProcessing
2026-04-23 02:37:19: (ID: 0x800002e6) [Not Started] ThirdParty FDLSSSceneViewFamilyUpscaler 2560x1440 -> 2560x1440
2026-04-23 02:37:19: (ID: 0x800002e7) [Not Started] DLSS
2026-04-23 02:37:19: (ID: 0x80000300) [Not Started] FRDGBuilder::Execute
2026-04-23 02:37:19: (ID: 0x800002fd) [Not Started] SlateUI Title = Blitz (64-bit Development PCD3D_SM6)
2026-04-23 02:37:19: (ID: 0x800002fe) [Not Started] DrawDebugCanvas
2026-04-23 02:37:19: (ID: 0x800002ff) [Not Started] CanvasFlush
2026-04-23 02:37:19:
2026-04-23 02:37:19: Device 0, Pipeline AsyncCompute: (In: 0x800002b5, Out: 0x80000256)
2026-04-23 02:37:19: (ID: 0x800002ac) [ Active] Frame 5
2026-04-23 02:37:19: (ID: 0x800002f9) [ Active] FRDGBuilder::Execute
2026-04-23 02:37:19: (ID: 0x800002b5) [ Active] Scene
I got these:
[2026.04.20-12.23.20:281][ 6]LogD3D12RHI: Warning: GPU timeout: A payload (0x0000020E5BB38E00) on the [0x0000020E5BDE6840, 3D] queue has not completed after 5.000000 seconds.
[2026.04.20-12.23.20:281][ 6]LogD3D12RHI: Error: GPU crash detected:
- Device 0 OK (no device removed reason)
[2026.04.20-12.23.20:281][ 6]LogRHI: Error: Active GPU breadcrumbs:
Device 0, Pipeline Graphics: (In: 0x8000002a, Out: 0x8000002a)
(ID: 0x80000026) [ Active] Frame 5
(ID: 0x8000002a) [ Active] FRDGBuilder::Execute
(ID: 0x8000002b) [Not Started] FRDGBuilder::Execute
(ID: 0x8000002c) [Not Started] Frame 6
(ID: 0x8000002d) [Not Started] WorldTick
(ID: 0x8000002e) [Not Started] SendAllEndOfFrameUpdates
(ID: 0x8000002f) [Not Started] FRDGBuilder::Execute
(ID: 0x80000030) [Not Started] FRDGBuilder::Execute
Device 0, Pipeline AsyncCompute: (In: 0x8000002c, Out: 0x80000026)
No breadcrumb nodes found for this queue.
At some point, I also added additional logs and call stack dumps to the RHI Breadcrumbs macro, and after that the crash rate dropped significantly, almost 0 in my runs. This could indicate that the issue is not directly related to GPU work itself, and may instead involve a CPU-side concurrency problem.
What do you think about this and what would you suggest as the next steps to investigate it further?
Also during my experiments I was able to capture this GPU Hang with d3debug, gpuvalidation, gpucrashdebugging, dred, but there was no additional data, I guess because it is not detected as a GPU crash by the driver. Anyway, I will try enabling lightweight DRED for the next public test to gather more information.
[Attachment Removed]