Aftermath GPU crash/hang dumps failing on standalone cooked builds

Development Rendering
, ,
1

For some reason, when GPU crashes or hangs occur in the standalone Win64 cooked build of the game, Aftermath crash dump writing always times out:

[2026.05.19-22.41.26:578][289]LogD3D12RHI: Error: Result failed

at J:\w\sod3-release-build\Engine\Source\Runtime\D3D12RHI\Private\D3D12Viewport.cpp:554

with error DXGI_ERROR_DEVICE_REMOVED with Reason: DXGI_ERROR_DEVICE_REMOVED

Viewport=0x0000021016F1B800, Num=3, Size=(1920,1080), PF=18, DXGIFormat=0x18, Fullscreen=0, AllowTearing=1

[2026.05.19-22.41.29:578][289]LogNvidiaAftermath: Warning: Timed out while waiting for Aftermath to start the GPU crash dump.

We have tried increasing the r.GPUCrashDebugging.Aftermath.DumpStartWaitTime and r.GPUCrashDebugging.Aftermath.DumpProcessWaitTime to 10 and 30 seconds respectively, but this did not help. GPU hangs and crashes in Editor builds do get written out correctly, however. It is only standalone builds which appear to be affected.

When I run the standalone build locally and run “GPUDebugCrash hang” from the console, it does write out a dump. “GPUDebugCrash pagefault” appears to do nothing - the console says that it is scheduling the crash, but the game just keeps running normally. Between this and the legitimate GPU crashes being somewhat rare and seemingly random, I have not been able to confirm the behavior on my PC with a legitimate crash yet, so this could be affecting only some users. However, it has been consistent so far in our Backtrace crash dumps that all standalone GPU crashes/device lost from all users show the timeout (or nothing, if they have a non-Nvidia GPU), and all Editor GPU crashes do have attached .nv_gpudmp data.

For at least some issues, we can just wait for it to occur in an Editor build and work from there, but it would be nice to be able to get better debug info for game crashes.

[Attachment Removed]

2

Hi there,

Sometimes Aftermath can time out if there are enough options enabled that it doesn’t have time to collect data - we occasionally see this with -nvaftermathall which enables everything.

Which of the r.GPUCrashDebugging.Aftermath options do you have enabled?

r.GPUCrashDebugging.Aftermath.Markers

r.GPUCrashDebugging.Aftermath.Callstack

r.GPUCrashDebugging.Aftermath.ResourceTracking

r.GPUCrashDebugging.Aftermath.TrackAll

r.GPUCrashDebugging.Aftermath.ShaderRegistration

r.GPUCrashDebugging.Aftermath.DumpShaderDebugInfo

“GPUDebugCrash pagefault” appears to do nothing

This is a known issue - I believe it was causing a pagefault at one point, but it gets “handled” by the driver now. You might be able to disable the driver handling that with some NvSetting.exe flag.

Do you at least get breadcrumbs in your GPU crash logs?

[Attachment Removed]

3

For what it’s worth, we’ve had breadcrumbs enabled in Test/Shipping in FN for years https://dev.epicgames.com/community/snippets/7LBe/unreal-engine-enable-breadcrumbs-in-shipping-or-test-builds

What kind of performance hit were you seeing with them enabled in shipping?

So, either we have at least 5 different GPU issues that each fired once in the last day, or the RHI breadcrumbs are not trustworthy.

We’ve seen an issue in the past where GPU crashes with random looking breadcrumbs were the result of [driver shader heap [Content removed] and usually the crashes occurred after 30+ minutes. That particular issue was Nvidia only and fixed in the 595 driver branch.

I’m passing this issue along to my colleague who is more familiar with the Aftermath implementation in case he has additional advice.

[Attachment Removed]

4

In Fortnite we use the UE 5.8 defaults, which means in we don’t have r.GPUCrashDebugging.Aftermath.ResourceTracking enabled or r.GPUCrashDebugging.Aftermath.DumpShaderDebugInfo because of the perf/mem cost. Sometimes Aftermath doesn’t produce anything and we time out waiting for DumpStartWaitTime, but we use the default value of 3 seconds.

When you tested the GPUDebugCrash hang console command - was that in a shipping build? If not, you should be able to enable that command with ALLOW_GPUDEBUGCRASH_IN_SHIPPING to test the Aftermath dumps locally in a shipping build.

[Attachment Removed]

5

I’m not sure if it’s related, but I’ve been working with Nvidia chasing a GPU TDR that is occurring in vanilla UE in a default open world project left running for 36+ hours, sometimes with an Aftermath dump and sometimes not, and the dump reveals nothing useful, just says “Automated analysis could not determine a cause for the GPU error.” without any active warps etc. Unfortunately, the repro seems to have stopped occurring for both us and Nvidia as of last week as we’ve been trying to get a watchdog kernel dump to determine if the TDR is really from UE or another program or Windows. However the error is Device 0 Removed: DXGI_ERROR_DEVICE_HUNG not DXGI_ERROR_DEVICE_REMOVED, so might be completely unrelated. Have you reached out to Nvidia with your repro?

[Attachment Removed]

6

The only flag we have active in the current build is r.GPUCrashDebugging.Aftermath.ResourceTracking, although I just enabled DumpShaderDebugInfo as well, so that should be in tomorrow’s builds.

We do sometimes get RHI breadcrumbs and sometimes not. I’m looking right now through logs reported back from our active invite-only alpha test, so all should be the exact same build. So far, only one log file has a full set of RHI breadcrumbs, and in that log, there is nothing at all reported about Aftermath initialization, neither success nor failure. That particular run was on an RTX A2000, which is not a gaming card to my understanding, so perhaps the drivers for that GPU don’t include Aftermath support? All other logs I’ve looked at so far have logging of Aftermath initializing successfully but timing out on the dump output, and none of them have any breadcrumbs. It’s a small sample size, but so far it’s 8 logs with Aftermath and no breadcrumbs, and 1 log with no Aftermath and yes breadcrumbs.

It’s possible we have some CVar or other option set incorrectly in a config file somewhere, which would explain why we don’t seem to have this issue in Editor builds. I also just integrated a couple small code changes from one of our partner studios that relate to Aftermath dump writing, so it’s possible that those changes may help.

[Attachment Removed]

7

It’s looking like the lack of RHI breadcrumbs in all those logs may be because they are from a Shipping build. We’re outputting RHI breadcrumbs from Dev and Test builds even though Aftermath continues timing out, but so far, every log I’ve looked at has zero overlap in what they say were active on the graphics and compute queues at the time of the failure.

So, either we have at least 5 different GPU issues that each fired once in the last day, or the RHI breadcrumbs are not trustworthy. Unfortunately, the GPU issues are not common/reproducible enough to try running with RHI/RDG serialization enabled just to repro the crash, and we also can’t set those options for the entire studio to cast a wider net because of the negative effects on productivity.

Best hope I think is still to get the Aftermath dumps to stop timing out. The changes we made yesterday do not seem to have helped, but today I enabled r.GPUCrashDebugging.Aftermath.Markers and DumpShaderDebugInfo for everyone, and bumped the DumpStartWaitTime again, from 10 seconds to 15. We’ll see if that helps.

[Attachment Removed]

8

I don’t know that we’ve seen any performance hit with enabling the markers/breadcrumbs in Shipping at all. I think we’ve just never turned it on for some reason. I’m going to make sure it’s enabled for our next Shipping alpha build that we push out.

These reports are coming with NVidia driver versions all the way up to 610.47, so it’s probably not that. :frowning:

[Attachment Removed]

9

I tested the console command in a Development build. I don’t think any of the real problems that we are having here are related to build configurations.

I have one of our QA testers running with r.GPUCrashDebugging and r.GPUCrashDebugging.Aftermath.Enabled both set to zero and Aftermath Monitor running in the background instead. He gets shader debug info writing out to that folder, so it’s definitely hooking itself into the driver, but he’s still getting no .nv-gpudmp files being written out when it crashes.

Logs show DXGI_ERROR_DEVICE_REMOVED with GPU breadcrumbs pointing consistently at ray tracing work as active, but no dump is written. I’m not sure if there’s a way you can check Aftermath Monitor logs or Windows events to see whether it’s the same timeout issue, but it seems at least possible that the problem is “something we’re doing is preventing Aftermath from even starting to write the dump” in general as opposed to something about the way we’re handling the Aftermath callbacks within UE.

Might need to separately contact Nvidia support directly to see if they can help.

Log output from crash yesterday:

[2026.06.10-23.33.39:836][ 69]LogD3D12RHI: Error: hr failed 
 at J:\w\sod3-main-new-build\Engine\Source\Runtime\D3D12RHI\Private\D3D12Resources.cpp:1105 
 with error DXGI_ERROR_DEVICE_REMOVED with Reason: DXGI_ERROR_DEVICE_REMOVED
 
[2026.06.10-23.33.39:837][ 69]LogD3D12RHI: Error: GPU crash detected:
	- Device 0 Removed: DXGI_ERROR_DEVICE_REMOVED
 
[2026.06.10-23.33.39:837][ 69]LogRHI: Error: Active GPU breadcrumbs:
 
	Device 0, Pipeline Graphics: (In: 0x91bedeb9, Out: 0x91beddfa)
		(ID: 0x91bedce3) [     Active]	Frame 474067
		(ID: 0x91bedd84) [     Active]		SceneRender - ViewFamilies
		(ID: 0x91bedeb8) [     Active]			RenderGraphExecute - /ViewFamilies
		(ID: 0x91bedd8a) [     Active]				Scene
		(ID: 0x91beddf7) [   Finished]					CompositionBeforeBasePass
		(ID: 0x91beddf8) [   Finished]						DeferredDecals BeforeBasePass
		(ID: 0x91beddf9) [   Finished]							Decals (Relevant: 446, Total: 2480)
		(ID: 0x91beddfa) [     Active]					RayTracingDynamicGeometry
		(ID: 0x91bedeb9) [     Active]						RayTracingDynamicGeometryUpdate
		(ID: 0x91bede04) [Not Started]					BasePass
		(ID: 0x91bede06) [Not Started]						NonNaniteBasePass
		(ID: 0x91beded5) [Not Started]							ParallelDraw (Index: 0, Num: 1)
		(ID: 0x91bede07) [Not Started]						NaniteBasePass
		(ID: 0x91bede08) [Not Started]							Nanite::BasePass
		(ID: 0x91bede09) [Not Started]								Nanite::ShadeBinning
		(ID: 0x91bede0c) [Not Started]						EditorPrimitives
		(ID: 0x91beded6) [Not Started]						ParallelDraw (Index: 0, Num: 1)
		(ID: 0x91bede0e) [Not Started]					StochasticLighting
		(ID: 0x91bede0f) [Not Started]					CopyStencilToLightingChannels
		(ID: 0x91bede13) [Not Started]					DiffuseIndirectAndAO
		(ID: 0x91bede14) [Not Started]						LumenScreenProbeGather
		(ID: 0x91bede15) [Not Started]							UpdateRadianceCaches
		(ID: 0x91bede19) [Not Started]					VirtualShadowMapMarkPages
		(ID: 0x91bede1a) [Not Started]						ShadowDepths
		(ID: 0x91bede1b) [Not Started]							FVirtualShadowMapArray::BeginMarkPages
		(ID: 0x91bede1c) [Not Started]					ShadowDepths
		(ID: 0x91bede1d) [Not Started]						BuildRenderingCommandsDeferred(Culling=On)
		(ID: 0x91bede1e) [Not Started]						FVirtualShadowMapArray::BuildPageAllocation
		(ID: 0x91bede1f) [Not Started]							InitializePhysicalPages
		(ID: 0x91bede20) [Not Started]						RenderVirtualShadowMaps(Nanite)
		(ID: 0x91bede21) [Not Started]							Shadow Depths - VSM (Nanite)
		(ID: 0x91bede24) [Not Started]								Nanite::DrawGeometry
		(ID: 0x91bede25) [Not Started]									MainPass
		(ID: 0x91bede26) [Not Started]										InstanceCulling
		(ID: 0x91bede27) [Not Started]										NodeAndClusterCull
		(ID: 0x91bede28) [Not Started]										CalculateSafeRasterizerArgs
		(ID: 0x91bede29) [Not Started]									BuildPreviousOccluderHZB(VSM)
		(ID: 0x91bede2a) [Not Started]									PostPass
		(ID: 0x91bede2b) [Not Started]										InstanceCulling
		(ID: 0x91bede2c) [Not Started]										NodeAndClusterCull
		(ID: 0x91bede2d) [Not Started]										CalculateSafeRasterizerArgs
 
	Device 0, Pipeline AsyncCompute: (In: 0x91beddfb, Out: 0x91beddfb)
		(ID: 0x91bedce3) [     Active]	Frame 474067
		(ID: 0x91bedd84) [     Active]		SceneRender - ViewFamilies
		(ID: 0x91bedeb8) [     Active]			RenderGraphExecute - /ViewFamilies
		(ID: 0x91bedd8a) [     Active]				Scene
		(ID: 0x91beddcc) [   Finished]					LumenSceneUpdate: 45 card captures 0.006M texels
		(ID: 0x91beddfa) [   Finished]					RayTracingDynamicGeometry
		(ID: 0x91bedeb9) [   Finished]						RayTracingDynamicGeometryUpdate
		(ID: 0x91bedec1) [   Finished]						Build
		(ID: 0x91beddfb) [     Active]					RayTracingScene
 
[2026.06.10-23.33.39:837][ 69]LogD3D12RHI: Error: DRED: No breadcrumb head found.
[2026.06.10-23.33.39:837][ 69]LogD3D12RHI: Error: DRED: No PageFault data.
[2026.06.10-23.33.39:837][ 69]LogD3D12RHI: Error: Video Memory Stats from frame ID 474068:
[2026.06.10-23.33.39:837][ 69]LogD3D12RHI: Error: 	Local Budget:	11242.00 MB
[2026.06.10-23.33.39:837][ 69]LogD3D12RHI: Error: 	Local Used:	7023.15 MB
[2026.06.10-23.33.39:837][ 69]LogD3D12RHI: Error: 	System Budget:	63562.82 MB
[2026.06.10-23.33.39:837][ 69]LogD3D12RHI: Error: 	System Used:	 699.84 MB
[2026.06.10-23.33.39:837][ 69]LogD3D12RHI: Error: Shader diagnostic messages and asserts:
 
	Device: 0, Queue 3D:
		No shader diagnostics found for this queue.
 
	Device: 0, Queue Copy:
		No shader diagnostics found for this queue.
 
	Device: 0, Queue Compute:
		No shader diagnostics found for this queue.

[Attachment Removed]

10

He gets shader debug info writing out to that folder, so it’s definitely hooking itself into the driver, but he’s still getting no .nv-gpudmp files being written out when it crashes.

Ray tracing GPU crashes are often in internal shaders, but I’ve seen .nv-gpudmp files get produced with those and when you open them in Aftermath it list the crash as being inside an internal shader.

If you have a repro, toggling r.raytracing.geometry.* types till it goes away is one common way to narrow down the source. The other recommendation is to test with r.RayTracing.PersistentSBT=0 and/or D3D12.ResidencyManagement=0 to determine if the issue is related to SBT management.

Example ray tracing GPU [Content removed]

In this, possibly related GPU crash, the licensee mentioned they weren’t getting Aftermath dumps and might try disabling DLSS/Reflex and anything that may be introducing a shim between the renderer, but I’m not sure whether it worked for them: [Content removed]

[Attachment Removed]

11

We’re trying turning various things off with raytracing, including r.Raytracing.AsyncBuild 0. We can try these as well.

I’m increasingly suspicious that the reason we’re not getting any Aftermath dumps from either the engine itself or Aftermath Monitor is that the cause of the DXGI_ERROR_DEVICE_REMOVED isn’t from the GPU side in the first place, but something on the CPU side. We’re trying runs with the D3D12 debug layer enabled to try to catch anything that comes up through that, and there have been a couple warnings/errors that have shown up so far. Made a couple changes already to try to eliminate those, but it’s too early to tell yet whether that solved our “GPU Crash” issues.

If it is a CPU side thing that’s failing somewhere within the RHI thread interfacing with D3D12, I’m not sure why we wouldn’t be getting more useful RHI thread breadcrumbs though. In almost every case, the DEVICE_REMOVED is being reported just on either Present() or a create resources call, and both the DRED breadcrumbs in the log and the Breadcrumbs_RHIThread_0.txt are empty.

[Attachment Removed]

12

I saw the thing about disabling the shims as well. I’m not sure how I can determine whether we’ve successfully disabled all possible shims, but I know there’s at least the Xbox Game Bar, possibly Steam overlay, and also whatever FPS overlay stuff the GPU manufacturer software includes.

[Attachment Removed]