Aftermath GPU Crash Dump Issues in UE 5.4 — Files Not Generated/Not Copied to Saved/Crashes

Development Rendering
, ,
1

Hello,

We are using Unreal Engine 5.4. Since Aftermath isn’t fully integrated in this version, we’ve implemented custom fixes via Perforce changelists to enable it. However, we’re encountering two persistent issues with GPU crash dumps:

  1. Dump File Creation: Aftermath does not consistently generate GPU crash dump files (

*.nvdbg1. ,

*.nv-gpudmp1. ) when a GPU crash occurs.
2. File Copying Failure: Even when dump files are generated in the working directory, they frequently fail to copy to the

Saved/Crashes1. folder.

We’ve added code to handle the copying of these files upon crash detection. Our investigation suggests the issue might relate to crash-handling threads not initializing reliably, preventing the copy logic from executing.

Could you please help us understand:

  1. What are the common reasons why Aftermath might fail to generate the crash dump files in UE 5.4?
  2. More critically, why might the dump files fail to copy to

Saved/Crashes1. even when they exist? Are there known limitations with crash-handling thread spawning in this scenario?

Additional question: What is the recommended method to enable silent, automatic crash report submission (bypassing the user interaction window)?

Any insights, debugging suggestions, or known workarounds would be greatly appreciated.

Thank you for your assistance.

2

Perforce CLs I used is from this thread

[Content removed]

3

Hi Andrei.

“Dump File Creation: Aftermath does not consistently generate GPU crash dump files”

There’s quite a few reasons why Aftermath may fail to generate dump files, some on the driver side and some on Unreal’s side.

On crashes you should see a fairly verbose dump of the Aftermath state, such as engine states, page faults (if any), running shaders, etc. The log should say a little bit more about what went wrong, do you see anything like the below in the log files?

LogNvidiaAftermath: Timed out while waiting for Aftermath to start the GPU crash dump

We allow the driver crash handling 3 seconds (r.GPUCrashDebugging.Aftermath.DumpStartWaitTime) before skipping Aftermath dumps all-together, and 10 seconds (r.GPUCrashDebugging.Aftermath.DumpProcessWaitTime) to finish processing it.

“We’ve added code to handle the copying of these files upon crash detection. Our investigation suggests the issue might relate to crash-handling threads not initializing reliably, preventing the copy logic from executing.”

Yeah that seems like a bug, I’ll check that out.

“What are the common reasons why Aftermath might fail to generate the crash dump files in UE 5.4?”

The driver reset wasn’t initiated by the NV drivers (e.g., OS / DX middle-layer), or was of a fault type that Aftermath doesn’t support. Or, crash handling wasn’t initiated in a timely manner from the driver (out of our control)

“More critically, why might the dump files fail to copy to”

This “should” work, with exception that it only copies over the aftermath dumps that are specific to the current session. Again, I’ll see what might be happening here.

“Additional question: What is the recommended method to enable silent, automatic crash report submission (bypassing the user interaction window)?”

Let me get back to you on this one.

4

“we’ve implemented custom fixes via Perforce changelists”

To be sure, could you please list the CLs?

5

With regards to

“What is the recommended method to enable silent, automatic crash report submission (bypassing the user interaction window)?”

From the crash reporter client you have “-ImplicitSend” and “-HideSubmitAndRestart”. Game side, you may configure this from ini’s with “bImplicitSend” in the “CrashReportClient” section, e.g. DefaultEngine.ini,:

[CrashReportClient] bImplicitSend=True

6

Hi, just checking if that CL was helpful, or if there were any integration issues?

7

Hello, thanks for your answer!

“We allow the driver crash handling 3 seconds (r.GPUCrashDebugging.Aftermath.DumpStartWaitTime) before skipping Aftermath dumps all-together, and 10 seconds (r.GPUCrashDebugging.Aftermath.DumpProcessWaitTime) to finish processing it.”

Hm… I’ll think about it, possibly it can help a little bit.

“To be sure, could you please list the CLs?”

Sure:

Shader symbols on Windows:

  • 40501660 - generating shader symbols on 5.4 code base (without large refactoring and DDC changes);
  • 39004424 - Multithreading fix during shader PDBs exporting;
  • 39142585- Minor fix;

Aftermath fixes:

  • 34200808 - Add shader PDB callback;
  • 39543979 - Copying crash files to Saved/Crashes;
8

I would also consider 39634273, fixes some issues around threading.

9

Hey Miguel, we’re also investigating this.

We can’t access 39634273.

We can access all 5 CLs which Andrei mentioned, though.

Pavel

10

Agreed with Pavel, I also can’t access to this CL

11

Can you see 39634316?

12

Yes, thank you!

13

Thank you, Miguel!

I’ll check it out and come back with an answer

14

Thank you, it’s work.

Also, I keep looking at 39634316 CL. Hope, it can help us

15

Hi,

yes, there are integration issues. We don’t have large RHI refactoring 30973133 in 5.4 version so I couldn’t quickly understand the idea how use this CL with old code base.

We will switch to version 5.5 in 6 months so I’ll be back to this problem later if it’s needed.

Thank you for helping :slight_smile:

P.S. Though, I upgraded aftermath SDK to last version. Possibly it will fix some issues with poor dump.