What are the best practices for diagnosing these in a live environment in Unreal? Since we can’t reproduce it locally, we can’t enable anything that would drastically impact the player experience.
We’ve enabled lightweight DRED, but that does not give us any actionable information. Attached is a log with the gameplay logging truncated that shows the startup/initialization and then what we get from DRED.
We see there’s a .nv-gpudmp, but no meaningful debug information there with no .nvdbg to go with it. I see from this other thread Hard to Reproduce GPU Hangs from MMU Faults - #5 by Prismatic_Lance that r.GPUCrashDebugging.Aftermath.DumpShaderDebugInfo=1 will generate that - is that true for shipping builds as well? And does that have any negative player impact? That thread was a case where QA could repro it, which is not the case for us.
Are there other cvars/settings you recommend that will help us collect more info without too much overhead or impact on the live build?
[Attachment Removed]
Steps to Reproduce
We’re unable to reproduce these GPU crashes internally. We get an occasional crash report, but nothing consistent. We’re primarily encountering this in our live builds that are in player’s hands. This is happening on recent drivers like 610.88 as well as older drivers. Sometimes it’s a hang/timeout, others it’s a DMA from a compute shader, but it’s just named compute_02 with no indication of which one. There’s no specific gpu/vendor/driver version/rhi to narrow it down.
[Attachment Removed]
Hi Tommy,
Have you gotten a chance to read through this guide by my colleague Alex Peterson: Aftermath GPU Crash Debugging Workflow | Tutorial. He breaks down the workflow to produce an Aftermath nv-gpudump file. These settings should also be available in shipping builds, as we have enabled them by default starting in 5.5. Would it be possible for you to share the cvars you use in your shipping builds related to Aftermath? Perhaps there is a setting you are currently missing that needs to be toggled on.
Best,
Tim
[Attachment Removed]
Thanks for sharing, it’s nice to have all that info in one place.
The doc specifies "Start your Editor with
-nvaftermathall -nomaterialshaderddc -gpucrashdebugging", so none of that appears to be enabled by default in shipping builds.
The crux of my question isn’t so much “what needs to be enabled to generate useful GPU crash debugging info”, but “what is appropriate in shipping builds, based on the tradeoffs of those settings”.
-nvaftermathall for example, sets CVarAftermath_TrackAll, which defaults to 0. Presumably if there were no impact, it would just be enabled all the time, not configurable. So what is the impact?
-gpucrashdebugging looks like it’s equivalent to r.GPUCrashDebugging, which also defaults to 0.
[Attachment Removed]
Yeah, that’s a valid concern. Usually, when enabling all of these extra debugging options, you would run into small perf hits at runtime due to the extra validation and larger packaged game sizes, since you would also have to ship the title with shader debug symbols. However, we have made significant progress in recent engine versions to reduce the number of shader permutations and shader bytecode sizes, and we have begun testing full Aftermath tracking (-nvaftermathall) in shipping builds for our own titles. In essence, you can enable it fully, but you should do a profiling pass on your build to measure the performance impact.
As for -gpucrashdebugging, I would not enable it in a shipping build, since the driver validation layer adds a non-negligible performance cost. However, you do not need it to run Aftermath.
[Attachment Removed]
Is there no way to just build and keep the debug symbols internally without also shipping them to players, and just associate the symbols after the fact like you would a .pdb?
[Attachment Removed]
Also, I assume the overhead is purely on the GPU, not the CPU, is that the case? We are well under budget on the GPU, so I think it would be easier to justify if the overhead was GPU only.
[Attachment Removed]
Is there no way to just build and keep the debug symbols internally without also shipping them to players, and just associate the symbols after the fact like you would a .pdb?
Yes, that is doable. I wasn’t clear before, but you should compile your shaders in full debug mode, but for shipping, you can strip the debug symbols and ship the game with the regular shaders. You can then keep the PDBs on your end. When you receive a crash dump from your players, provide Aftermath with a path to the full PDBs, and it should re-associate the shaders properly.
Also, I assume the overhead is purely on the GPU, not the CPU, is that the case?
There is a slight CPU overhead, since Aftermath has to write extra event markers into the GPU command stream before the stream is executed on the GPU. The cost is negligible enough to let Aftermath run in a shipping build. NVIDIA also confirms this on their webpage: https://docs.nvidia.com/nsight-aftermath/SDK/index.html
[Attachment Removed]