Random GPU crashes and timeouts on Windows with D3D12 mostly on app startup

Development Rendering
, ,
21

One of my colleagues had a suggestion that could be helpful. We introduced a change that would dump active breadcrumbs when the GPU payload was taking too long - this is from 5.7 after the breadcrumbs refactor, but you may be able to take the idea behind it and implement it for the 5.5 breadcrumbs:

CL#44248799 (f4ce60) D3D12RHI: dump the active breadcrumbs tree when a GPU payload takes too long

- This will help investigate crash reports where the full GPU crash handling was not triggered, but the interrupt thread detected the timeout of a payload.

- Currently only the timeout message is printed to the log. Now we’ll also get the active breadcrumbs from each queue.

UE 5.7 CL#45349090 (679703)

Another suggestion was to see whether the crash goes away when D3D12 residency management is disabled:

-dpcvars=D3D12.ResidencyManagement=0

We also recently fixed an issue where RHI tasks could run out of order, but this may not affect your engine version:

CL#49856210 (d7982c) Fix RHI task pipe chaining

- RHI task pipes must enforce FIFO ordering of tasks that are enqueued, however this wasn’t always happening. Tasks on the RHIThread pipe could run out-of-order.

- The SubmitToGPU task closes the RHIThread pipe so it can hold a reference the LastSubmit event. The next enqueued task on the RHIThread pipe would create a new TaskGraph task (because FTaskPipe::Current is null), but the previous chain’s completion event was not inserted as a prerequisite.

- This could lead to chains on the RHI thread running in reverse order. This may have been causing deleted RHI resource crashes or other RHI / GPU crashes.

UE 5.8 CL#52603973 (3a484e)

And another thought that occurred to me is maybe you’re playing some video during these first few frames or have some custom splash? If so, try disabling any video playback to see if the crash goes away.

[Attachment Removed]

22

Hello Alex,

Thank you for your suggestions. I’m going to try them and get back to you with updates.

Regarding the increased timeout, yes, I probably checked the wrong log file. When I ran it one more time, I got the same error: “GameThread timed out waiting for RenderThread”.

[Attachment Removed]

23

Hello,

Sorry for the delay.

I’ve returned to this topic and have some updates/findings:

I tried the things you proposed in your last message:

  • “D3D12RHI: dump the active breadcrumbs tree when a GPU payload takes too long” - this can’t be easily applied to UE 5.5.4 because of merge conflicts, so I wasn’t able to test it.
  • “-dpcvars=D3D12.ResidencyManagement=0” - I checked this, and it doesn’t make any difference.
  • “Fix RHI task pipe chaining” - I checked this as well, and it doesn’t make any difference.
  • I also double checked that there are no any intro playing when crash occurs

I was also finally able to capture a utrace from a run (Shipping build) with the crash. Please find attached two traces from the same device and scenario: one with the crash and one without it.

The main finding so far is that using the -clearPSODriverCache argument significantly increases the crash rate.

I made the following runs, in this order:

  • Default: 0/100 crashes
  • clearPSODriverCache: 18/100 crashes on 3/4 devices
  • clearPSODriverCache: 19/100 crashes on 3/4 devices
  • Default: 3/100 crashes on 3/4 devices, but all crashes happened in the first runs
  • Default: 0/100 crashes
  • clearPSODriverCache: 19/100 crashes on 3/4 devices

Devices with crashes:

  • NVIDIA GeForce GTX 1050
  • NVIDIA GeForce RTX 3060
  • AMD Radeon™ Graphics

Device with no crashes:

  • Radeon RX 580 Series

Please share any ideas you might have based on these findings.

[Attachment Removed]

24

Thanks for providing the extra traces.

What is BzGpuPresent - is that a wrapper?

We may be looking in the wrong place, but I’d like to rule out PSO Precaching as the source of these issues if possible by trying a few things.

-dpcvars=r.PSOPrecaching=0

That will turn of precaching entirely.

Alternately, test with low and no PSOPrecompilePool threads using r.PSOPrecache.PrecompileThreadPoolSize and r.PSOPrecache.PrecompileThreadPoolPercentOfHardwareThreads to see if PSO compile concurrency is part of the issue.

Also test using less PSOPrecaching with

r.PSOPrecache.GlobalShaders=0

r.PSOPrecache.Components=0

r.PSOPrecache.PostProcessingMaterial=0

This will cause hitching but I’d like to know if we can rule out PSO compilation.

There are some CLs that are fixes in this area after 5.5 that may be needed:

CL#38493633 (6c8cd6) Fixed PSO stat registration crash caused by separate synchronization scopes

CL#40357307 (bad09d) Fixed pipeline state cache cleanup on non-rendering threads/tasks

- Fixes rare assert on GPU crash handling

CL#41715756 (0fa6c7) Fix unmatched EndCPU calls for RHI breadcrumbs used in PSO compilation

- The PSO compilation tasks call WalkIn/WalkOut to push the breadcrumb that was at the top of the RHICmdList’s stack at the point compilation is kicked. However, since PSO compil…

CL#46756962 (cac4a0) Use actual compute shader hash instead of pointer hash for compute PSO precache caching

CL#47843022 (4dae45) PR #13754: Fix PSO precaching crash when referencing Breadcrumb data from RHI command list after memory is freed

[Attachment Removed]

25

Hello Alex,

Yes, BzGpuPresent is our custom scoped named event. On Windows, it is located in FRHICommandEndDrawingViewport::Execute.

I also thought about PSO Precaching, because in the traces from the Development build that I was able to get recently, there is a lot of PSO Precaching happening right before the crash. However, after disabling it, I am still able to reproduce the same crash. I was also able to reproduce it with -onethread, but the crash rate is much lower with this parameter.

I am attaching the traces with this crash from the Development build. There are the following cases: default / no precaching / no precaching + onethread.

As I understand it, there is no need to test other r.PSOPrecache CVars if the crash reproduces even with r.PSOPrecaching 0.

However, I am going to try the CLs you shared to see if they make any difference.

Please let me know if you have any other ideas about what else it could be.

I’m thinking it might be some other PSO compilation triggered by something other than Precaching.

[Attachment Removed]

26

As I understand it, there is no need to test other r.PSOPrecache CVars if the crash reproduces even with r.PSOPrecaching 0.

Thanks for testing that, seeing it crash without precaching and with onethread means it’s at least not solely precaching.

That long Present had me thinking - there were a few issues around 5.5 that related to HWND re-use on windows and Present. Do you have these CLs?

CL#38629989 (d3f049) Relaxed presentation check with additional logging

CL#46533494 (60243c) Flush the GPU before creating a SwapChain. This is an attempt to prevent issues from HWNDs being reused by the Windows runtime.

I didn’t see the errors about present in the logs provided, but the PSO compilation could be putting extra stress on the driver and in an automated build farm if you’re stopping/starting these tests often, potentially it’s more likely to get the HWND reuse issue.

[Attachment Removed]

27

Hello Alex,

Thanks for sharing your thoughts.

I’ve applied the CLs you provided, but the crash is still happening.

We’ve started working on the transition to UE 5.7, so the next step will probably be to check whether the same crash can be reproduced there. I’ll let you know once we run those tests.

[Attachment Removed]

28

OK, good to know you’re upgrading which should hopefully help. Another thing that occurred to me is that potentially there are some driver shader optimizations happening during startup. You might try turning those off as a test of whether that is having an affect on the crash. The other thing I wanted to call out since Present is showing up early on is that there could be a crash in something that is hooking Present like Streamline.

It hooks Present like this (from https://github.com/NVIDIA-RTX/Streamline/blob/019994e18d256a3e92347888deb527feb7f58bc0/source/core/sl.interposer/dxgi/dxgiSwapchain.cpp#L280 ), and might be related to recently reported crashes such as this one: [Content removed]

Also, in 5.8 we added experimental support for running Present on the submission thread (r.D3D12.PresentOnSubmissionThread). When the present happens on the RHI thread, we have to flush the submission thread to be able to take ownership of the swapchain / graphics queue to do the present call. This implicitly syncs the RHIT and submission threads. When Present happens on the submission thread, that flush doesn’t happen, so we aren’t implicitly syncing anymore.

The change is made in the following changelists:

CL#46685858 (91ca64) - Allow DX12 Present to be called on the Submission Thread (r.D3D12.PresentOnSubmissionThread)

- Refactor Present calls to use FRHIPresentArg in order have the right game frame number correlating with the Present counter

- Add more instrumentation …

CL#48289837 (7a476d) Move debug information after the syncronization in order to avoid data race between GetBackBuffer_RHIThread and PresentOnSubmissionThread/SetBackBufferIndex_RHIThread

It’s currently off because it had issues with multi GPU and possibly OpenXR and other third party plugins / modules. There’s was also reported hitching on some consoles, possibly due to the submission thread running ahead and weird feedback loops happening - still under investigation.

[Attachment Removed]

29

Hello,

Based on the data you were able to share, there appears to be more than one crash occurring:

  • Timeout in LumenReflections on Graphics Queue
  • Timeout in GPUSkinCache on Async Compute Queue
  • Unknown page fault in a fragment shader
  • Unknown page fault, possibly use before creation is finished, in a compute shader

This ticket discusses a very similar Lumen Reflections crash, and it appears to be related to HWRT.

[Content removed]

This ticket seems to discuss a potentially similar GPUSkinCache crash. Keep in mind this ticket is not in English:

[Content removed]

CL: 38905831 between 5.5 and 5.6 fixes async compute overlaps with Lumen Reflections.

CL: 41760130 also fixes a Lumen Reflections crash

Since the issues you are reproducing most frequently are the timeouts, can you try reproducing them with Lumen Reflections disabled, and separately, GPU Skin Cache disabled?

[Attachment Removed]

30

Hello,

Both your logs seem to show that the hang happened in two different passes.

One thing that could be helpful to try is to increase the submission timeout CVar, “r.D3D12.SubmissionTimeout”, to a larger value to see if the pass that is causing the timeout eventually ends.

I noticed that you do not have “-d3ddebug” enabled in your command line. Can you please do your reproductions with it as well? From your previous notes, it seemed that gpuvalidation was the only one that was failing.

[Attachment Removed]

31

Hello,

For the crash, can you try deleting the “Engine\Binaries\Win64\D3D12” folder? There might be a mismatch between the D3D12Core.dll and d3d12SDKLayers.dll libraries that re-fetching them could fix the crash with -d3ddebug and -gpuvalidation.

Please let us know if this helps.

[Attachment Removed]

32

Hello,

There were further changes to this solution in Unreal Engine 5.6. These can be found in CL: 39658041 on //UE5/Main.

The changes include the ability to control the max scratch size via a CVar “r.RHICmd.MaxAccelerationStructureBuildScratchSize”. It is possible that lowering this value for specific cards (including the NVidia GeForce RTX 2060 SUPER used in the log “Blitz (35) (aftermath)”)

Please let us know if this helps.

[Attachment Removed]