GameThread timed out waiting for RenderThread after 120.00 seconds

Development Rendering
, ,
1

Hello, one of our players sent us their crash dump from a freeze. Here is the crash message from the CrashContext:

<ErrorMessage>GameThread timed out waiting for RenderThread after 120.00 seconds:

0x00007ff936aa1d44 ntdll.dll!UnknownFunction

0x00007ff934317eef KERNELBASE.dll!UnknownFunction

0x00007ff6a8165787 darkhours-Win64-Shipping.exe!UnknownFunction

0x00007ff6a7f6eb9f darkhours-Win64-Shipping.exe!UnknownFunction

0x00007ff6a7f7597d darkhours-Win64-Shipping.exe!UnknownFunction

0x00007ff6a7f75e5e darkhours-Win64-Shipping.exe!UnknownFunction

0x00007ff6a9eba7ad darkhours-Win64-Shipping.exe!UnknownFunction

0x00007ff6a9eb630a darkhours-Win64-Shipping.exe!UnknownFunction

Check log for full callstack.</ErrorMessage>

And in the logs we have this:

LogD3D12RHI: Warning: GPU timeout: A payload (0x0000023AC982AD20) on the [0x0000023AC6FA7F40, 3D] queue has not completed after 5.000001 seconds.

LogD3D12RHI: Warning: GPU timeout: A payload (0x0000023B08BC0100) on the [0x0000023AC6FA84C0, Compute] queue has not completed after 5.001336 seconds.

We’ve been seeing unexplained GPU crashes in our game for several weeks now. Do you have any idea or any lead to help us debug this crash?

The logs and CrashContext are attached.

2

Hello,

Thank you for reaching out.

I’ve been assigned this issue, and we will be looking into this crash for you.

3

Hello,

This crash looks like a GPU Timeout - the GPU likely entered an infinite loop or taken an extremely long time to complete a task.

The user seems to have suspended TDR (Timeout Detection and Recovery) that would normally reset the GPU in this case, leading to UE detecting and crashing due to the unresponsive Render Thread. Normal behavior would have been a GPU crash after 5 seconds, in line with the LogD3D12RHI Warning messages.

With these types of crashes, the callstack and crash dumps do not provide useful information.

Are you able to reproduce the crash yourself? If so, the following GPU debugging options might help you get more information to debug the crash.

-d3ddebug -rdgdebug -rdgimmediate -forcerhibypass -gpucrashdebugging -gpuvalidationThis talk from Unreal Fest 2023 goes into more details on GPU crash debugging. The GPU Breadcrumbs feature it discusses can help you narrow down what pass is causing the issue.

Please let us know if this helps.