My project keeps getting a GPU crash error when trying to play or render a level sequence. I am doing this in a world partitioned level with an 8x8km landscape and few actors. It is quite annoying, since it is not the only source of GPU crashes me and my team are getting: they are pretty common, and happen in the most random situations. Can anyone help us by sharing a tip to solve/bypass this?
Hey there @GonzaloBernalNFT! Could you post the crash logs, standard logs, and the hardware/drivers? I’ve been seeing an issue with level sequences and a couple of specific combinations of things happening and working on figuring out the cause/adding to the bug report. Thanks!
I have the same issue with one specific level sequence. It’s a fairly big one with a lot of characters, 400+ tracks, and 4500+ items. What’s the limit of an level sequence? It’s very consistent with this file, sometimes it crashes instantly, sometimes it takes a couple of minutes, @SupportiveEntity here’s the crash log:
----- a lot of previous LogHavokCloth logs like below emitted -----
[2022.10.28-19.55.21:311][911]LogHavokCloth: [SECRET_FILE][SECRET_CLOTH]: Cloth Item does not support collision with static bodies.
[2022.10.28-19.55.21:311][911]LogHavokCloth: [SECRET_FILE][SECRET_CLOTH]: Cloth Item does not support collision with static bodies.
[2022.10.28-19.55.21:339][911]LogHavokCloth: [SECRET_FILE][SECRET_CLOTH]: Cloth Item does not support collision with static bodies.
[2022.10.28-19.55.21:339][911]LogHavokCloth: [SECRET_FILE][SECRET_CLOTH]: Cloth Item does not support collision with static bodies.
[2022.10.28-19.55.21:504][911]LogAkAudio: Error: Audio command queue is full, blocking caller. Reduce number of calls to sound engine or boost command queue memory.
[2022.10.28-19.55.22:217][911]LogTemp: Warning: FBoundSceneComponentBatch::ResolveObjects: No bound objects returned for FGuid:
[2022.10.28-19.55.22:217][911]LogTemp: Warning: FBoundSceneComponentBatch::ResolveObjects: No bound objects returned for FGuid:
[2022.10.28-19.55.22:217][911]LogTemp: Warning: FBoundSceneComponentBatch::ResolveObjects: No bound objects returned for FGuid:
[2022.10.28-19.55.22:217][911]LogTemp: Warning: FBoundSceneComponentBatch::ResolveObjects: No bound objects returned for FGuid:
[2022.10.28-19.55.22:386][911]LogFileHelpers: Editor autosave (incl. external actors) for 'MAP' took 14.292
[2022.10.28-19.55.22:387][911]LogFileHelpers: Editor autosave (incl. sublevels & external actors) for all levels took 14.292
[2022.10.28-19.55.40:439][917]LogD3D12RHI: Error: ID3D12Device::RemoveDevice: Device removal has been triggered for the following reason (DXGI_ERROR_DEVICE_HUNG: The Device took an unreasonable amount of time to execute its commands, or the hardware crashed/hung. As a result, the TDR (Timeout Detection and Recovery) mechanism has been triggered. The current Device Context was executing commands when the hang occurred. The application may want to respawn and fallback to less aggressive use of the display hardware).
[2022.10.28-19.55.40:446][917]LogD3D12RHI: Error: ID3D12Device::RemoveDevice: Device removal has been triggered for the following reason (DXGI_ERROR_DEVICE_HUNG: The Device took an unreasonable amount of time to execute its commands, or the hardware crashed/hung. As a result, the TDR (Timeout Detection and Recovery) mechanism has been triggered. The current Device Context was executing commands when the hang occurred. The application may want to respawn and fallback to less aggressive use of the display hardware).
[2022.10.28-19.55.40:446][917]LogD3D12RHI: Error: ID3D12CommandQueue::ExecuteCommandLists: Using ResourceBarrier on Command List (0x0000019E50575220:'FD3D12CommandListData (GPU 0)'): Before state (0x0: D3D12_RESOURCE_STATE_[COMMON|PRESENT]) of resource (0x0000019F89A58000:'BackBuffer0') (subresource: 0) specified by transition barrier does not match with the state (0x4: D3D12_RESOURCE_STATE_RENDER_TARGET) specified in the previous call to ResourceBarrier
[2022.10.28-19.55.40:446][917]LogD3D12RHI: Error: ID3D12Device::RemoveDevice: Device removal has been triggered for the following reason (DXGI_ERROR_DEVICE_HUNG: The Device took an unreasonable amount of time to execute its commands, or the hardware crashed/hung. As a result, the TDR (Timeout Detection and Recovery) mechanism has been triggered. The current Device Context was executing commands when the hang occurred. The application may want to respawn and fallback to less aggressive use of the display hardware).
[2022.10.28-19.55.40:446][917]LogD3D12RHI: Error: ID3D12CommandQueue::ExecuteCommandLists: Using ResourceBarrier on Command List (0x0000019E50575220:'FD3D12CommandListData (GPU 0)'): Before state (0x0: D3D12_RESOURCE_STATE_[COMMON|PRESENT]) of resource (0x0000019F89A58000:'BackBuffer0') (subresource: 0) specified by transition barrier does not match with the state (0x4: D3D12_RESOURCE_STATE_RENDER_TARGET) specified in the previous call to ResourceBarrier
[2022.10.28-19.55.40:446][917]LogD3D12RHI: Error: ID3D12CommandQueue::ExecuteCommandLists: Command lists must be closed before execution.
[2022.10.28-19.55.40:446][917]LogD3D12RHI: Error: pResource->Map(Subresource, pReadRange, reinterpret_cast<void**>(&pData)) failed
at P:\P4\WORKSPACE\Engine\Source\Runtime\D3D12RHI\Private\D3D12RHIPrivate.h:1186
with error DXGI_ERROR_DEVICE_REMOVED with Reason: DXGI_ERROR_DEVICE_HUNG
[2022.10.28-19.55.40:446][917]LogD3D12RHI: Error: [GPUBreadCrumb] Last tracked GPU operations:
[2022.10.28-19.55.40:446][917]LogD3D12RHI: Error: [GPUBreadCrumb] 3D Queue 0 - No Data
[2022.10.28-19.55.40:446][917]LogD3D12RHI: Error: [GPUBreadCrumb] Compute Queue 0 - No Data
[2022.10.28-19.55.40:447][917]LogD3D12RHI: Error: [GPUBreadCrumb] Copy Queue 0 - No Data
[2022.10.28-19.55.40:447][917]LogD3D12RHI: Error: DRED: No PageFault data.
[2022.10.28-19.55.40:447][917]LogD3D12RHI: Error: ID3D12Device::RemoveDevice: Device removal has been triggered for the following reason (DXGI_ERROR_DEVICE_HUNG: The Device took an unreasonable amount of time to execute its commands, or the hardware crashed/hung. As a result, the TDR (Timeout Detection and Recovery) mechanism has been triggered. The current Device Context was executing commands when the hang occurred. The application may want to respawn and fallback to less aggressive use of the display hardware).
[2022.10.28-19.55.40:447][917]LogD3D12RHI: Error: ID3D12CommandQueue::ExecuteCommandLists: Using ResourceBarrier on Command List (0x0000019E50575220:'FD3D12CommandListData (GPU 0)'): Before state (0x0: D3D12_RESOURCE_STATE_[COMMON|PRESENT]) of resource (0x0000019F89A58000:'BackBuffer0') (subresource: 0) specified by transition barrier does not match with the state (0x4: D3D12_RESOURCE_STATE_RENDER_TARGET) specified in the previous call to ResourceBarrier
[2022.10.28-19.55.40:447][917]LogD3D12RHI: Error: ID3D12CommandQueue::ExecuteCommandLists: Command lists must be closed before execution.
[2022.10.28-19.55.40:447][917]LogD3D12RHI: Error: ID3D12CommandQueue::ExecuteCommandLists: Using ResourceBarrier on Command List (0x0000019E50575220:'FD3D12CommandListData (GPU 0)'): Before state (0x0: D3D12_RESOURCE_STATE_[COMMON|PRESENT]) of resource (0x0000019F89A58000:'BackBuffer0') (subresource: 0) specified by transition barrier does not match with the state (0x4: D3D12_RESOURCE_STATE_RENDER_TARGET) specified in the previous call to ResourceBarrier
[2022.10.28-19.55.46:022][917]LogD3D12RHI: Error: ID3D12Device::RemoveDevice: Device removal has been triggered for the following reason (DXGI_ERROR_DEVICE_HUNG: The Device took an unreasonable amount of time to execute its commands, or the hardware crashed/hung. As a result, the TDR (Timeout Detection and Recovery) mechanism has been triggered. The current Device Context was executing commands when the hang occurred. The application may want to respawn and fallback to less aggressive use of the display hardware).
[2022.10.28-19.55.46:023][917]LogD3D12RHI: Error: ID3D12CommandQueue::ExecuteCommandLists: Using ResourceBarrier on Command List (0x0000019E50575220:'FD3D12CommandListData (GPU 0)'): Before state (0x0: D3D12_RESOURCE_STATE_[COMMON|PRESENT]) of resource (0x0000019F89A58000:'BackBuffer0') (subresource: 0) specified by transition barrier does not match with the state (0x4: D3D12_RESOURCE_STATE_RENDER_TARGET) specified in the previous call to ResourceBarrier
[2022.10.28-19.55.46:023][917]LogD3D12RHI: Error: ID3D12CommandQueue::ExecuteCommandLists: Command lists must be closed before execution.
[2022.10.28-19.55.46:023][917]LogD3D12RHI: Error: ID3D12CommandQueue::ExecuteCommandLists: Using ResourceBarrier on Command List (0x0000019E50575220:'FD3D12CommandListData (GPU 0)'): Before state (0x0: D3D12_RESOURCE_STATE_[COMMON|PRESENT]) of resource (0x0000019F89A58000:'BackBuffer0') (subresource: 0) specified by transition barrier does not match with the state (0x4: D3D12_RESOURCE_STATE_RENDER_TARGET) specified in the previous call to ResourceBarrier
[2022.10.28-19.55.46:023][917]LogD3D12RHI: Error: ID3D12CommandQueue::ExecuteCommandLists: Using Draw on Command List (0x0000019E4C370BD0:'FD3D12CommandListData (GPU 0)'): Resource state (0x0: D3D12_RESOURCE_STATE_[COMMON|PRESENT]) of resource (0x0000019F652EC390:'BackBuffer0') (subresource: 0) is invalid for use as a render target. Expected State Bits (all): 0x4: D3D12_RESOURCE_STATE_RENDER_TARGET, Actual State: 0x0: D3D12_RESOURCE_STATE_[COMMON|PRESENT], Missing State: 0x4: D3D12_RESOURCE_STATE_RENDER_TARGET.
[2022.10.28-19.55.46:023][917]LogD3D12RHI: Error: ID3D12Device::RemoveDevice: Device removal has been triggered for the following reason (DXGI_ERROR_DEVICE_HUNG: The Device took an unreasonable amount of time to execute its commands, or the hardware crashed/hung. As a result, the TDR (Timeout Detection and Recovery) mechanism has been triggered. The current Device Context was executing commands when the hang occurred. The application may want to respawn and fallback to less aggressive use of the display hardware).
[2022.10.28-19.55.46:023][917]LogD3D12RHI: Error: ID3D12CommandQueue::ExecuteCommandLists: Using ResourceBarrier on Command List (0x0000019E50575220:'FD3D12CommandListData (GPU 0)'): Before state (0x0: D3D12_RESOURCE_STATE_[COMMON|PRESENT]) of resource (0x0000019F89A58000:'BackBuffer0') (subresource: 0) specified by transition barrier does not match with the state (0x4: D3D12_RESOURCE_STATE_RENDER_TARGET) specified in the previous call to ResourceBarrier
[2022.10.28-19.55.46:023][917]LogD3D12RHI: Error: ID3D12CommandQueue::ExecuteCommandLists: Command lists must be closed before execution.
[2022.10.28-19.55.46:023][917]LogD3D12RHI: Error: ID3D12CommandQueue::ExecuteCommandLists: Using ResourceBarrier on Command List (0x0000019E50575220:'FD3D12CommandListData (GPU 0)'): Before state (0x0: D3D12_RESOURCE_STATE_[COMMON|PRESENT]) of resource (0x0000019F89A58000:'BackBuffer0') (subresource: 0) specified by transition barrier does not match with the state (0x4: D3D12_RESOURCE_STATE_RENDER_TARGET) specified in the previous call to ResourceBarrier
[2022.10.28-19.55.46:023][917]LogD3D12RHI: Error: ID3D12CommandQueue::ExecuteCommandLists: Using Draw on Command List (0x0000019E4C370BD0:'FD3D12CommandListData (GPU 0)'): Resource state (0x0: D3D12_RESOURCE_STATE_[COMMON|PRESENT]) of resource (0x0000019F652EC390:'BackBuffer0') (subresource: 0) is invalid for use as a render target. Expected State Bits (all): 0x4: D3D12_RESOURCE_STATE_RENDER_TARGET, Actual State: 0x0: D3D12_RESOURCE_STATE_[COMMON|PRESENT], Missing State: 0x4: D3D12_RESOURCE_STATE_RENDER_TARGET.
[2022.10.28-19.55.46:023][917]LogD3D12RHI: Error: ID3D12CommandQueue::ExecuteCommandLists: Using ResourceBarrier on Command List (0x0000019E4C370BD0:'FD3D12CommandListData (GPU 0)'): Before state (0x4: D3D12_RESOURCE_STATE_RENDER_TARGET) of resource (0x0000019F652EC390:'BackBuffer0') (subresource: 0) specified by transition barrier does not match with the state (0x0: D3D12_RESOURCE_STATE_[COMMON|PRESENT]) specified in the previous call to ResourceBarrier
[2022.10.28-19.55.46:023][917]LogD3D12RHI: Error: pResource->Map(Subresource, pReadRange, reinterpret_cast<void**>(&pData)) failed
at P:\P4\WORKSPACE\Engine\Source\Runtime\D3D12RHI\Private\D3D12RHIPrivate.h:1186
with error DXGI_ERROR_DEVICE_REMOVED with Reason: DXGI_ERROR_DEVICE_HUNG
[2022.10.28-19.55.46:342][917]LogWindows: Error: Error reentered: pResource->Map(Subresource, pReadRange, reinterpret_cast<void**>(&pData)) failed
at P:\P4\WORKSPACE\Engine\Source\Runtime\D3D12RHI\Private\D3D12RHIPrivate.h:1186
with error DXGI_ERROR_DEVICE_REMOVED with Reason: DXGI_ERROR_DEVICE_HUNG
- Fatal error: [File:P:/P4/WORKSPACE/Engine/Source/Runtime/D3D12RHI/Private/D3D12Util.cpp] [Line: 691]
pResource->Map(Subresource, pReadRange, reinterpret_cast<void**>(&pData)) failed
at P:\P4\WORKSPACE\Engine\Source\Runtime\D3D12RHI\Private\D3D12RHIPrivate.h:1186
with error DXGI_ERROR_DEVICE_REMOVED with Reason: DXGI_ERROR_DEVICE_HUNG
[2022.10.28-19.55.46:342][917]LogHAL: Callstack:
0x00007fface898686 UE4Editor-FATCrashHandler.dll!WindowsCrashHandler::PlatformNotifyPreCrash() [P:\P4\WORKSPACE\Engine\Source\Runtime\FATCrashPipeline\FATCrashHandler\Private\Src\Windows\WindowsCrashHandler.cpp:267]
0x00007fface89543b UE4Editor-FATCrashHandler.dll!CrashHandler::NotifyPreCrash() [P:\P4\WORKSPACE\Engine\Source\Runtime\FATCrashPipeline\FATCrashHandler\Private\Src\CrashHandler.cpp:250]
0x00007ffb09c5b3fe UE4Editor-Core.dll!ReportGPUCrash() [P:\P4\WORKSPACE\Engine\Source\Runtime\Core\Private\Windows\WindowsPlatformCrashContext.cpp:1702]
0x00007ffb09c5db26 UE4Editor-Core.dll!FWindowsErrorOutputDevice::Serialize() [P:\P4\WORKSPACE\Engine\Source\Runtime\Core\Private\Windows\WindowsErrorOutputDevice.cpp:78]
0x00007ffb09977250 UE4Editor-Core.dll!FOutputDevice::LogfImpl() [P:\P4\WORKSPACE\Engine\Source\Runtime\Core\Private\Misc\OutputDevice.cpp:61]
0x00007ffac6e50587 UE4Editor-D3D12RHI.dll!D3D12RHI::VerifyD3D12Result() [P:\P4\WORKSPACE\Engine\Source\Runtime\D3D12RHI\Private\D3D12Util.cpp:691]
0x00007ffac6de9dda UE4Editor-D3D12RHI.dll!FD3D12Device::GetQueryData() [P:\P4\WORKSPACE\Engine\Source\Runtime\D3D12RHI\Private\D3D12Query.cpp:196]
0x00007ffac6df162c UE4Editor-D3D12RHI.dll!FD3D12DynamicRHI::RHIGetRenderQueryResult() [P:\P4\WORKSPACE\Engine\Source\Runtime\D3D12RHI\Private\D3D12Query.cpp:111]
0x00007ffb2e87b3d9 UE4Editor-RenderCore.dll!FRealtimeGPUProfilerFrame::UpdateStats() [P:\P4\WORKSPACE\Engine\Source\Runtime\RenderCore\Private\ProfilingDebugging\RealtimeGPUProfiler.cpp:443]
0x00007ffb2e82d68a UE4Editor-RenderCore.dll!FRealtimeGPUProfiler::EndFrame() [P:\P4\WORKSPACE\Engine\Source\Runtime\RenderCore\Private\ProfilingDebugging\RealtimeGPUProfiler.cpp:911]
0x00007ff6b295d7cb UE4Editor.exe!EndFrameRenderThread() [P:\P4\WORKSPACE\Engine\Source\Runtime\Launch\Private\LaunchEngineLoop.cpp:4697]
0x00007ff6b295b4ad UE4Editor.exe!TEnqueueUniqueRenderCommandType<`FEngineLoop::Tick'::`43'::EndFrameName,<lambda_075c4013d054921635cd939835247b8e> >::DoTask() [P:\P4\WORKSPACE\Engine\Source\Runtime\RenderCore\Public\RenderingThread.h:184]
0x00007ff6b295e65c UE4Editor.exe!TGraphTask<TEnqueueUniqueRenderCommandType<`FEngineLoop::Tick'::`43'::EndFrameName,<lambda_075c4013d054921635cd939835247b8e> > >::ExecuteTask() [P:\P4\WORKSPACE\Engine\Source\Runtime\Core\Public\Async\TaskGraphInterfaces.h:905]
0x00007ffb096dff0a UE4Editor-Core.dll!FNamedTaskThread::ProcessTasksNamedThread() [P:\P4\WORKSPACE\Engine\Source\Runtime\Core\Private\Async\TaskGraph.cpp:714]
0x00007ffb096e037e UE4Editor-Core.dll!FNamedTaskThread::ProcessTasksUntilQuit() [P:\P4\WORKSPACE\Engine\Source\Runtime\Core\Private\Async\TaskGraph.cpp:606]
0x00007ffb2e8ffee7 UE4Editor-RenderCore.dll!RenderingThreadMain() [P:\P4\WORKSPACE\Engine\Source\Runtime\RenderCore\Private\RenderingThread.cpp:373]
0x00007ffb2e9047f4 UE4Editor-RenderCore.dll!FRenderingThread::Run() [P:\P4\WORKSPACE\Engine\Source\Runtime\RenderCore\Private\RenderingThread.cpp:509]
0x00007ffb09c7ac19 UE4Editor-Core.dll!FRunnableThreadWin::Run() [P:\P4\WORKSPACE\Engine\Source\Runtime\Core\Private\Windows\WindowsRunnableThread.cpp:86]
0x00007ffb09c717fa UE4Editor-Core.dll!FRunnableThreadWin::GuardedRun() [P:\P4\WORKSPACE\Engine\Source\Runtime\Core\Private\Windows\WindowsRunnableThread.cpp:35]
0x00007ffb5c667034 KERNEL32.DLL!UnknownFunction []
0x00007ffb5c9226a1 ntdll.dll!UnknownFunction []
My hardware and software:
i9-10980XE
RTX3080 with driver version: 31.0.15.1694
UE 4.27
however, this doesn’t only happen to me.
Judging by the log it looks like either resources ran out and it went over and caused cascading failures, or that it was close to maxing out and remained that way for so long it timed out.
You can get a little extra time before the timeout by following this tutorial but be VERY CAREFUL when playing in the registry:
I’ve tried it, and no crash anymore, but I am more insterested in how this happened. Does it mean it’s normal or adviced to extend the delays during development? Also, even I did add the two keys, the asset would be too slow to process. Any advice? @SupportiveEntity
So it’s kind of a side-effect of graphics in engine getting back to a point where we can’t just throw more resources at it. Some operations are going to take immutably more time than standard application, so this just helps make sure that buffer is wide enough never to clip your head as you come out of the gate for legitimate actions. It’s still necessary to have the delay timeout however as it’s a failsafe.
Another thing I’m noticing is that some GPU’s built in software with overlays and other goodies are causing similar crashes for other users. Disabling Shadowplay/Afterburner etc. should net you a tiny fraction more overhead and maybe a bit more stability. It’s also recommended using studio drivers instead of game ready drivers for Unreal if you hadn’t already made the switch.
Now on to the speed of the asset, this gets a bit harder. I don’t work so much in the art pipeline myself. I’m primarily a programmer who assists on the art side, and since I work primarily with indie developers most scenes don’t usually get heavy enough to cause too many issues. When they have, sometimes local hardware wasn’t enough and they required it to be rendered at a render farm but generally that’s other art tools and I’m unaware of the landscape around them for UE. Sorry!
If possible, I’d recommend making a separate post here in the forums and seeing if anyone more versed in the more video production side of UE could give some tips.