While investigating GPU Hang crashes in our game we’ve encountered multiple times two kinds of scenario:
* Async Compute Pipeline is executing TexturePoolCopyOps while Graphics is doing different things.
* Graphics context returned HRESULT device hung error while waiting on Async Compute executing TexturePoolCopyOps.
We’ve searched the code and we’ve found a todo
// @todo dev-pr - explicit use of graphics context - nothing is synchronizing async compute - needs refactor
I think that accessing the context explicitly may be dangerous for multi-gpu systems and in general if the system doesn’t create a queue with the specific ERHIPipeline type.
This breadcrumb string “TexturePoolCopyOps” can only be found in
FD3D12TextureAllocatorPool::BeginFrame and 5.8 seem to have the same todo comment.
Is it safe to assume that ideally flushing Async Compute barriers on possibly shared resources could mitigate this kind of issue?
[Attachment Removed]
Hello, yes, there could be a synchronization issue that we haven’t resolved from the parallel RHI translation effort from 5.6, though it is hard to tell without a bit more information. Do you have any crash logs or GPU breadcrumbs that you can share with us? Can you also try to reproduce the crash with the following command-line arguments:
-rdgimmediate -onethread -rhivalidationThat should give us a call stack pointing directly to the crash site, although it is not guaranteed that you will reproduce the crash with this setup. Please let me know if you have any questions
[Attachment Removed]
Hi Tim,
thank you for your message. Unfortunately we cannot share the full log but, I can share the GPU breadcrumbs. I’ll try and repro the crash as you suggested and come back to you.
LogD3D12RHI: Error: GPU crash detected:
- Device 0 Removed: DXGI_ERROR_DEVICE_RESET
LogD3D12RHI: Error: Shader diagnostic messages and asserts:
Device: 0, Queue 3D:
No shader diagnostics found for this queue.
Device: 0, Queue Copy:
No shader diagnostics found for this queue.
Device: 0, Queue Compute:
No shader diagnostics found for this queue.
LogRHI: Error: Active GPU breadcrumbs:
Device 0, Pipeline Graphics: (In: 0x80bfdb56, Out: 0x80bfdb56)
(ID: 0x80bfd4cd) [ Finished] Frame 8657
(ID: 0x80bfdb53) [ Finished] RenderGraphExecute - Slate
(ID: 0x80bfdb51) [ Finished] SlateUI Title = %s
(ID: 0x80bfdb52) [ Finished] ElementBatch
(ID: 0x80bfdb55) [ Finished] BufferPoolCopyOps
(ID: 0x80bfdb56) [ Active] TexturePoolCopyOps
Device 0, Pipeline AsyncCompute: (In: 0x80bfdb56, Out: 0x80bfdb56)
(ID: 0x80bfd4cd) [ Finished] Frame 8657
(ID: 0x80bfdb53) [ Finished] RenderGraphExecute - Slate
(ID: 0x80bfdb54) [ Finished] FRDGBuilder::SubmitBufferUploads
(ID: 0x80bfdb55) [ Finished] BufferPoolCopyOps
(ID: 0x80bfdb56) [ Active] TexturePoolCopyOps
[Attachment Removed]
Hi Lorenzo, I am still looking into this for you. In the meantime, have you been able to reproduce the crash with the flags I shared earlier?
[Attachment Removed]