UE Render Crash DXGI_ERROR_DEVICE_REMOVED with Reason DXGI_ERROR_DEVICE_HUNG

Hi,

Looking for suggestions to fix the below issue which is occurring mostly during the Movie Render Queue.

We are trying to render a 4K Square (4096x4096) EXR image with the “High Resolution” render setting enabled and tried various tile sizes from 2, 4, 8 and 16. However, with all these permutations and combinations, the rendering is crashing immediately with the below-pasted error snipper from the crash log.

Any help/input/suggestion to fix this behaviour will be greatly appreciated.

GPU: NVIDIA Quadro RTX 8000 (48GB VRAM)

Please let us know if there is any other information required.

Thanks,

[2021.10.10-20.26.17:458][389]LogD3D12RHI: Display: QueryHeapCount is now 188416 elements
[2021.10.10-20.26.17:542][389]LogD3D12RHI: Warning: Descriptor cache ran out of sub allocated descriptor blocks! Moving to Context local View heap strategy
[2021.10.10-20.26.17:542][389]LogD3D12RHI: This should only happen in the Editor where it doesn't matter as much. If it happens in game you should increase the device global heap size!
[2021.10.10-20.26.17:542][389]LogD3D12RHI: Warning: Descriptor cache ran out of sub allocated descriptor blocks! Moving to Context local View heap strategy
[2021.10.10-20.26.17:542][389]LogD3D12RHI: This should only happen in the Editor where it doesn't matter as much. If it happens in game you should increase the device global heap size!
[2021.10.10-20.26.17:542][389]LogD3D12RHI: Warning: Descriptor cache ran out of sub allocated descriptor blocks! Moving to Context local View heap strategy
[2021.10.10-20.26.17:542][389]LogD3D12RHI: This should only happen in the Editor where it doesn't matter as much. If it happens in game you should increase the device global heap size!
[2021.10.10-20.26.17:542][389]LogD3D12RHI: Warning: Descriptor cache ran out of sub allocated descriptor blocks! Moving to Context local View heap strategy
[2021.10.10-20.26.17:542][389]LogD3D12RHI: This should only happen in the Editor where it doesn't matter as much. If it happens in game you should increase the device global heap size!
[2021.10.10-20.26.17:542][389]LogD3D12RHI: Warning: Descriptor cache ran out of sub allocated descriptor blocks! Moving to Context local View heap strategy
[2021.10.10-20.26.17:542][389]LogD3D12RHI: This should only happen in the Editor where it doesn't matter as much. If it happens in game you should increase the device global heap size!
[2021.10.10-20.26.17:542][389]LogD3D12RHI: Warning: Descriptor cache ran out of sub allocated descriptor blocks! Moving to Context local View heap strategy
[2021.10.10-20.26.17:542][389]LogD3D12RHI: This should only happen in the Editor where it doesn't matter as much. If it happens in game you should increase the device global heap size!
[2021.10.10-20.26.17:542][389]LogD3D12RHI: Warning: Descriptor cache ran out of sub allocated descriptor blocks! Moving to Context local View heap strategy
[2021.10.10-20.26.17:542][389]LogD3D12RHI: This should only happen in the Editor where it doesn't matter as much. If it happens in game you should increase the device global heap size!
[2021.10.10-20.26.17:542][389]LogD3D12RHI: Warning: Descriptor cache ran out of sub allocated descriptor blocks! Moving to Context local View heap strategy
[2021.10.10-20.26.17:542][389]LogD3D12RHI: This should only happen in the Editor where it doesn't matter as much. If it happens in game you should increase the device global heap size!
[2021.10.10-20.26.18:198][389]LogD3D12RHI: Warning: Descriptor cache ran out of sub allocated descriptor blocks! Moving to Context local View heap strategy
[2021.10.10-20.26.18:198][389]LogD3D12RHI: This should only happen in the Editor where it doesn't matter as much. If it happens in game you should increase the device global heap size!
[2021.10.10-20.26.22:418][389]LogD3D12RHI: Error: hr failed 
 at D:/Build/++UE4/Sync/Engine/Source/Runtime/D3D12RHI/Private/D3D12Resources.cpp:450 
 with error DXGI_ERROR_DEVICE_REMOVED with Reason: DXGI_ERROR_DEVICE_HUNG

[2021.10.10-20.26.22:418][389]LogD3D12RHI: Error: [GPUBreadCrumb] Last tracked GPU operations:
[2021.10.10-20.26.22:418][389]LogD3D12RHI: Error: [GPUBreadCrumb]	3D Queue 0 - End: BeginRenderingPrePass
[2021.10.10-20.26.22:418][389]LogD3D12RHI: Error: [GPUBreadCrumb]	Compute Queue 0 - No Data
[2021.10.10-20.26.22:418][389]LogD3D12RHI: Error: [GPUBreadCrumb]	Copy Queue 0 - No Data
[2021.10.10-20.26.22:418][389]LogD3D12RHI: Error: DRED: No PageFault data.
[2021.10.10-20.26.26:316][389]LogWindows: Error: Error reentered: Fatal error: [File:D:/Build/++UE4/Sync/Engine/Source/Runtime/D3D12RHI/Private/D3D12Util.cpp] [Line: 671] 
hr failed 
 at D:/Build/++UE4/Sync/Engine/Source/Runtime/D3D12RHI/Private/D3D12Resources.cpp:450 
 with error DXGI_ERROR_DEVICE_REMOVED with Reason: DXGI_ERROR_DEVICE_HUNG