HLOD Builder Commandlet GPU device getting removed.

Development Rendering
, ,
1

This question was created in reference to: [DXGI_ERROR_DEVICE_REMOVED while building HLOD (on Windows [Content removed]

The linked question seems similar, although the error we’re seeing is a little different than reported there.

This only started occurring after updating to 5.8. We’re planning to try the “-warp” parameter mentioned in the other thread, but it will be a number of hours before we know if it helps or not.

[2026.08.11-05.06.32:870][250]LogHLODBuilder: Display: Evaluating HLOD rebuild policies...
[2026.08.11-05.06.32:870][250]LogHLODBuilder: Display:  * HLODRebuildPolicyHashCompare -> ApproveRebuild
[2026.08.11-05.06.32:870][250]LogHLODBuilder: Display: Evaluated HLOD rebuild policies, final decision: ApproveRebuild
[2026.08.11-05.06.33:458][250]LogD3D12RHI: Error: GPU crash detected:
 
	- Device 0 Removed: DXGI_ERROR_DRIVER_INTERNAL_ERROR
 
 
[2026.08.11-05.06.33:458][250]LogD3D12RHI: Display: D3D12 CreateReservedResource failed with params:
	Resource Dimension: 1
	Resource Width: 2147483648
	Resource Height: 1
	Format: 0
	Resource Flags: 4
[2026.08.11-05.06.33:458][250]LogD3D12RHI: Error: Adapter->CreateReservedResource( Desc, GetGPUMask(), InCreateD3D12Access, InResourceStateMode, InCreateD3D12Access, nullptr, &NewResource, Name, false) failed 
 at R:\ue58-tfb\Engine\Source\Runtime\D3D12RHI\Private\D3D12Allocation.cpp:1590 
 with error DXGI_ERROR_DEVICE_REMOVED
 
[2026.08.11-05.06.33:458][250]LogD3D12RHI: Error: Shader diagnostic messages and asserts:
 
 
 
	Device: 0, Queue 3D:
 
		No shader diagnostics found for this queue.
 
 
 
	Device: 0, Queue Copy:
 
		No shader diagnostics found for this queue.
 
 
 
	Device: 0, Queue Compute:
 
		No shader diagnostics found for this queue.
 
 
[2026.08.11-05.06.33:458][250]LogRHI: Error: Active GPU breadcrumbs:
 
 
 
	Device 0, Pipeline Graphics: (In: 0x8023f033, Out: 0x8023f02f)
 
		(ID: 0x8023f02a) [   Finished]	TexturePoolCopyOps
 
		(ID: 0x8023f02b) [   Finished]	GPUResourceUpdate
 
		(ID: 0x8023f02c) [   Finished]		RenderGraphExecute - 
 
		(ID: 0x8023f02d) [   Finished]			FRDGBuilder::SubmitBufferUploads
 
		(ID: 0x8023f02e) [   Finished]	CanvasClear
 
		(ID: 0x8023f02f) [     Active]	SendAllEndOfFrameUpdates
 
		(ID: 0x8023f030) [     Active]	SceneRender - ViewFamilies
 
		(ID: 0x8023f031) [     Active]		RayTracingGeometry
 
		(ID: 0x8023f033) [     Active]		GPUSkinCache_UpdateSkinningBatches
 
 
 
	Device 0, Pipeline AsyncCompute: (In: 0x8023f030, Out: 0x8023f02b)
 
		(ID: 0x8023f02b) [     Active]	GPUResourceUpdate
 
		(ID: 0x8023f02c) [     Active]		RenderGraphExecute - 
 
		(ID: 0x8023f030) [     Active]	SceneRender - ViewFamilies
 
 
[2026.08.11-05.06.36:460][250]LogNvidiaAftermath: Warning: Timed out while waiting for Aftermath to start the GPU crash dump.
[2026.08.11-05.06.36:460][250]LogD3D12RHI: Error: DRED: No breadcrumb head found.
[2026.08.11-05.06.36:460][250]LogD3D12RHI: Error: DRED: No PageFault data.
[2026.08.11-05.06.36:460][250]LogD3D12RHI: Error: Video Memory Stats from frame ID 2251:
[2026.08.11-05.06.36:460][250]LogD3D12RHI: Error: 	Local Budget:	23557.00 MB
[2026.08.11-05.06.36:460][250]LogD3D12RHI: Error: 	Local Used:	11784.75 MB
[2026.08.11-05.06.36:460][250]LogD3D12RHI: Error: 	System Budget:	208644.08 MB
[2026.08.11-05.06.36:460][250]LogD3D12RHI: Error: 	System Used:	 275.45 MB
[2026.08.11-05.06.36:468][250]LogD3D12RHI: Error: GPU Crashed or D3D Device Removed.

[Attachment Removed]

2

Hi,

Can you share the full log? I would like to review the D3D adapter that is being used.

Martin

[Attachment Removed]

3

There was a problem with the platform an the upload failed. Can you try again?

[Attachment Removed]

4

It appears that the problem is the GPU is running out of virtual address space. That would likely be caused by the GC not running often enough as the graphic resources are only cleared after the World was GC’ed. I was able to find that we resolved a similar problem at the editor level by running the GC after each HLOD which would be overkill for the commandlet.

I’m attaching a modified version of the HLOD builder code that collects garbage every 32 actors. Can you give this a try and let me know if it helps?

I added .txt to the file name to bypass the file type filter.

I will discuss the situation with the system owner so we can provide an official fix later.

Regards,

Martin

[Attachment Removed]

5

We decided to push the fix into the incoming 5.8.3 release. You won’t need keep the modification when you update to this version.

Martin

[Attachment Removed]

6

Yeah, here is a copy.

[Attachment Removed]

7

I edited the main question and attached there which seemed to have worked. But attaching to a reply has failed every attempt.

[Attachment Removed]

8

Thank you very much! We will give this a go and report back.

[Attachment Removed]

9

The supplied solution did indeed resolve the issue. The HLOD build was able to complete successfully. Thanks!

[Attachment Removed]