I ended up running insights just to see if anything specific pops up in there regarding this.
One rare instance, it restarted the gpu and left bread crumbs as well. Adding them here in case anyone can figure out whats going on, because looks like devs sure don’t look at forums.
From CrashContext
<CallStack>KERNELBASE 0x00007ffedd8b0000 + 2cf19 UnrealEditor-Core 0x00007ffe62890000 + 603963 UnrealEditor-D3D12RHI 0x00007ffe43fc0000 + 16b1ac UnrealEditor-D3D12RHI 0x00007ffe43fc0000 + 16c755 UnrealEditor-D3D12RHI 0x00007ffe43fc0000 + f84b9 UnrealEditor-D3D12RHI 0x00007ffe43fc0000 + 10f8bc UnrealEditor-D3D12RHI 0x00007ffe43fc0000 + 117367 UnrealEditor-D3D12RHI 0x00007ffe43fc0000 + 1272ef UnrealEditor-Core 0x00007ffe62890000 + 684e22 UnrealEditor-Core 0x00007ffe62890000 + 678657 KERNEL32 0x00007ffede940000 + 17344 ntdll 0x00007ffedfdd0000 + 526b1</CallStack>
<IsCrashed>true</IsCrashed>
<Registers />
<ThreadID>12560</ThreadID>
<ThreadName>RHIInterruptThread</ThreadName>
Breadcrumbs rom Logs
[2023.09.29-18.03.42:923][378]LogPlayLevel: Display: Destroying online subsystem :Context_3
[2023.09.29-18.03.48:175][575]LogTexture: Display: Building textures: /Engine/Transient.Texture2D_6 (G8, 128x128 x1x1x1) (Required Memory Estimate: 0.432286 MB), EncodeSpeed: Fast
[2023.09.29-18.03.52:091][575]LogViewport: Scene viewport resized to 1170x900, mode Windowed.
[2023.09.29-18.04.14:146][588]LogD3D12RHI: Error: CurrentQueue.Fence.D3DFence->GetCompletedValue() failed
at D:\build\++UE5\Sync\Engine\Source\Runtime\D3D12RHI\Private\D3D12Submission.cpp:980
with error DXGI_ERROR_DEVICE_REMOVED with Reason: DXGI_ERROR_DEVICE_HUNG
[2023.09.29-18.04.14:146][588]LogRHI: Error: [Aftermath] Status: Timeout
[2023.09.29-18.04.14:146][588]LogRHI: Error: [Aftermath] Failed to get Aftermath stack data
[2023.09.29-18.04.14:146][588]LogD3D12RHI: Error: [GPUBreadCrumb] Last tracked GPU operations:
[2023.09.29-18.04.14:147][588]LogD3D12RHI: Error: [GPUBreadCrumb] 3D Queue 0
Breadcrumbs: > Frame 9585 [Active]
Breadcrumbs: | BufferPoolCopyOps [Finished]
Breadcrumbs: | TexturePoolCopyOps [Finished]
Breadcrumbs: | WorldTick [Finished]
Breadcrumbs: | SendAllEndOfFrameUpdates [Finished]
Breadcrumbs: | NiagaraGpuComputeDispatch [Finished]
Breadcrumbs: | ClearGPUMessageBuffer [Finished]
Breadcrumbs: | VirtualTextureClear [Finished]
Breadcrumbs: > Scene [Active]
Breadcrumbs: | Niagara::GPUProfiler_BeginFrame [Finished]
Breadcrumbs: | NiagaraGpuComputeDispatch [Finished]
Breadcrumbs: | AccessModePass[Graphics] (Textures: 0, Buffers: 1) [Finished]
Breadcrumbs: | FFXSystem::PreRender [Finished]
Breadcrumbs: | FGPUSortManager::OnPreRender [Finished]
Breadcrumbs: | GPUScene.UploadDynamicPrimitiveShaderDataForView [Finished]
Breadcrumbs: | BuildRenderingCommandsDeferred(Culling=On) [Finished]
Breadcrumbs: | AccessModePass[Graphics] (Textures: 7, Buffers: 4) [Finished]
Breadcrumbs: | ClearDepthStencil (SceneDepthZ) [Finished]
Breadcrumbs: | PrePass DDM_AllOpaqueNoVelocity (Forced by Nanite) [Finished]
Breadcrumbs: | ComputeLightGrid [Finished]
Breadcrumbs: | BeginOcclusionTests [Finished]
Breadcrumbs: | BuildHZB(ViewId=0) [Finished]
Breadcrumbs: | SubmitCommands [Finished]
Breadcrumbs: | FenceOcclusionTests [Finished]
Breadcrumbs: | SkyAtmosphereLUTs [Finished]
Breadcrumbs: | CaptureConvolveSkyEnvMap [Finished]
Breadcrumbs: | LumenSceneUpdate: 2 card captures 0.031M texels [Finished]
Breadcrumbs: | ClearBuffer(Lumen.FeedbackAllocator Size=4bytes) [Finished]
Breadcrumbs: | ClearBuffer(Lumen.Feedback Size=43776bytes) [Finished]
Breadcrumbs: | LumenSceneLighting [Finished]
Breadcrumbs: | GBufferClear [Finished]
Breadcrumbs: | SkyAtmosphereEditor [Finished]
Breadcrumbs: | BasePass [Finished]
Breadcrumbs: | CopyStencilToLightingChannels [Finished]
Breadcrumbs: | FVirtualShadowMapArray::BuildPageAllocation [Finished]
Breadcrumbs: | ShadowDepths [Finished]
Breadcrumbs: | DispatchToRHI [Finished]
Breadcrumbs: | Readback [Finished]
Breadcrumbs: | ClearStencil (SceneDepthZ) [Finished]
Breadcrumbs: | DiffuseIndirectAndAO [Finished]
Breadcrumbs: | ClearTranslucencyLightingVolumeCompute 64 [Finished]
Breadcrumbs: | Lights [Finished]
Breadcrumbs: | FilterTranslucentVolume 64x64x64 Cascades:2 [Finished]
Breadcrumbs: | VolumetricCloud [Finished]
Breadcrumbs: | VolCloudReconstruction [Finished]
Breadcrumbs: | TSR ComputeMoireLuma 1170x900 [Finished]
Breadcrumbs: | SkyAtmosphere [Finished]
Breadcrumbs: | ExponentialHeightFog [Finished]
Breadcrumbs: | VolCloudComposeOverScene [Finished]
Breadcrumbs: | SetSceneTexturesUniformBuffer [Finished]
Breadcrumbs: | FFXSystem::PostRenderOpaque [Finished]
Breadcrumbs: | AsyncGpuTraceHelper::PostRenderOpaque [Finished]
Breadcrumbs: | Niagara::PostRenderFinish [Finished]
Breadcrumbs: | UnsetSceneTexturesUniformBuffer [Finished]
Breadcrumbs: | FGPUSortManager::OnPostRenderOpaque [Finished]
Breadcrumbs: | Translucency [Finished]
Breadcrumbs: | VirtualTextureFeedbackCopy [Finished]
Breadcrumbs: > PostProcessing [Active]
Breadcrumbs: | TemporalSuperResolution(sg.AntiAliasingQuality=3) 1170x900 -> 1170x900 [Finished]
Breadcrumbs: | LocalExposure [Finished]
Breadcrumbs: | Histogram [Finished]
Breadcrumbs: | SceneDownsample [Finished]
Breadcrumbs: | LocalExposure - Blurred Luminance [Finished]
Breadcrumbs: | HistogramEyeAdaptation (CS) [Finished]
Breadcrumbs: | CopyEyeAdaptationToTexture (CS) [Finished]
Breadcrumbs: | EnqueueCopy(EyeAdaptationBuffer) [Finished]
Breadcrumbs: | BloomSetup 585x450 (PS) [Finished]
Breadcrumbs: | SceneDownsample [Finished]
Breadcrumbs: > Bloom [Active]
Breadcrumbs: | GaussianBlur.BloomX 19x15 (PS, Static) [Finished]
Breadcrumbs: | GaussianBlur.BloomY 19x15 (PS, Static) [Finished]
Breadcrumbs: | GaussianBlur.BloomX 37x29 (PS, Static) [Finished]
Breadcrumbs: | GaussianBlur.BloomY 37x29 (PS, Static) [Finished]
Breadcrumbs: | GaussianBlur.BloomX 74x57 (PS, Static) [Finished]
Breadcrumbs: | GaussianBlur.BloomY 74x57 (PS, Static) [Finished]
Breadcrumbs: | GaussianBlur.BloomX 147x113 (PS, Static) [Finished]
Breadcrumbs: | GaussianBlur.BloomY 147x113 (PS, Static) [Finished]
Breadcrumbs: | GaussianBlur.BloomX 293x225 (PS, Static) [Finished]
Breadcrumbs: > GaussianBlur.BloomY 293x225 (PS, Static) [Active]
Breadcrumbs: > GaussianBlur.BloomX 585x450 (PS, Static) [Active]
Breadcrumbs: GaussianBlur.BloomY 585x450 (PS, Static) [Not started]
Breadcrumbs: Tonemap 1170x900 (PS GammaOnly=0) [Not started]
Breadcrumbs: EditorSelectionOutlines [Not started]
Breadcrumbs: CompositeEditorPrimitives 1170x900 MSAA=4 [Not started]
Breadcrumbs: Submit Lumen surface cache feedback [Not started]
Breadcrumbs: ReleaseRayTracingResources [Not started]
Breadcrumbs: RenderFinish [Not started]
Breadcrumbs: ExtractUniformBuffer [Not started]
Breadcrumbs: EnqueueCopy(GPUMessageManager.MessageBuffer) [Not started]
Breadcrumbs: AccessModePass[Graphics] (Textures: 14, Buffers: 5) [Not started]
Breadcrumbs: CanvasBatchedElements [Not started]
Breadcrumbs: CanvasBatchedElements [Not started]
Breadcrumbs: CanvasBatchedElements [Not started]
Breadcrumbs: CanvasBatchedElements [Not started]
Breadcrumbs: CanvasBatchedElements [Not started]
Breadcrumbs: CanvasBatchedElements [Not started]
Breadcrumbs: SlateUI Title = Train_BRDF - Unreal Editor [Not started]
Breadcrumbs: SlateUI Title = <none> [Not started]
Breadcrumbs: Frame 9586 [Not started]
[2023.09.29-18.04.14:185][588]LogD3D12RHI: Error: [GPUBreadCrumb] Copy Queue 0
[2023.09.29-18.04.14:195][588]LogD3D12RHI: Error: [GPUBreadCrumb] Compute Queue 0
[2023.09.29-18.04.14:207][588]LogD3D12RHI: Error: DRED: Last tracked GPU operations:
[2023.09.29-18.04.14:207][588]LogD3D12RHI: Error: DRED: Commandlist "FD3D12CommandList (GPU 0)" on CommandQueue "3D Queue (GPU 0)", 35 completed of 84
[2023.09.29-18.04.14:207][588]LogD3D12RHI: Error: Op: 0, EndEvent
[2023.09.29-18.04.14:207][588]LogD3D12RHI: Error: Op: 1, BeginEvent [BloomSetup 585x450 (PS)]
[2023.09.29-18.04.14:207][588]LogD3D12RHI: Error: Op: 2, EndEvent
[2023.09.29-18.04.14:207][588]LogD3D12RHI: Error: Op: 3, BeginEvent [SceneDownsample]
[2023.09.29-18.04.14:207][588]LogD3D12RHI: Error: Op: 4, BeginEvent [Downsample(Scene(1/4) Quality=High PS) 585x450 -> 293x225]
[2023.09.29-18.04.14:207][588]LogD3D12RHI: Error: Op: 5, EndEvent
[2023.09.29-18.04.14:207][588]LogD3D12RHI: Error: Op: 6, BeginEvent [Downsample(Scene(1/8) Quality=High PS) 293x225 -> 147x113]
[2023.09.29-18.04.14:207][588]LogD3D12RHI: Error: Op: 7, EndEvent
[2023.09.29-18.04.14:207][588]LogD3D12RHI: Error: Op: 8, BeginEvent [Downsample(Scene(1/16) Quality=High PS) 147x113 -> 74x57]
[2023.09.29-18.04.14:207][588]LogD3D12RHI: Error: Op: 9, EndEvent
[2023.09.29-18.04.14:207][588]LogD3D12RHI: Error: Op: 10, BeginEvent [Downsample(Scene(1/32) Quality=High PS) 74x57 -> 37x29]
[2023.09.29-18.04.14:207][588]LogD3D12RHI: Error: Op: 11, EndEvent
[2023.09.29-18.04.14:207][588]LogD3D12RHI: Error: Op: 12, BeginEvent [Downsample(Scene(1/64) Quality=High PS) 37x29 -> 19x15]
[2023.09.29-18.04.14:207][588]LogD3D12RHI: Error: Op: 13, EndEvent
[2023.09.29-18.04.14:207][588]LogD3D12RHI: Error: Op: 14, EndEvent
[2023.09.29-18.04.14:208][588]LogD3D12RHI: Error: Op: 15, BeginEvent [Bloom]
[2023.09.29-18.04.14:208][588]LogD3D12RHI: Error: Op: 16, BeginEvent [GaussianBlur.BloomX 19x15 (PS, Static)]
[2023.09.29-18.04.14:208][588]LogD3D12RHI: Error: Op: 17, EndEvent
[2023.09.29-18.04.14:208][588]LogD3D12RHI: Error: Op: 18, BeginEvent [GaussianBlur.BloomY 19x15 (PS, Static)]
[2023.09.29-18.04.14:208][588]LogD3D12RHI: Error: Op: 19, EndEvent
[2023.09.29-18.04.14:208][588]LogD3D12RHI: Error: Op: 20, BeginEvent [GaussianBlur.BloomX 37x29 (PS, Static)]
[2023.09.29-18.04.14:208][588]LogD3D12RHI: Error: Op: 21, EndEvent
[2023.09.29-18.04.14:208][588]LogD3D12RHI: Error: Op: 22, BeginEvent [GaussianBlur.BloomY 37x29 (PS, Static)]
[2023.09.29-18.04.14:208][588]LogD3D12RHI: Error: Op: 23, EndEvent
[2023.09.29-18.04.14:208][588]LogD3D12RHI: Error: Op: 24, BeginEvent [GaussianBlur.BloomX 74x57 (PS, Static)]
[2023.09.29-18.04.14:208][588]LogD3D12RHI: Error: Op: 25, EndEvent
[2023.09.29-18.04.14:208][588]LogD3D12RHI: Error: Op: 26, BeginEvent [GaussianBlur.BloomY 74x57 (PS, Static)]
[2023.09.29-18.04.14:208][588]LogD3D12RHI: Error: Op: 27, EndEvent
[2023.09.29-18.04.14:208][588]LogD3D12RHI: Error: Op: 28, BeginEvent [GaussianBlur.BloomX 147x113 (PS, Static)]
[2023.09.29-18.04.14:208][588]LogD3D12RHI: Error: Op: 29, EndEvent
[2023.09.29-18.04.14:208][588]LogD3D12RHI: Error: Op: 30, BeginEvent [GaussianBlur.BloomY 147x113 (PS, Static)]
[2023.09.29-18.04.14:208][588]LogD3D12RHI: Error: Op: 31, EndEvent
[2023.09.29-18.04.14:208][588]LogD3D12RHI: Error: Op: 32, BeginEvent [GaussianBlur.BloomX 293x225 (PS, Static)]
[2023.09.29-18.04.14:208][588]LogD3D12RHI: Error: Op: 33, EndEvent
[2023.09.29-18.04.14:208][588]LogD3D12RHI: Error: Op: 34, BeginEvent [GaussianBlur.BloomY 293x225 (PS, Static)] - LAST COMPLETED
[2023.09.29-18.04.14:208][588]LogD3D12RHI: Error: Op: 35, EndEvent
[2023.09.29-18.04.14:208][588]LogD3D12RHI: Error: Op: 36, BeginEvent [GaussianBlur.BloomX 585x450 (PS, Static)]
[2023.09.29-18.04.14:208][588]LogD3D12RHI: Error: Op: 37, EndEvent
[2023.09.29-18.04.14:208][588]LogD3D12RHI: Error: Op: 38, BeginEvent [GaussianBlur.BloomY 585x450 (PS, Static)]
[2023.09.29-18.04.14:208][588]LogD3D12RHI: Error: Op: 39, EndEvent
[2023.09.29-18.04.14:208][588]LogD3D12RHI: Error: Op: 40, EndEvent
[2023.09.29-18.04.14:208][588]LogD3D12RHI: Error: Op: 41, BeginEvent [Tonemap 1170x900 (PS GammaOnly=0)]
[2023.09.29-18.04.14:208][588]LogD3D12RHI: Error: Op: 42, EndEvent
[2023.09.29-18.04.14:208][588]LogD3D12RHI: Error: Op: 43, BeginEvent [EditorSelectionOutlines]
[2023.09.29-18.04.14:208][588]LogD3D12RHI: Error: Op: 44, BeginEvent [OutlineDepth 1170x900]
[2023.09.29-18.04.14:208][588]LogD3D12RHI: Error: Op: 45, BeginEvent [EditorSelectionDepth]
[2023.09.29-18.04.14:208][588]LogD3D12RHI: Error: Op: 46, EndEvent
[2023.09.29-18.04.14:208][588]LogD3D12RHI: Error: Op: 47, BeginEvent [DrawOutlineBorder]
[2023.09.29-18.04.14:208][588]LogD3D12RHI: Error: Op: 48, EndEvent
[2023.09.29-18.04.14:208][588]LogD3D12RHI: Error: Op: 49, EndEvent
[2023.09.29-18.04.14:208][588]LogD3D12RHI: Error: Op: 50, BeginEvent [OutlineColor 1170x900]
[2023.09.29-18.04.14:208][588]LogD3D12RHI: Error: Op: 51, EndEvent
[2023.09.29-18.04.14:208][588]LogD3D12RHI: Error: Op: 52, EndEvent
[2023.09.29-18.04.14:208][588]LogD3D12RHI: Error: Op: 53, BeginEvent [CompositeEditorPrimitives 1170x900 MSAA=4]
[2023.09.29-18.04.14:208][588]LogD3D12RHI: Error: Op: 54, BeginEvent [PopulateDepth 1170x900 MSAA]
[2023.09.29-18.04.14:208][588]LogD3D12RHI: Error: Op: 55, EndEvent
[2023.09.29-18.04.14:208][588]LogD3D12RHI: Error: DRED: No PageFault data.
[2023.09.29-18.04.14:208][588]LogD3D12RHI: Error: Memory Info from frame ID 9627:
[2023.09.29-18.04.14:208][588]LogD3D12RHI: Error: Budget: 11465.31 MB
[2023.09.29-18.04.14:208][588]LogD3D12RHI: Error: Used: 5642.53 MB
Logs from separate freezes
[2023.09.12-10.37.15:184][ 1]Running Python start-up script 'D:/Epic Games/UE_5.3/Engine/Plugins/Animation/ControlRig/Content/Python/init_unreal.py': 1.039058 secs
[2023.09.12-10.37.15:397][ 1]LogAssetRegistry: AssetRegistryGather time 0.6125s: AssetDataDiscovery 0.1586s, AssetDataGather 0.3692s, StoreResults 0.0847s.
[2023.09.12-10.37.15:891][ 1]LogCollectionManager: Fixed up redirectors for 0 collections in 0.000008 seconds (updated 0 objects)
[2023.09.12-11.22.14:965][ 93]LogAnalytics: Display: [UEEditor.Rocket.Release] APIServer = https://datarouter.ol.epicgames.com/. AppVersion = 5.3.0-27405482+++UE5+Release-5.3
[2023.09.12-11.22.15:251][ 93]LogAnalyticsSessionSummarySender: Sent summary session report for: AppId=UEEditor.Rocket.Release SessionId={623AF6C9-4050-1FB8-D4A9-02BC3381F948}
[2023.09.12-11.36.34:541][671]LogDerivedDataCache: C:/Users/sande/AppData/Local/UnrealEngine/Common/DerivedDataCache: Maintenance finished in +00:00:32.547 and deleted 0 files with total size 0 MiB and 0 empty folders. Scanned 16652 files in 18681 folders with total size 6819 MiB.
[2023.09.12-12.03.06:152][430]LogD3D12RHI: Warning: GPU timeout: A payload (0x00000A8A9B045200) on the [0x00000A8A859E8300, 3D] queue has not completed after 5.000000 seconds.
[2023.09.29-10.20.18:879][647]LogViewport: Scene viewport resized to 1130x885, mode Windowed.
[2023.09.29-10.26.39:201][ 62]LogSlate: Window 'Plugins' being destroyed
[2023.09.29-10.26.39:207][ 62]LogSlate: Window 'Plugins' being destroyed
[2023.09.29-10.26.39:463][ 62]LogDirectoryWatcher: A directory notification for '../../../Engine/Plugins/' was aborted.
[2023.09.29-10.27.06:414][400]LogD3D12RHI: Warning: GPU timeout: A payload (0x0000086F21971E00) on the [0x0000086D6E288300, 3D] queue has not completed after 5.000000 seconds.
[2023.09.29-19.04.04:421][236]LogAudioMixer: FMixerPlatformXAudio2::StopAudioStream() called. InstanceID=2
[2023.09.29-19.04.04:426][236]LogAudioMixer: FMixerPlatformXAudio2::StopAudioStream() called. InstanceID=2
[2023.09.29-19.04.04:537][236]LogUObjectHash: Compacting FUObjectHashTables data took 1.44ms
[2023.09.29-19.04.04:647][237]LogPlayLevel: Display: Destroying online subsystem :Context_1
[2023.09.29-19.04.11:016][339]LogViewport: Scene viewport resized to 1170x900, mode Windowed.
