I know this is not the best solution, but it does actually work. dev-5.8 fixed the crash for me aswell and it even works better than 5.7. Just hope we get 5.8.1 with these fixes integrated soon, so we don’t have to risk it with some possibly unstable changes from the dev branch
Adding an NVIDIA datapoint with breadcrumbs, since most of the deep diagnostics in this thread were RADV.
Setup: RTX 4070 Ti, driver 595, Linux Mint 22.3, UE 5.8 built from the 5.8 branch (CL 55302956) with dev-5.8 cherry-picks 67e386ce (UE-381095 readback barrier) and 5c3a3626 (UE-385632 semaphore recycle) applied. Project: Lumen + VSM + Substrate + SM6, fresh/near-empty level.
Symptoms: VK_ERROR_DEVICE_LOST → TerminateOnGPUCrash, windowed editor only — -RenderOffscreen sessions have never crashed. No kernel Xid. Crashes correlate strongly with interaction (viewport navigation, panel churn), not uptime; an idle editor survives indefinitely. Five crashes observed:
- 2x breadcrumbs on
Pipeline AsyncCompute, “No breadcrumb nodes found for this queue” (stock CL 55302956) -
- 1x with
r.Vulkan.AllowAsyncCompute=0: moved toPipeline Graphics, frame 6-8 at boot, last finished nodes were SkyAtmosphere LUTs / CaptureConvolveSkyEnvMap (all Finished — submission-level, not a workload fault) -
- 1x with both dev-5.8 cherry-picks applied: survived ~10 min of active use, then
Pipeline Graphicsdevice-lost (In: 0x8008f1f8, Out: 0x8008f1f7 — one submission outstanding, no nodes) -
- 1x with
-noraytracing: frame 7 at boot — so RT is not (solely) the trigger on NVIDIA
- 1x with
- Current state: cherry-picks +
r.Vulkan.EnablePipelineLRUCache=1+r.Vulkan.AllowAsyncCompute=0— stable: 20-minute soak under sustained interaction (looping Sequencer playback + viewport navigation) with zero crashes, where the same session previously died in seconds-to-minutes. The interaction correlation plus the LRU cache being the decisive mitigation points at pipeline-lifetime racing (consistent with theNotifyDeletedGraphicsPSOimmediate-destroy suspicion raised in the 5.7.x thread) still being reachable on NVIDIA after UE-385632/UE-381095.
- 1x with both dev-5.8 cherry-picks applied: survived ~10 min of active use, then
- Happy to provide full logs/breadcrumb dumps if useful.
- 1x with
1 Like
Having this same issue, tested in compiled UE and Binary distro from Epic, and the same issue in both attempts (Compiled and Binary from Epic)
Setup: NVIDIA GeForce RTX 3060 Laptop GPU, driver 595.71.5
➜ UnrealEngine git:(release) ✗ lspci -k | grep -A 3 -E "(VGA|3D)"
0000:00:02.0 VGA compatible controller: Intel Corporation TigerLake-H GT1 [UHD Graphics] (rev 01)
DeviceName: Onboard - Video
Subsystem: Razer USA Ltd. Device 2017
Kernel driver in use: i915
--
0000:01:00.0 VGA compatible controller: NVIDIA Corporation GA106M [GeForce RTX 3060 Mobile / Max-Q] (rev a1)
Subsystem: Razer USA Ltd. Device 2017
Kernel driver in use: nvidia
Kernel modules: nvidiafb, nouveau, nvidia_drm, nvidia
➜ UnrealEngine git:(release) ✗ nvidia-smi --query-gpu=driver_version,name --format=csv
driver_version, name
595.71.05, NVIDIA GeForce RTX 3060 Laptop GPU
➜ UnrealEngine git:(release) ✗ vulkaninfo --summary 2>/dev/null | grep -E "GPU id|driverVersion|deviceName"
driverVersion = 595.71.5.0
deviceName = NVIDIA GeForce RTX 3060 Laptop GPU
driverVersion = 26.0.3
deviceName = Intel(R) UHD Graphics (TGL GT1)
driverVersion = 26.0.3
deviceName = llvmpipe (LLVM 21.1.8, 256 bits)
➜ UnrealEngine git:(release) ✗ glxinfo | grep -i "OpenGL version"
OpenGL version string: 4.6.0 NVIDIA 595.71.05
➜ UnrealEngine git:(release) ✗ dmesg | tail -50 | grep -iE "amdgpu|nvidia|timeout|reset|fault"
dmesg: read kernel buffer failed: Operation not permitted
➜ UnrealEngine git:(release) ✗ sudo dmesg | tail -50 | grep -iE "amdgpu|nvidia|timeout|reset|fault"
[sudo: authenticate] Password:
[ 9198.520458] nvidia-modeset: WARNING: GPU:0: Got unknown DPCD revision 0.0 for Samsung S34J55x (DP-5.2), HDR may not work
[ 9198.599694] nvidia-modeset: WARNING: GPU:0: Got unknown DPCD revision 0.0 for Samsung S34J55x (DP-5.2), HDR may not work
[ 9198.667156] nvidia-modeset: WARNING: GPU:0: Got unknown DPCD revision 0.0 for Samsung S34J55x (DP-5.2), HDR may not work
[ 9198.728003] nvidia-modeset: WARNING: GPU:0: Got unknown DPCD revision 0.0 for Samsung S34J55x (DP-5.2), HDR may not work
[ 9198.792751] nvidia-modeset: WARNING: GPU:0: Got unknown DPCD revision 0.0 for Samsung S34J55x (DP-5.2), HDR may not work
[ 9198.851552] nvidia-modeset: WARNING: GPU:0: Got unknown DPCD revision 0.0 for Samsung S34J55x (DP-5.2), HDR may not work
➜ UnrealEngine git:(release) ✗
Hello,
We have multiple bugs/issues/vendors/versions being discussed here. It started with Nvidia in the title, then it took a turn to RADV at some point. I’ll try to clear things up a bit maybe I hope… 
My original reply was here was targeted at RADV users:
I was suggesting to test fixes meant for RADV users from main or dev-5.8 for people with that ability. So folks…
- …on RADV using 5.8.0 binaries or source reporting it still repros are consistent with expectations
- …on RADV using dev-5.8 source builds that have less frequent issues are consistent with expectations
- …on Nvidia saying neither of these fixes help them are consistent with expectations as well since we hadn’t observed those issues on Nvidia (although the datapoint/confirmation is very much appreciated!)
Botton line, with those fixes targeted at editor+engine bugs specific to RADV users, all vendors should be getting mostly the same experience.
Another category of editor crashes we have identified is in our windowing with both X11 and Wayland (both behaving slightly differntly). JohnathanP’s description sums it up very nicely:
“Crashes correlate strongly with interaction (viewport navigation, panel churn), not uptime”. That “panel churn” includes when you open/close windows, but also tooltips, notification, comboboxes, etc.
For this last category, multiple changes have been made to hopefully fix/improve the situation, it’s still unclear if we caught all of them (there were many potential causes). The pipeline lifetimes have indeed been “extended” in this release, it would be surprising (although not impossible?) if this were still the main issue. These other fixes (different from the RADV ones) live in Main or Dev-5.8, they should mostly be there by now (some of these fixes touch the libs we link into so it might be necessary to run the setup scripts again to pull them if I’m not mistaken?). For those who can, I would also suggest alternating between X11 and Wayland to see if you get the same results.
Thanks for all your efforts digging into this, and thanks for your patience!
Cheers,
Thanks for looking into this.
I’m running the source code of UE right now through my AI agent tool on CLion, selected Claude
Is still working on a few points on the release branch so far
Audit swapchain semaphore lifecycle
On Linux/desktop VULKAN_USE_IMAGE_ACQUIRE_FENCES == 1, so AcquireImageIndex does a blocking CPU WaitForFence after every acquire (VulkanSwapChain.cpp:771-777). This is important context for finding #1. Now I have everything needed. Let me compile the findings.
Audit Findings: VulkanRHI swapchain semaphore/fence lifecycle (Linux DEVICE_LOST)
I traced the full acquire → wait → signal → present pipeline across VulkanSwapChain.cpp, VulkanViewport.cpp, VulkanQueue.cpp, VulkanContext.cpp/.h, and VulkanSynchronization.cpp. Findings below, highest-confidence first.
FINDING 1 — RenderingDoneSemaphores double-signal when present is skipped after FlushCommands (binary semaphore signaled twice without an intervening wait)
Files/lines:
Signal enqueued: VulkanViewport.cpp:776-779 (Context.AddSignalSemaphore(BackBufferData[AcquiredImageIndex].RenderingDoneSemaphores))
Signal actually executed by queue: VulkanViewport.cpp:820 (Context.FlushCommands(EVulkanFlushFlags::WaitForSubmission)) → VulkanContext.cpp:118 → VulkanQueue.cpp:409-419 (vkQueueSubmit with pSignalSemaphores)
Wait that is supposed to consume it: VulkanSwapChain.cpp:793-798 inside Present(), reached only via DoCheckedSwapChainJob (VulkanViewport.cpp:194)
Semaphore is a plain binary semaphore: VulkanViewport.cpp:448 (new FVulkanSemaphore(Device) — no Timeline flag).
Code path (the bug):
FVulkanViewport::Present acquires an image, then at line 778 enqueues RenderingDoneSemaphores[AcquiredImageIndex] as a signal semaphore on the payload.
Line 820 FlushCommands(WaitForSubmission) submits that payload → vkQueueSubmit signals the binary semaphore. This happens unconditionally, before any present.
Present is then attempted via DoCheckedSwapChainJob (line 847). But the present wait (vkQueuePresentKHR waiting on that same semaphore) can be skipped:
DoCheckedSwapChainJob's inner lambda returns early "Healthy" without presenting if Viewport->SwapChain == nullptr (VulkanViewport.cpp:182-185) or AcquiredImageIndex == -1 (VulkanViewport.cpp:188-192). Between the FlushCommands at 820 and the present, DoCheckedSwapChainJob may call RecreateSwapchain (line 223) on an OUT_OF_DATE/SURFACE_LOST retry, which sets SwapChain/AcquiredImageIndex such that the next SwapChainJob() iteration presents a fresh state — the originally-signaled semaphore for the old image is never waited on.
Also the window-invalid guard at VulkanViewport.cpp:837-845 returns before DoCheckedSwapChainJob — but note this path is reached only when bFailedToDelayAcquireBackbuffer was false, i.e. the signal at 778 already ran. So the semaphore is left signaled with no present wait at all.
Next frame, the same image index is acquired again and line 778 signals the same binary semaphore a second time.
Why it violates the spec: Vulkan requires a binary semaphore to be in the unsignaled state at the moment a vkQueueSubmit signal operation targets it, and every signal must be paired with exactly one wait that unsignals it (VUID-vkQueueSubmit-pSignalSemaphores-00067 and the semaphore-signal-operation ordering rules). Signaling a binary semaphore that is already signaled (or has an outstanding unconsumed signal) is undefined behavior.
How it manifests as DEVICE_LOST on Linux: RADV and NVIDIA both implement binary semaphores as real GPU sync primitives. A second signal on an already-signaled binary semaphore corrupts the driver's payload tracking; the subsequent present or submit that eventually does wait on it either waits on a stale payload or the driver faults the queue → VK_ERROR_DEVICE_LOST on the next vkQueueSubmit/vkQueuePresentKHR/vkWaitForFences. This is exactly the class of failure seen during window resize / surface-lost churn on XWayland/Wayland where the skip paths at lines 182-192 and 837-845 fire.
Confidence: High (the signal-before-present ordering and the multiple present-skip returns are all in-file and unconditional on the signal side).
Minimal fix: Do not enqueue the rendering-done signal unless the present that consumes it is guaranteed to run. Move the AddSignalSemaphore so it is only added on the code path that reaches vkQueuePresentKHR, or make it recoverable: if the present is skipped (any early-return in DoCheckedSwapChainJob/window-invalid at 837), drain the already-submitted signal with a throwaway wait before reuse. Concretely, in FVulkanViewport::Present, gate line 778 on the same window-valid / swapchain-valid conditions checked at 837-845, and in the skip branches call Device.WaitUntilIdle() (already done in DestroySwapchain/RecreateSwapchain, but NOT on the plain window-invalid return at 839-844) and then recreate the RenderingDoneSemaphores entry (allocate a fresh binary semaphore) so the next frame starts from a known-unsignaled state. The cleanest spec-correct fix is to allocate the rendering-done signal semaphore per-present (or use a timeline semaphore) rather than one persistent binary semaphore per image index reused every frame.
FINDING 2 — Window-invalid early return leaves the just-signaled RenderingDoneSemaphores pending and does NOT wait-idle
Files/lines: VulkanViewport.cpp:837-845
if (!FVulkanPlatform::IsWindowValid(WindowHandle))
{
AcquiredImageIndex = -1;
if (SwapChain) { SwapChain->ResetCurrentImageIndex(); }
return false; // <-- signal from line 778 was already submitted at line 820
}
Code path: For the non-DelayAcquire path (Linux desktop default), TryAcquireImageIndex succeeds → line 778 adds the signal → line 820 FlushCommands submits & signals the binary semaphore → line 837 sees the window went invalid (common on Wayland popup teardown) → returns false. The signal is now outstanding with no consuming present wait. ResetCurrentImageIndex() only clears CPU-side CurrentImageIndex; it does nothing to the pending semaphore. Next frame re-signals it (feeds Finding 1).
Why it violates the spec / DEVICE_LOST: Same binary-semaphore double-signal rule as Finding 1. Additionally the acquire semaphore for this frame (waited-on by the flushed command buffer at VulkanViewport.cpp:247) is fine here, but the rendering-done semaphore is orphaned.
Confidence: High.
Minimal fix: Before returning at line 844, if AcquiredImageIndex >= 0 was signaled, either wait-idle (Device.WaitUntilIdle()) and reallocate that image's RenderingDoneSemaphores, or restructure so the signal is only added once present is committed (see Finding 1 fix — a single fix covers both).
FINDING 3 — Acquire semaphore reused while a previous acquire on it may still be pending, when r.Vulkan.CpuWaitForFence is disabled
Files/lines:
Ring rotation: VulkanSwapChain.cpp:711-712 (SemaphoreIndex = (SemaphoreIndex + 1) % ImageAcquiredSemaphore.Num())
Semaphore array sized to image count: VulkanSwapChain.cpp:659-663 (ImageAcquiredSemaphore.SetNum(NumSwapChainImages))
Passed to acquire: VulkanSwapChain.cpp:728-734
The wait is submitted separately by the caller: VulkanViewport.cpp:247 (Context.AddWaitSemaphore(..., AcquiredSemaphore)), consumed at the next FlushCommands.
The safety CPU wait that hides this: VulkanSwapChain.cpp:771-777 (WaitForFence), guarded by VULKAN_USE_IMAGE_ACQUIRE_FENCES.
GShouldCpuWaitForFence cvar (VulkanSwapChain.cpp:41-47) is declared but not consulted in AcquireImageIndex — the fence wait is unconditional under the compile flag; the cvar is dead. (Reported as a correctness gap, not style.)
Code path / hazard: SemaphoreIndex cycles modulo NumSwapChainImages. The acquire semaphore is signaled by the presentation engine and waited on by the command-buffer submit that renders into the image. Because there is exactly one semaphore per image and vkAcquireNextImageKHR will not return an image whose previous present hasn't completed, the ring index generally lines up. BUT the wait for ImageAcquiredSemaphore[SemaphoreIndex] is not submitted synchronously inside AcquireImageIndex — it is deferred to the caller's next FlushCommands. The only thing guaranteeing the previous use of ImageAcquiredSemaphore[SemaphoreIndex] has been fully waited before re-passing it to vkAcquireNextImageKHR is the blocking WaitForFence at lines 771-777. On the Android platform header (VULKAN_USE_IMAGE_ACQUIRE_FENCES == 0) that wait is compiled out, and if a desktop build ever compiles it out or someone wires up the unused GShouldCpuWaitForFence cvar to skip it, re-passing a still-pending acquire semaphore to vkAcquireNextImageKHR is a direct spec violation (VUID-vkAcquireNextImageKHR-semaphore-01779: the semaphore must have no pending signal or wait operations).
Why it violates the spec / DEVICE_LOST: Passing a binary semaphore with an outstanding signal/wait to vkAcquireNextImageKHR is UB; RADV in particular will fault → DEVICE_LOST.
Confidence: Medium (latent — masked on Linux desktop today by the unconditional fence wait; live on the Android path and on any config that removes the wait; also the GShouldCpuWaitForFence cvar being non-functional is a real trap).
Minimal fix: Decouple acquire-semaphore count from image count and size the ring to at least NumSwapChainImages + 1 so a re-signal can never target a semaphore whose wait hasn't been submitted; and either honor GShouldCpuWaitForFence explicitly or delete the cvar so no one disables the fence wait believing it is safe. Best: keep the per-image fence wait (lines 771-777) mandatory on all WSI platforms lacking present-fence/VK_EXT_swapchain_maintenance1.
FINDING 4 — Acquire semaphore's wait is never submitted on the bFirstUse == false + fallback path, but the acquire still signaled it
Files/lines: VulkanViewport.cpp:236-270 (TryAcquireImageIndex) vs VulkanViewport.cpp:60-87 (AcquireBackBufferImage fallback to dummy).
Code path: In AcquireBackBufferImage (non-DelayAcquire path via OnGetBackBufferImage), if TryAcquireImageIndex returns true, the acquire semaphore's wait was added at VulkanViewport.cpp:247 and will be consumed at the next flush — correct. But TryAcquireImageIndex adds the wait before it can fail on subsequent logic; it cannot early-return after 247 here, so that is fine. The real asymmetry is between success and the fallback branch (lines 78-85): on fallback, no acquire happened, so no orphaned wait — also fine. However, when AcquireImageIndex returns OutOfDate/SurfaceLost after vkAcquireNextImageKHR already succeeded on a prior call in the same frame is not possible here.
The genuine concern: AcquireImageIndex increments NumAcquireCalls and signals the semaphore only on success (VulkanSwapChain.cpp:737-747 restore SemaphoreIndex and return on OUT_OF_DATE/SURFACE_LOST before the semaphore is considered used). That path is correct — on those errors the driver did not signal the semaphore, and the index is rolled back. No orphaned wait here. I could not construct a concrete "acquire succeeded but wait never submitted" case on the standard path, because AddWaitSemaphore at line 247 executes in the same synchronous scope right after a successful acquire.
Confidence: Low / effectively a non-finding — the acquire→wait pairing is structurally sound on the success path; the error paths correctly roll back before marking the semaphore used. Reported for completeness so it is not re-investigated.
FINDING 5 — ImageAcquiredFences reset/wait pairing is correct; no fence-reuse-without-reset bug
Files/lines: VulkanSwapChain.cpp:718 (ResetFence before acquire), :774 (WaitForFence after), allocation signaled at :656 (AllocateFence(true)).
The fence is created signaled, but AcquireImageIndex calls ResetFence (line 718) on ImageAcquiredFences[SemaphoreIndex] before passing it to vkAcquireNextImageKHR, then waits on it (line 774). ResetFence (VulkanSynchronization.cpp:175-182) is a no-op-safe guard. One subtlety: the fence created signaled at construction is never waited-then-reset before its first acquire — but line 718 resets it before first use, so vkAcquireNextImageKHR always gets an unsignaled fence. No violation.
Confidence: High that this is NOT a bug (verified the reset-before-submit ordering).
Summary / recommended action
The one real, ship-blocking hazard is Findings 1 + 2 (same root cause): the rendering-done binary semaphore is signaled unconditionally at VulkanViewport.cpp:778/820 but the paired present wait is skipped on multiple Linux WSI churn paths (VulkanViewport.cpp:182-192, 837-845, and the recreate loop in DoCheckedSwapChainJob), producing a binary-semaphore double-signal → DEVICE_LOST. Fix by only signaling when the present is committed, or by allocating the signal semaphore per-present / using a timeline semaphore. Finding 3 is a latent ring-sizing + dead-cvar trap worth hardening. Findings 4 and 5 are clean. agentId: af25ba5274443df09 (use SendMessage with to: 'af25ba5274443df09', summary: '<5-10 word recap>' to continue this agent) subagent_tokens: 97799 tool_uses: 16 duration_ms: 186879
Audit submission and queue sync
I now have a full understanding. Let me compile the findings. I have identified several concrete hazards. Let me verify one more thing about the WaitForAnyFence timeout behavior when NumFences != NumSyncs (the branch is skipped entirely — a potential busy-spin, not device lost). And confirm the SemaphoreStorage dangling-pointer risk since pointers into a TArray are stored in SubmitInfo while the array keeps growing.
The SemaphoreStorage.Reserve(NumWaitSemaphores + NumSignalSemaphores) — with timelines, NumSignalSemaphores already includes += NumPayloads (line 273), so reservation is adequate and pointers stay stable. Good — that one is safe. Let me finalize.
VulkanRHI DEVICE_LOST Audit — Command Buffer Submission, Queue Sync, Semaphore Ordering
I read VulkanSubmission.cpp, VulkanQueue.cpp/.h, VulkanCommandBuffer.cpp/.h, VulkanSubmission.h, plus VulkanSynchronization.cpp/.h and VulkanContext.h for cross-references. Findings below, highest confidence first.
Finding 1 — Cross-queue binary-semaphore wait can be reordered incorrectly, producing a wait on an unsignaled semaphore
File: VulkanQueue.cpp:172-247 (SubmitQueuedPayloads / CanBeProcessed) and VulkanQueue.cpp:409-419 (signal registration) Code path: A payload's cross-queue dependency on a binary FVulkanSemaphore is gated on the CPU by the SignaledSemas map, not by the GPU. A payload is only allowed to submit once every wait-semaphore handle is present in SignaledSemas (populated when the producing queue's payload was submitted, line 415). The check then Removes the entry (line 219). Why it violates Vulkan rules: The map is keyed purely by VkSemaphore handle with no generation/value. A binary semaphore is a single-use signal→wait pair. If the same FVulkanSemaphore object (same handle) is reused across frames/payloads, or if two payloads legitimately reference the same handle, the first consumer Removes it and a later consumer will never see it in the map → it will keep breaking out of the submit loop (that consumer stalls), while the actual GPU signal already fired. Conversely, because presence in the map is only a CPU bookkeeping proxy, a submit can be released to vkQueueSubmit while the producing queue's submit is merely recorded on the CPU but the wait semaphore is a binary sema that the GPU has not yet been told to signal in the right order — a binary semaphore that is waited-on before its signal op is queued is illegal and RADV/NV will fault. Manifests as DEVICE_LOST on Linux: RADV enforces binary-semaphore signal-before-wait strictly; a wait submitted with no matching pending signal (or a double-waited binary sema) yields VK_ERROR_DEVICE_LOST at the next vkQueueSubmit/vkWaitSemaphores. On NVIDIA it typically surfaces as a queue hang then TDR→DEVICE_LOST. Confidence: medium (depends on whether any code path reuses a binary FVulkanSemaphore handle across payloads; the map design assumes globally-unique-per-signal handles). Fix: Key SignaledSemas by {VkSemaphore, uint64 signalGeneration} or store a count, and assert exactly-once consumption. Better: track the producing queue's timeline value for the semaphore and gate on CompletedTimelineSemaphoreValue, rather than removing on first consume. At minimum, replace checkSlow(NumRemoved > 0) (line 220) with a hard check compiled in shipping to catch the missing-signal case before it reaches the driver.
Finding 2 — WaitForAnyFence is skipped when any next-payload fence is already signaled, but the skip condition is wrong and can spin / mis-wait
File: VulkanSubmission.cpp:411-429 Code path: In the fence (non-timeline) path, it collects only unsignaled fences into Fences[] (line 419-422), then only waits if (NumFences == NumSyncs) (line 426). Why it's a hazard: If some—but not all—queues' next fences are already signaled, NumFences < NumSyncs, so the wait is skipped entirely and the function returns having done no blocking wait. WaitAndProcessInterruptQueue then relies on ProcessInterruptQueue() making progress. If the already-signaled queue's payload is not actually poppable (e.g. it was already processed on a prior pass and Peek returns a different, still-pending payload), this becomes a busy-spin on the interrupt thread at TPri_Highest (line 159). That is not directly DEVICE_LOST, but the pegged high-priority interrupt thread starves the submission thread, delaying signals and can trip the 10 s timeout waits elsewhere. Note also WaitForAnyFence with FenceHandles.Num()==0 (all signaled) calls vkWaitForFences(count=0, …, waitAll=false) which is undefined/immediate-return — guarded here by the NumFences == NumSyncs gate, but fragile. Manifests: Not a direct fault; indirect DEVICE_LOST via watchdog/timeout when the interrupt thread starves submission under fence mode (r.Vulkan.Submission.AllowTimelineSemaphores 0, common on older Mesa without timeline support). Confidence: low-medium. Fix: Always wait when NumFences > 0; only skip the wait when NumFences == 0. Drop the NumFences == NumSyncs condition — a partially-signaled set still needs to block on the remaining fences (or return without spinning).
Finding 3 — Command buffer Reset() (pool recycling) is gated only by payload completion, but secondary command buffers are reset without confirming their own GPU completion
File: VulkanSubmission.cpp:639-644 (CompletePayload → CommandBuffer->Reset()), VulkanCommandBuffer.cpp:404-435 (Reset), VulkanCommandBuffer.cpp:541-551 (SetSubmitted) Code path: CompletePayload runs after the payload's timeline value is reached (ProcessInterruptQueue, VulkanQueue.cpp:518) and calls Reset() on each primary command buffer, which recursively resets ExecutedSecondaryCommandBuffers (line 407-411). Reset() only transitions Submitted → NeedReset; the actual vkResetCommandBuffer happens lazily in Begin() (VulkanCommandBuffer.cpp:260-263). Why it's mostly OK but has one hole: Primary completion does imply secondary completion if the secondary was executed via vkCmdExecuteCommands inside that primary. But Reset() early-returns unless State == EState::Submitted (line 415). A secondary buffer that was appended to a payload via Merge (VulkanSubmission.cpp:59-61) but for some path not marked Submitted (e.g. a parallel context whose primary was submitted on a different queue timeline than the one whose completion triggered CompletePayload) would be reset while State != Submitted → the recursive reset is a no-op and it silently leaks the NeedReset transition, OR—if it was marked Submitted by an unrelated earlier payload—it can be reset/rebegun while the GPU still references it. The correctness hinges entirely on TimelineSemaphoreValue being the completion of the specific queue that executed the secondary. Merged/parallel payloads across queues break that assumption. Manifests as DEVICE_LOST on Linux: Resetting/re-recording a command buffer (via vkResetCommandBuffer in Begin) that the GPU is still executing is a classic use-after-reset → RADV/NV DEVICE_LOST. The VK_COMMAND_POOL_CREATE_RESET_COMMAND_BUFFER_BIT pool flag (VulkanCommandBuffer.cpp:673) makes per-buffer reset legal but does not make it safe while in-flight. Confidence: medium (the single-queue common path is safe; the cross-queue parallel/merge path is where the completion attribution is questionable). Fix: Have each FVulkanCommandBuffer record the (Queue, TimelineSemaphoreValue) it was actually submitted under at SetSubmitted() time, and in CompletePayload/Reset assert/verify that buffer's own submitted timeline value <= CompletedTimelineSemaphoreValue of its queue before allowing the Submitted → NeedReset transition. Do not rely on the enclosing payload's queue.
Finding 4 — SetSubmitted() and Reset() take the pool lock only around the state flip, but FlushProfilerEvents and handle collection run lock-free during submit while the interrupt thread may concurrently Reset
File: VulkanQueue.cpp:392-403 (submit loop touches CommandBuffer on submission thread), VulkanSubmission.cpp:641-644 (CompletePayload calls Reset on interrupt thread), VulkanCommandBuffer.cpp:541-551 / 404-435 Code path: On the submission thread, SubmitPayloads iterates a payload's command buffers calling CommandBuffer->FlushProfilerEvents(...) (line 396) and SetSubmitted() (line 402). On the interrupt thread, CompletePayload for an earlier payload calls CommandBuffer->Reset(), which touches ExecutedSecondaryCommandBuffers, EndedBarrierEvents, and profiler-related bit flags. The pool FCriticalSection is only held around the State assignment (line 414-434 / 548-549), not around ExecutedSecondaryCommandBuffers.Empty() (line 411) or EndedBarrierEvents iteration (line 426-430), nor around FlushProfilerEvents/BeginEvents.Reset(). Why it violates rules: Two threads mutate the same FVulkanCommandBuffer's non-atomic members (ExecutedSecondaryCommandBuffers, EventStream, BeginEvents, EndedBarrierEvents) without a common lock. A given buffer object should only be in one payload at a time, so this is safe if payloads never share buffers and a buffer is never simultaneously "being submitted" and "being completed" — but nothing enforces that invariant in code. Under Merge and secondary-buffer sharing it is plausible for the interrupt thread's Reset of a completed primary to recurse into a secondary that the submission thread is concurrently flushing. Manifests as DEVICE_LOST on Linux: A torn ExecutedSecondaryCommandBuffers/EndedBarrierEvents array can cause vkCmdResetEvent (VulkanCommandBuffer.cpp:236-239, in End) or vkResetCommandBuffer to run against a freed/aliased handle, or a secondary handle to be submitted after reset → driver fault. Confidence: low-medium (requires buffer sharing across concurrently-live payloads; hard to prove without the context layer, but the locking is demonstrably narrower than the data it protects). Fix: Hold CommandBufferPool.GetCS() across the entire Reset() body (including ExecutedSecondaryCommandBuffers/EndedBarrierEvents mutation) and across SetSubmitted()/FlushProfilerEvents, or make the buffer lifecycle single-threaded by moving profiler flush to the interrupt thread. Add a debug assert that a buffer is never in the Submitted state when SetSubmitted is called.
Finding 5 — Timeline "empty payload" completion value can reference a value that is never signaled if it is the first submit on the queue
File: VulkanQueue.cpp:344-350 Code path: For a payload with no cmd buffers/semaphores/sparse binds, Payload->TimelineSemaphoreValue = NextTimelineSemaphoreValue - 1; — i.e. it claims completion equal to the previous submit's timeline value. Why it's a hazard: NextTimelineSemaphoreValue starts at 1 (VulkanQueue.h:136). If an empty CPU-event-only payload is the first thing enqueued on a queue (e.g. the profiler-init payloads created in InitializeSubmissionPipe, VulkanSubmission.cpp:176-178, or RHIBlockUntilGPUIdle's empty payloads on an idle queue), it sets TimelineSemaphoreValue = 0. In ProcessInterruptQueue, Payload->TimelineSemaphoreValue (0) <= CompletedTimelineSemaphoreValue (0) → treated complete immediately (fine). But if it is NextTimelineSemaphoreValue - 1 where the previous submit was on the timeline but the empty payload is dequeued/interrupt-checked before that previous submit's fence/timeline is actually signaled, the empty payload is reported complete and its SyncPoints/SubmissionEvents fire early. Downstream CPU work (e.g. RHIBlockUntilGPUIdle, EOP tasks) then proceeds believing the GPU reached a point it has not. Manifests as DEVICE_LOST on Linux: Indirect — early sync-point completion lets the CPU free/reset resources (command buffers via Finding 3, descriptor pools at VulkanSubmission.cpp:646-649) still referenced by the not-actually-finished prior submit → driver fault on RADV/NV. Confidence: low-medium. Fix: For empty payloads, set TimelineSemaphoreValue to the last actually-submitted value on this queue (NextTimelineSemaphoreValue - 1 is correct only when a real submit preceded it in the same batch). Guard against the value-0 / first-submit case explicitly, and ensure empty payloads inherit completion of the genuinely-last GPU submit, not merely the counter minus one.
Non-issue verified (ruled out)
SemaphoreStorage dangling pointers: SubmitInfo.pWaitSemaphores/pSignalSemaphores point into SemaphoreStorage, which is Reserved up front (VulkanQueue.cpp:279) including the per-payload timeline additions (counted at line 273). No reallocation occurs during the loop, so the raw pointers stay valid through vkQueueSubmit. Safe.
WaitFlags lifetime: SubmitInfo.pWaitDstStageMask = Payload->WaitFlags.GetData() (line 385) — the payloads stay alive through submission (deleted only in ProcessInterruptQueue), so no dangling. Safe.
Timeline monotonicity within a queue: NextTimelineSemaphoreValue++ is strictly increasing per queue and each queue has its own timeline semaphore, so per-queue monotonic signaling is correct. Cross-queue values are independent (correct, since they're different semaphores).
Highest-priority to fix: Finding 1 (binary-semaphore CPU-map gating) and Finding 3 (cross-queue command-buffer reset attribution) — both are direct paths to a wait-on-never-signaled / reset-while-in-flight DEVICE_LOST on RADV, which is stricter than NVIDIA here. Recommend adding hard (shipping) asserts at VulkanQueue.cpp:220 and recording per-buffer submit timeline values as the first concrete steps. agentId: a8925553cd75c5a34 (use SendMessage with to: 'a8925553cd75c5a34', summary: '<5-10 word recap>' to continue this agent) subagent_tokens: 97316 tool_uses: 14 duration_ms: 156351
Audit barriers and layout transitions
I have a comprehensive understanding now. Let me compile my findings. The key architectural fact is that this RHI relies on the high-level RHI transition tracking to put color attachments into COLOR_ATTACHMENT_OPTIMAL before a LOAD render pass, and the render pass hardcodes initialLayout rather than reading the tracked layout. Let me note the specific concrete hazards.
VulkanRHI DEVICE_LOST Audit — Pipeline Barriers, Layout Transitions, Render Passes
I focused on barrier correctness, swapchain/present layouts, and render-pass load/store attachment layouts. Findings below, ordered by confidence/severity.
Finding 1 — Render pass color-attachment initialLayout is hardcoded to COLOR_ATTACHMENT_OPTIMAL on any non-CLEAR load, ignoring the actual tracked layout
File: Private/VulkanRenderTargetLayout.cpp:344-345
Code path: For every color attachment, initialLayout = (loadOp == CLEAR) ? UNDEFINED : COLOR_ATTACHMENT_OPTIMAL. On ELoad/DontLoad this asserts the image is already in COLOR_ATTACHMENT_OPTIMAL.
Why it violates Vulkan: If the resource's real layout at render-pass begin is anything other than COLOR_ATTACHMENT_OPTIMAL (e.g. still SHADER_READ_ONLY_OPTIMAL, GENERAL, or TRANSFER_DST_OPTIMAL), the render pass's implicit initialLayout transition is a lie. Vulkan requires initialLayout to match the image's current layout (or be UNDEFINED). Mismatch = undefined contents / driver treats memory as differently-tiled.
DEVICE_LOST on Linux: RADV and NVIDIA perform layout-dependent decompression / metadata (DCC/HTILE-equiv) on the implicit transition. Reading with the wrong assumed source layout corrupts the compression metadata state, producing a GPU page fault or hang reported as VK_ERROR_DEVICE_LOST. This is exactly the "attachment layout does not match what the render pass expects" hazard.
Mitigating context: In practice the high-level RHI transition tracker is supposed to transition RTs to RTV/COLOR_ATTACHMENT_OPTIMAL before the pass, so this is usually consistent. The bug bites when the source access was Unknown/Discard or when a DontLoad target was last left in a non-attachment layout. Also note DontLoad maps to LOAD_OP_DONT_CARE, not CLEAR, so it still takes the COLOR_ATTACHMENT_OPTIMAL branch even though contents are discardable.
Confidence: medium (design relies on external tracking; real defect surfaces on the DontLoad-from-non-attachment case).
Fix: For DontLoad (LOAD_OP_DONT_CARE) set initialLayout = VK_IMAGE_LAYOUT_UNDEFINED (contents not needed, avoids the assertion of a specific source layout). For genuine ELoad, thread the RHI-tracked current layout into the attachment description instead of hardcoding, mirroring what the depth path already does via CurrentDepthLayout (line 409).
Finding 2 — MSAA resolve attachment inherits initialLayout = COLOR_ATTACHMENT_OPTIMAL from the color desc
File: Private/VulkanRenderTargetLayout.cpp:352 (Desc[N+1] = Desc[N] copies initialLayout), then only loadOp/storeOp/samples are overwritten.
Code path: Resolve target attachment description is memberwise-copied from the multisample color attachment, so it keeps initialLayout = COLOR_ATTACHMENT_OPTIMAL (when the color load wasn't CLEAR) and finalLayout = COLOR_ATTACHMENT_OPTIMAL.
Why it violates Vulkan: A resolve target is frequently a fresh/transient texture whose real layout is UNDEFINED or SHADER_READ_ONLY. Declaring initialLayout = COLOR_ATTACHMENT_OPTIMAL claims contents/metadata that don't exist. Since loadOp is forced to DONT_CARE (line 354) the initialLayout should be UNDEFINED.
DEVICE_LOST on Linux: Same decompression-metadata corruption mechanism as Finding 1; resolve write into an image the driver thinks was already color-attachment-compressed can fault.
Confidence: medium.
Fix: After the copy at line 352, explicitly set Desc[NumAttachmentDescriptions + 1].initialLayout = VK_IMAGE_LAYOUT_UNDEFINED; (loadOp is already DONT_CARE, so previous contents are irrelevant).
Finding 3 — Present transitions the backbuffer from UNDEFINED → PRESENT_SRC_KHR, discarding the just-rendered image contents
File: Private/VulkanViewport.cpp:765 (immediate-acquire path) and identical pattern at :263 (first-use clear path).
Code path: In Present, for the non-delay-acquire path with a valid AcquiredImageIndex, VulkanSetImageLayout(..., VK_IMAGE_LAYOUT_UNDEFINED, VK_IMAGE_LAYOUT_PRESENT_SRC_KHR, ...).
Why it violates Vulkan: oldLayout = UNDEFINED explicitly tells the driver "contents may be discarded." This backbuffer was just rendered into (it is Data.Texture, the acquired swapchain image the frame drew to). The correct oldLayout is COLOR_ATTACHMENT_OPTIMAL (or whatever the last write layout was). Using UNDEFINED is legal API-wise but semantically wrong — it permits the driver to throw away the frame.
DEVICE_LOST on Linux: On RADV this typically manifests as a black/garbage present rather than a hard device loss, but combined with DCC it can leave the presented image in a compressed state the display engine can't scan out, which some Mesa/NVIDIA WSI paths escalate to a present/device error. This is squarely the hazard-#1/#3 "presenting an image whose layout/contents don't match present expectations."
Confidence: medium (contents-loss is definite; DEVICE_LOST escalation is driver-dependent). Note line 765's own check(RHIBackBuffer->Image == Data.Texture->Image) confirms this is the live rendered image, not a scratch one.
Fix: Use the real prior layout as oldLayout (VK_IMAGE_LAYOUT_COLOR_ATTACHMENT_OPTIMAL, matching the render pass finalLayout), not UNDEFINED, so contents are preserved into PRESENT_SRC_KHR. If the RHI genuinely doesn't track the last layout here, pull it from the backbuffer's tracked access state rather than defaulting to UNDEFINED.
Finding 4 — PRESENT_SRC_KHR gets access mask 0 with BOTTOM_OF_PIPE/COLOR_ATTACHMENT_OUTPUT but no write-visibility guarantee for the surface-transform copy path
File: Private/VulkanBarriers.cpp:87-89, 159-161, 244-250; interaction with Private/VulkanViewport.cpp:608, 705.
Code path: GetVkAccessMaskForLayout(PRESENT_SRC_KHR) = 0 and GetVkStageFlagsForLayout = BOTTOM_OF_PIPE. In CopyImageToBackBuffer, the barrier PRESENT_SRC_KHR → TRANSFER_SRC/SHADER_READ (line 608) derives srcAccess = 0, srcStage = BOTTOM_OF_PIPE for the source, and UNDEFINED → dst for the destination (line 609).
Why it can violate Vulkan: Transitioning out of PRESENT_SRC with srcStage = BOTTOM_OF_PIPE, srcAccess = 0 relies entirely on the acquire semaphore for the swapchain image's availability. For SrcSurface (the RenderingBackBuffer, an offscreen image previously written as a render target and left in PRESENT_SRC_KHR at line 542), there is no semaphore — its prior writes are ordered only by BOTTOM_OF_PIPE/access=0, which does not make prior color-attachment writes visible to the subsequent transfer/shader read. This is a genuine missing write→read visibility (hazard #2).
DEVICE_LOST on Linux: Reading the source before its render-target writes are flushed produces stale/garbage reads; not usually a hard hang, but on RADV the layout transition out of a PRESENT_SRC-tagged (uncompressed) image that still has pending DCC-compressed writes can fault. Lower likelihood of DEVICE_LOST than Findings 1-3.
Confidence: low-medium.
Fix: When the SrcSurface is the internal RenderingBackBuffer (not a real acquired swapchain image protected by a semaphore), give the "out of present" barrier a real srcStageMask/srcAccessMask reflecting the last write (COLOR_ATTACHMENT_OUTPUT / COLOR_ATTACHMENT_WRITE), rather than the present defaults of BOTTOM_OF_PIPE/0.
Finding 5 — AddImageAccessTransition silently substitutes InOutLayout for an UNDEFINED source without validating it matches, and forces GENERAL on unknown dest
File: Private/VulkanBarriers.cpp:1760-1778
Code path: If SrcLayout == UNDEFINED it is replaced with InOutLayout (the caller's cached layout) and access recomputed; the else branch only ensure()s equality (non-fatal in shipping). If DstLayout == UNDEFINED it is forced to GENERAL.
Why it can violate Vulkan: If the caller's cached InOutLayout has drifted from the image's real layout (e.g. after an aliasing/discard or a path that bypassed tracking), the emitted oldLayout mismatches actual layout. The ensure is compiled out in Shipping, so the mismatch proceeds to vkCmdPipelineBarrier silently.
DEVICE_LOST on Linux: Same compression-metadata corruption as Finding 1 — wrong oldLayout on a compressed image faults on RADV/NVIDIA.
Confidence: low (depends on caller cache correctness; this is the internal helper, not the tracked public path).
Fix: Promote the ensure(SrcLayout == InOutLayout) to a real runtime guard in dev builds and, more importantly, when they disagree prefer the image's actual tracked layout (InOutLayout) as oldLayout unconditionally rather than trusting the access-derived value — and log, so drift is caught before it reaches the driver.
Notes / non-findings (checked, look correct)
NotifyDeletedRenderTarget (VulkanRenderpass.cpp:222) correctly destroys framebuffers referencing a freed image, so hazard #4 (dangling attachment image after resize/free) is handled — DestroySwapchain calls Device.NotifyDeletedImage and idles the device first (VulkanViewport.cpp:558, 576). No defect found there.
Depth/stencil initialLayout/finalLayout correctly use the tracked CurrentDepthLayout/CurrentStencilLayout (VulkanRenderTargetLayout.cpp:409-446) and guard against UNDEFINED finalLayout — this is the correct pattern that the color path (Finding 1) does not follow.
The UNDEFINED → TRANSFER_DST transitions in VulkanTexture.cpp (initial upload/clear) are legitimate discards of never-initialized images.
Highest priority: Findings 1 and 3 — both are the classic "layout the driver assumes ≠ actual layout / discarded contents where contents are required" pattern that most directly produces VK_ERROR_DEVICE_LOST on Mesa/RADV and NVIDIA. The safest concrete fixes are: (1) set initialLayout = UNDEFINED for DONT_CARE loads and thread the tracked layout for genuine LOAD, and (3) stop using UNDEFINED as the present barrier's oldLayout for the rendered backbuffer. agentId: a8480c4ba4538ce1f (use SendMessage with to: 'a8480c4ba4538ce1f', summary: '<5-10 word recap>' to continue this agent) subagent_tokens: 125957 tool_uses: 13 duration_ms: 119726
It is still working once the code is worked on. I can submit for review and analysis and will reply to the Thread with the merge request so everyone can see the code.
Some of these are interesting to follow up on, others can be disproved if you challenge the results.
Looking at that first one… VULKAN_USE_IMAGE_ACQUIRE_FENCES has always bothered me… But given that we do have and it waits on the image right after acquiring it, why are semaphores still used in that case. 
It would be interesting to see if anyone could reproduce the issue with Vulkan validation enabled (adding -vulkandebug to the command line). It would dump extra messages in the logs for spec violations (things like double signal on binary semaphores would generate errors in the logs for example). It’s also possible that the overhead of adding validation changes the timing to hide the issue (which is also good imformation to have).
Cheers,