Vulkan RHI submits batches that wait on binary semaphores before the corresponding signal operations are submitted (violates VUID-vkQueueSubmit-pWaitSemaphores-03238); deadlocks on Mesa/RADV Linux

General Issues and Bug Reporting (Unreal Engine)
1

Summary

Vulkan RHI submits batches that wait on binary semaphores before the corresponding signal operations are submitted (violates VUID-vkQueueSubmit-pWaitSemaphores-03238); deadlocks on Mesa/RADV Linux

What Type of Bug are you experiencing?

Editor

Steps to Reproduce

  1. Linux with any Mesa-based Vulkan driver (e.g. RADV; tested on AMD RX 9070 XT), Mesa without the robustness fix from MR1.
    (On RT-capable AMD hardware, getting past editor startup currently also requires MR2, which fixes an unrelated RADV descriptor bug.)
  2. Open any UE 5.8 project in UnrealEditor with the Vulkan RHI.
  3. Wait ~10-60 seconds after the viewport appears.
  4. Alternatively, on any platform/driver: run with the Khronos validation layer enabled - VUID-vkQueueSubmit2-semaphore-03873 / VUID-vkQueueSubmit-pWaitSemaphores-03238 errors are reported.

Expected Result

The editor runs normally. Per the Vulkan spec
(VUID-vkQueueSubmit-pWaitSemaphores-03238), every batch that waits on a
binary semaphore must be submitted only after the corresponding signal
operation has been submitted for execution:

“All elements of the pWaitSemaphores member of all elements of pSubmits
created with a VkSemaphoreType of VK_SEMAPHORE_TYPE_BINARY must reference
a semaphore signal operation that has been submitted for execution and any
semaphore signal operations on which it depends (if any) must have also
been submitted for execution.”

Observed Result

The editor freezes, then the watchdog aborts with “GameThread timed out
waiting for RenderThread” about 40 seconds later.

The RHISubmission thread is deadlocked inside the driver’s vkQueueSubmit2:
Mesa’s shared Vulkan runtime performs a blocking wait for each binary wait
semaphore’s signal operation to exist (it is entitled to, per the VUID
above). The signal operation is produced later by the very thread that is
now blocked, so it never arrives.

I instrumented the driver to check, at vkQueueSubmit2 time, whether each
binary wait semaphore already had a pending signal operation. In a
75-second editor session, 36,757 waits were submitted before their signal
operation existed, on both the graphics and the async compute queue —
i.e. this is steady-state behavior of the parallel submission pipeline
(FVulkanPayload processing in VulkanSubmission.cpp / FVulkanQueue::Submit
in VulkanQueue.cpp), hundreds of violations per frame, not a rare race:
payloads are submitted to one queue while the payload that signals the
binary semaphores they wait on has not yet been submitted (typically to
another queue, e.g. graphics vs async compute).

On drivers that pass unmaterialized syncobj waits to the kernel this
happens to work, which is presumably why it went unnoticed.

Affects Versions

5.8
5.7

Platform(s)

Linux

For crash reports, include your callstack

Deadlocked RHISubmission thread (gdb, inside vkQueueSubmit2):

#0  ioctl
#1  drmIoctl
#2  drmSyncobjTimelineWait
#3  vk_drm_syncobj_wait_many (abs_timeout_ns=INT64_MAX)
#4  __vk_sync_wait (wait_flags=VK_SYNC_WAIT_PENDING, abs_timeout_ns=UINT64_MAX)
#5  vk_sync_wait
#6  vk_queue_submit_move_binary_waits_to_temps (vk_queue.c:1025)
#7  vk_queue_submit
#8  vk_queue_merge_submit
#9  vk_common_QueueSubmit2 (submitCount=15)
#10 FVulkanQueue::Submit (VulkanQueue.cpp:486)
#11 FVulkanRayTracingPipelineLibraryCache::GetOrAddLibrary (VulkanRayTracing.cpp:1498, via task graph)

Subsequent watchdog abort (game thread):

0x... libc.so.6!pthread_cond_timedwait
0x... FPThreadEvent::Wait (GenericPlatformProcess.cpp:519)
0x... FRenderCommandFence::Wait
0x... FFrameEndSync::Sync (RenderingThread.cpp:2509)
0x... FEngineLoop::Tick (LaunchEngineLoop.cpp:6084)

Additional Notes

  • Mesa-side robustness fix (defer the blocking wait to the driver’s submit
    thread instead of deadlocking):
    MR1 — but the
    API usage is invalid regardless and can bite on any driver that takes the
    VUID at its word.
  • Related RADV fix needed to get UE 5.8 bindless past startup on RT-capable
    AMD GPUs (mesa/mesa#15697):
    MR2
  • Suggested engine-side fix: in the RHI submission pipeline, do not submit
    a payload until the payloads that signal its binary wait semaphores have
    been submitted to their queues (cross-queue ordering), or use timeline
    semaphores for cross-queue synchronization, which have no such
    requirement.
  • Likely duplicates / corroborating community reports (no root cause
    identified in them):
  • With both Mesa MRs applied locally, UnrealEditor 5.8 runs correctly on
    RADV/GFX1201 (bindless + ray tracing enabled).
2

Correction after deeper analysis: UE’s submission order is actually valid, and this is a Mesa bug, not an engine bug.

My original instrumentation checked whether the binary wait’s payload was materialized in the kernel at wait-submission time — but that’s not what the spec requires. Logging the API-level order instead shows UE submits the signal operations before the dependent waits in all cases (the validation layer agrees: no 03238/03873 errors). The payloads were unmaterialized only because Mesa’s own submission threading defers the signaling submit internally, and Mesa’s blocking wait inside vkQueueSubmit then deadlocks against itself.

The Mesa fix is here and makes UE 5.7/5.8 work correctly on Linux:

(plus Fix for RT-capable AMD GPUs).

Apologies for the incorrect attribution — no engine-side change is needed for this particular issue. The report can be closed. (The modal-dialog hang on NVIDIA linked above is unrelated to this mechanism and may still be worth Epic’s attention.)