[5.8.3][Linux/Wayland] Self-sizing modal dialogs render blank: bundled SDL 3.4.4 restores a stale 1x1 floating size (fixed upstream in SDL fcb5ef8a9c)

General Issues and Bug Reporting (Unreal Engine)
1

Summary

On Linux with the native Wayland SDL video driver, Unreal Editor modal dialogs that size themselves to their content (Yes/No message dialogs, “Create Animation Blueprint”, “Pick Parent Class”, …) open as an empty, see-through frame. Fixed-size dialogs (e.g. “Delete Assets”) render fine. The log keeps printing LogVulkanRHI: AcquireNextImage() failed due to the outdated swapchain, not even attempting to present.

Root cause (bundled SDL-gui-backend 3.4.4; already fixed upstream):

  1. Slate creates these windows fixed-size (non-resizable) at 1x1 and resizes them once the content is measured (SDL_SetWindowSize, e.g. 430x539).
  2. SDL commits the new xdg_toplevel before set_parent / xdg_dialog_v1.set_modal, so a tiling compositor (Hyprland here) first configures it as a regular window in a tile slot (1916x2117 on a 3840x2160 output), i.e. SDL treats it as tiled.
  3. While window->tiled is set, SDL_EVENT_WINDOW_RESIZED does not update window->floating, so it keeps the 1x1 creation size.
  4. The compositor then floats the dialog. For non-resizable windows SDL’s configure handler ignores the configure size and restores window->floating (1x1): wp_viewport.set_destination(1, 1) + set_min_size(1,1) / set_max_size(1,1) on every following configure (even pure focus changes). Everything Slate renders lands in a 1x1 viewport.

Upstream SDL fixed exactly this on main in libsdl-org/SDL@fcb5ef8a9c “wayland: Handle odd behavior with non-resizable windows in certain tiling compositors”. It is not in any 3.4.x release yet (checked up to release-3.4.18).

Request: cherry-pick fcb5ef8a9c into Engine/Source/ThirdParty/SDL3 (SDL-gui-backend), or update to an SDL that contains it.

Evidence (WAYLAND_DEBUG=client, “Create Animation Blueprint”, no compositor window rules):

-> xdg_toplevel#1504.set_min_size(1, 1) / set_max_size(1, 1)   # created fixed-size at 1x1
<- xdg_toplevel#1504.configure(1916, 2117, array[16])          # first configure: tile-sized slot
-> wp_viewport#1454.set_destination(430, 539)                  # Slate resizes to content
-> xdg_toplevel#1504.set_min_size(430, 539) / set_max_size(430, 539)
-> xdg_toplevel#1504.set_parent(...); xdg_dialog_v1.set_modal() # dialog role only now
<- xdg_toplevel#1504.configure(430, 539, array[20])            # compositor floats it at the right size
-> wp_viewport#1454.set_destination(1, 1)                      # SDL restores stale floating size
-> xdg_toplevel#1504.set_min_size(1, 1) / set_max_size(1, 1)   # repeated on every configure

Verified workaround: an LD_PRELOAD shim interposing SDL_SetWindowSize (exported from libUnrealEditor-ApplicationCore.so): for non-resizable, non-popup windows it marks them resizable with min = max = requested size, so the configure size is honored and the window stays effectively fixed. All affected dialogs then render. XWayland also avoids it but performs much worse here.

System: UE 5.8.3 (installed binary build), Vulkan RHI (SM6), MAILBOX present mode; Arch Linux (Omarchy), kernel 7.2.5, Hyprland 0.56.2 (Wayland); NVIDIA GeForce RTX 5090 Laptop GPU, driver 610.57.04 (PRIME offload). Independent of explicit sync (__NV_DISABLE_EXPLICIT_SYNC=1 changes nothing) and of present mode.

Related (hang variant with the same log line): Editor hangs forever in Linux upon opening modal dialogs, with a message about Vulkan swapchains being outdated.

What Type of Bug are you experiencing?

Platform Other

Steps to Reproduce

  1. Linux, native Wayland session with a tiling compositor (reproduced on Hyprland 0.56.2); SDL_VIDEO_DRIVER=wayland (SDL3 default on Wayland).
  2. Launch UnrealEditor 5.8.3 and open any project.
  3. Content Browser > right click > Animation > Animation Blueprint (or open any Yes/No message dialog, e.g. via Python: unreal.EditorDialog.show_message(“Test”, “Text”, unreal.AppMsgType.YES_NO)).

Expected Result

The dialog renders its content at its requested size (as it does under XWayland, and as fixed-size dialogs like “Delete Assets” already do under Wayland).

Observed Result

The dialog opens as an empty, see-through frame at the right size; nothing is ever drawn inside it. Resizing or moving it does not help; Enter/Esc still act on it blind. The log repeats “AcquireNextImage() failed due to the outdated swapchain”. WAYLAND_DEBUG shows SDL setting the dialog’s viewport destination and min/max size to 1x1 after every configure (see Summary).

Affects Versions

5.8

Platform(s)

Linux

Additional Notes

Fixed-size dialogs are unaffected because they are created at their real size, so window->floating is correct from the start. The upstream fix stores the new size in window->floating when a non-resizable window has ended up tiled (SDL_waylandwindow.c, Wayland_SetWindowSize). Happy to share the full WAYLAND_DEBUG log or the LD_PRELOAD shim source if useful.

2

Same issue here, I can confirm both the diagnosis and workaround.

UE 5.8.3 (installed from the binary build), Vulkan RHI (SM6), Gentoo, Linux 7.2.11, Hyprland 0.56.2 (native wayland tiling window manager), Nvidia Geforce RTX 4070 laptop, driver 615.78.08.

Symptoms match exactly: self-sizing modal dialogs (Yes/No confirmations, text-entry prompts) open as empty, see-through frames, while Enter/Esc/Tab still act on them blind. Fixed-size dialogs render fine. XWayland works-around it but isn’t a viable option for me.

An LD_PRELOAD shim along the lines described (interpose SDL_SetWindowSize; for non-resizable, non-popup windows set min = max = requested size and mark them resizable before forwarding the call) fixes every affected dialog for me, with no side effects on the main window, menus or tooltips so far.

+1 for cherry-picking libsdl-org/SDL@fcb5ef8a9c into the bundled SDL-gui-backend, or moving to an SDL release that includes it.