[5.8.3][Linux/Wayland] Self-sizing modal dialogs render blank: bundled SDL 3.4.4 restores a stale 1x1 floating size (fixed upstream in SDL fcb5ef8a9c)

General Issues and Bug Reporting (Unreal Engine)
1

Summary

On Linux with the native Wayland SDL video driver, Unreal Editor modal dialogs that size themselves to their content (Yes/No message dialogs, “Create Animation Blueprint”, “Pick Parent Class”, …) open as an empty, see-through frame. Fixed-size dialogs (e.g. “Delete Assets”) render fine. The log keeps printing LogVulkanRHI: AcquireNextImage() failed due to the outdated swapchain, not even attempting to present.

Root cause (bundled SDL-gui-backend 3.4.4; already fixed upstream):

  1. Slate creates these windows fixed-size (non-resizable) at 1x1 and resizes them once the content is measured (SDL_SetWindowSize, e.g. 430x539).
  2. SDL commits the new xdg_toplevel before set_parent / xdg_dialog_v1.set_modal, so a tiling compositor (Hyprland here) first configures it as a regular window in a tile slot (1916x2117 on a 3840x2160 output), i.e. SDL treats it as tiled.
  3. While window->tiled is set, SDL_EVENT_WINDOW_RESIZED does not update window->floating, so it keeps the 1x1 creation size.
  4. The compositor then floats the dialog. For non-resizable windows SDL’s configure handler ignores the configure size and restores window->floating (1x1): wp_viewport.set_destination(1, 1) + set_min_size(1,1) / set_max_size(1,1) on every following configure (even pure focus changes). Everything Slate renders lands in a 1x1 viewport.

Upstream SDL fixed exactly this on main in libsdl-org/SDL@fcb5ef8a9c “wayland: Handle odd behavior with non-resizable windows in certain tiling compositors”. It is not in any 3.4.x release yet (checked up to release-3.4.18).

Request: cherry-pick fcb5ef8a9c into Engine/Source/ThirdParty/SDL3 (SDL-gui-backend), or update to an SDL that contains it.

Evidence (WAYLAND_DEBUG=client, “Create Animation Blueprint”, no compositor window rules):

-> xdg_toplevel#1504.set_min_size(1, 1) / set_max_size(1, 1)   # created fixed-size at 1x1
<- xdg_toplevel#1504.configure(1916, 2117, array[16])          # first configure: tile-sized slot
-> wp_viewport#1454.set_destination(430, 539)                  # Slate resizes to content
-> xdg_toplevel#1504.set_min_size(430, 539) / set_max_size(430, 539)
-> xdg_toplevel#1504.set_parent(...); xdg_dialog_v1.set_modal() # dialog role only now
<- xdg_toplevel#1504.configure(430, 539, array[20])            # compositor floats it at the right size
-> wp_viewport#1454.set_destination(1, 1)                      # SDL restores stale floating size
-> xdg_toplevel#1504.set_min_size(1, 1) / set_max_size(1, 1)   # repeated on every configure

Verified workaround: an LD_PRELOAD shim interposing SDL_SetWindowSize (exported from libUnrealEditor-ApplicationCore.so): for non-resizable, non-popup windows it marks them resizable with min = max = requested size, so the configure size is honored and the window stays effectively fixed. All affected dialogs then render. XWayland also avoids it but performs much worse here.

System: UE 5.8.3 (installed binary build), Vulkan RHI (SM6), MAILBOX present mode; Arch Linux (Omarchy), kernel 7.2.5, Hyprland 0.56.2 (Wayland); NVIDIA GeForce RTX 5090 Laptop GPU, driver 610.57.04 (PRIME offload). Independent of explicit sync (__NV_DISABLE_EXPLICIT_SYNC=1 changes nothing) and of present mode.

Related (hang variant with the same log line): Editor hangs forever in Linux upon opening modal dialogs, with a message about Vulkan swapchains being outdated.

What Type of Bug are you experiencing?

Platform Other

Steps to Reproduce

  1. Linux, native Wayland session with a tiling compositor (reproduced on Hyprland 0.56.2); SDL_VIDEO_DRIVER=wayland (SDL3 default on Wayland).
  2. Launch UnrealEditor 5.8.3 and open any project.
  3. Content Browser > right click > Animation > Animation Blueprint (or open any Yes/No message dialog, e.g. via Python: unreal.EditorDialog.show_message(“Test”, “Text”, unreal.AppMsgType.YES_NO)).

Expected Result

The dialog renders its content at its requested size (as it does under XWayland, and as fixed-size dialogs like “Delete Assets” already do under Wayland).

Observed Result

The dialog opens as an empty, see-through frame at the right size; nothing is ever drawn inside it. Resizing or moving it does not help; Enter/Esc still act on it blind. The log repeats “AcquireNextImage() failed due to the outdated swapchain”. WAYLAND_DEBUG shows SDL setting the dialog’s viewport destination and min/max size to 1x1 after every configure (see Summary).

Affects Versions

5.8

Platform(s)

Linux

Additional Notes

Fixed-size dialogs are unaffected because they are created at their real size, so window->floating is correct from the start. The upstream fix stores the new size in window->floating when a non-resizable window has ended up tiled (SDL_waylandwindow.c, Wayland_SetWindowSize). Happy to share the full WAYLAND_DEBUG log or the LD_PRELOAD shim source if useful.