Updating UE5 to work on Wayland. I got it working, sort of, mostly. Swapchain issues.

The Pull request.

https://github.com/EpicGames/UnrealEngine/pull/13764

I started by adding mouse wheel tilt support, which I’m not 100% sure if it works because I can barely get past the swapchain crashing. But it’s in there now, giving people mouse wheel tilt access. I then started modifying code for wayland support.
LinuxWindow.cpp was near a complete rewrite to get things loaded in the correct order so that the correct window type would be created. The previous code seemed to be more made up of multiple layers of patching and was a little hard to read through so that section got a rewrite as well.

I’m mainly posting this because, I’ve got the basics all there now. I’m just unable to figure out what’s causing the swapchain crashing. It seems to be related to rapid tooltip window creation and deletion but also randomly happens on other window types.

Progress update:

I’ve managed to solve the swapchain crashes. I’ll have to come up with a better fix for now because using SDL_Delay is not ideal since it’s now adding a literal second to two every time you want to open a menu or hover over a selection. The issue seems to have been windows getting created and destroyed faster than the compositor could keep up with it, along with windows getting rendering calls that weren’t ready to receive them yet along with a litany of other things. Safety checks have been added to prevent bad things from happening now along with more logging to help pinpoint other issues.
I also managed to break tooltips entirely now so I got to figure out what to do about that. And, I also need to look into the random slate crash when using the new mouse wheel tilt feature, at least I was able to get far enough to test it more thoroughly this time.

Sucess! Finally! I got it working correctly lol. And, no swapchain crashing when moving the mouse around on it.
Now, I need to solve the 1ms-1second delay on tooltips in the Outliner area of the editor.

Next on the list:
Application doesn’t stay maximized when it’s restarted. It seems to be 1 pixel off from maximize and in the “restored down” state

The Notification Window with yes/no buttons is still showing outside the application and isn’t locked to it’s correct position. And is not interactable unless the application happens to start in a position where that window can’t go outside the monitor and is forced to be where it’s supposed to be.

Drag/Drop Window docking needs to be updated to work on wayland.

The mousewheel tilt crash seems to only happen with On mouse wheel tilt Left.

And probably some other things either don’t remember at the moment or will find as I’m testing it.

I’m still fighting the following error with tooltips. It’s fine if the mouse is slowly/deliberately moving from one entry to the next in the level outliner, but the moment I scroll wheel through the list it all breaks.

Assertion failed: NumFormats > 0 [File:./Runtime/VulkanRHI/Private/VulkanSwapChain.cpp] [Line: 212]

libUnrealEditor-Core.so!FDebug::CheckVerifyFailedImpl2(char const*, char const*, int, char16_t const*, …) [/home/cecil-merrell/UnrealEngine/UnrealEngine-UE5-MAIN-FORK/Engine/Source/./Runtime/Core/Private/Misc/AssertionMacros.cpp:755]
libUnrealEditor-VulkanRHI.so!FVulkanSwapChain::Create(VkInstance_T*, FVulkanDevice&, EPixelFormat&, unsigned int, unsigned int, bool, unsigned int*, TArray<VkImage_T*, TSizedDefaultAllocator<32> >&, signed char, FVulkanGenericPlatformWindowContext&, FVulkanSwapChainRecreateInfo*) [/home/cecil-merrell/UnrealEngine/UnrealEngine-UE5-MAIN-FORK/Engine/Source/./Runtime/VulkanRHI/Private/VulkanSwapChain.cpp:212]
libUnrealEditor-VulkanRHI.so!FVulkanViewport::CreateSwapchain(FVulkanCommandListContext&, FVulkanSwapChainRecreateInfo*, FVulkanGenericPlatformWindowContext&) [/home/cecil-merrell/UnrealEngine/UnrealEngine-UE5-MAIN-FORK/Engine/Source/./Runtime/VulkanRHI/Private/VulkanViewport.cpp:438]
libUnrealEditor-VulkanRHI.so!FVulkanViewport::RecreateSwapchain(FVulkanCommandListContext&, FVulkanGenericPlatformWindowContext&) [/home/cecil-merrell/UnrealEngine/UnrealEngine-UE5-MAIN-FORK/Engine/Source/./Runtime/VulkanRHI/Private/VulkanViewport.cpp:321]
libUnrealEditor-VulkanRHI.so!FVulkanViewport::DoCheckedSwapChainJob(FVulkanCommandListContext&) [/home/cecil-merrell/UnrealEngine/UnrealEngine-UE5-MAIN-FORK/Engine/Source/./Runtime/VulkanRHI/Private/VulkanViewport.cpp:219]
libUnrealEditor-VulkanRHI.so!FVulkanViewport::Present(FVulkanCommandListContext&, bool) [/home/cecil-merrell/UnrealEngine/UnrealEngine-UE5-MAIN-FORK/Engine/Source/./Runtime/VulkanRHI/Private/VulkanViewport.cpp:765]
libUnrealEditor-VulkanRHI.so!FVulkanCommandListContext::RHIEndDrawingViewport(FRHIViewport*, FRHIPresentArgs const&) [/home/cecil-merrell/UnrealEngine/UnrealEngine-UE5-MAIN-FORK/Engine/Source/./Runtime/VulkanRHI/Private/VulkanRHI.cpp:1269]
libUnrealEditor-RHI.so!FRHICommand<FRHICommandEndDrawingViewport, FRHICommandEndDrawingViewportString2326>::ExecuteAndDestruct(FRHICommandListBase&) [/home/cecil-merrell/UnrealEngine/UnrealEngine-UE5-MAIN-FORK/Engine/Source/Runtime/RHI/Public/RHICommandListCommandExecutes.inl:562]
libUnrealEditor-RHI.so!FRHICommandListBase::Execute() [/home/cecil-merrell/UnrealEngine/UnrealEngine-UE5-MAIN-FORK/Engine/Source/./Runtime/RHI/Private/RHICommandList.cpp:541]
libUnrealEditor-RHI.so!FRHICommandListExecutor::FTranslateState::Translate(FRHICommandListBase*) [/home/cecil-merrell/UnrealEngine/UnrealEngine-UE5-MAIN-FORK/Engine/Source/./Runtime/RHI/Private/RHICommandList.cpp:1091]
libUnrealEditor-RHI.so!UE::Core::Private::Function::TFunctionRefCaller<FRHICommandListExecutor::FSubmitState::Dispatch(FRHICommandListBase*)::$_0, void>::Call(void*) [/home/cecil-merrell/UnrealEngine/UnrealEngine-UE5-MAIN-FORK/Engine/Source/./Runtime/RHI/Private/RHICommandList.cpp:1042]
libUnrealEditor-RHI.so!FRHICommandListExecutor::FTaskPipe::Execute(FRHICommandListExecutor::FTaskPipe::FTask*, TRefCountPtr const&) const [/home/cecil-merrell/UnrealEngine/UnrealEngine-UE5-MAIN-FORK/Engine/Source/Runtime/Core/Public/Templates/Function.h:414]
libUnrealEditor-Core.so!TGraphTask<TFunctionGraphTaskImpl<void (ENamedThreads::Type, TRefCountPtr const&), (ESubsequentsMode::Type)0> >::ExecuteTask() [/home/cecil-merrell/UnrealEngine/UnrealEngine-UE5-MAIN-FORK/Engine/Source/Runtime/Core/Public/Templates/Function.h:414]
libUnrealEditor-Core.so!UE::Tasks::Private::FTaskBase::TryExecuteTask() [/home/cecil-merrell/UnrealEngine/UnrealEngine-UE5-MAIN-FORK/Engine/Source/Runtime/Core/Public/Tasks/TaskPrivate.h:518]
libUnrealEditor-Core.so!FNamedTaskThread::ProcessTasksNamedThread(int, bool) [/home/cecil-merrell/UnrealEngine/UnrealEngine-UE5-MAIN-FORK/Engine/Source/Runtime/Core/Public/Async/TaskGraphInterfaces.h:493]
libUnrealEditor-Core.so!FNamedTaskThread::ProcessTasksUntilQuit(int) [/home/cecil-merrell/UnrealEngine/UnrealEngine-UE5-MAIN-FORK/Engine/Source/./Runtime/Core/Private/Async/TaskGraph.cpp:687]
libUnrealEditor-RenderCore.so!FRHIThread::Run() [/home/cecil-merrell/UnrealEngine/UnrealEngine-UE5-MAIN-FORK/Engine/Source/./Runtime/RenderCore/Private/RenderingThread.cpp:212]
libUnrealEditor-Core.so!FRunnableThreadPThread::Run() [/home/cecil-merrell/UnrealEngine/UnrealEngine-UE5-MAIN-FORK/Engine/Source/./Runtime/Core/Private/HAL/PThreadRunnableThread.cpp:25]
libUnrealEditor-Core.so!FRunnableThreadPThread::_ThreadProc(void*) [/home/cecil-merrell/UnrealEngine/UnrealEngine-UE5-MAIN-FORK/Engine/Source/Runtime/Core/Private/HAL/PThreadRunnableThread.h:187]
libc.so.6!UnknownFunction(0xa3d63)
libc.so.6!UnknownFunction(0x1373bb)

The upcoming sdl 3.4.0 has some things that may be useful here. For starters, the notifications can be changed from being separate windows attached to the main editor window to being passed to the system for the os to handle instead. They’ve finally added support for system notifications through the system tray → SDL3/CategoryTray - SDL Wiki . And notification windows through https://wiki.libsdl.org/SDL3/CategoryMessagebox. There’s no need to craft custom modal windows for this anymore.

After dedicating more time than initially planned to Unreal Engine’s Linux integration and learning more about how both Unreal and Wayland work, I’ve concluded that extending the existing LinuxApplication for Wayland support is no longer the right approach.

The current codebase is fundamentally designed with X11 in mind. While it’s technically possible to add Wayland functionality, doing so requires constantly working around assumptions that don’t apply to Wayland—such as global coordinates, client-controlled window placement, and cursor warping. Each adjustment introduces more conditionals and edge cases, complicating the code further. And when one thing got fixed, I’d find out later that it only hid the problem or made others worse.

So, instead of layering fixes onto an incompatible foundation, I’m stepping back to pursue a cleaner, more sustainable solution.

Rather than modifying LinuxApplication, I plan to treat Wayland as its own distinct platform by introducing a separate application layer—tentatively named FWaylandApplication. This approach leaves the X11 implementation untouched and prevents the existing Linux code from becoming cluttered with special cases.

This isn’t a radical departure; Unreal Engine already follows this pattern for other platforms. The goal is to give Wayland the dedicated attention it deserves, rather than treating it as an extension of X11.

Attempting to replicate X11-style behavior on top of Wayland creates unnecessary friction. A dedicated platform layer allows Wayland to function as intended, without forcing it into an incompatible mold.

A properly implemented Wayland path addresses a number of long-standing issues in a way that is both cleaner and easier to maintain than the current approach. Fullscreen behavior can finally align with compositor expectations instead of relying on exclusive or borderless hacks, which removes a whole class of focus and mode-switching problems. Input handling becomes more flexible and robust, allowing packaged games to release input cleanly without forcing players to Alt-Tab out of the application just to interact with something else on a second monitor — a workaround that often leaves input grabs in a broken or inconsistent state on Wayland. When input is released, the cursor can move naturally outside the window without warping or confinement tricks, making multi-monitor use feel intentional rather than accidental. Multi-monitor support itself becomes simpler and more predictable, since the compositor manages output placement and fullscreen surfaces instead of the engine trying to infer global coordinates. UI elements such as tooltips, popups, and notifications can be implemented using proper parent-child relationships, which avoids positioning glitches and focus issues. HiDPI and fractional scaling can be handled correctly through compositor-provided scale factors rather than guesswork, and HDR can be supported where the compositor exposes the necessary protocols. Because the Unreal Editor and packaged builds share the same platform layer, these improvements apply consistently across both, rather than existing only in editor-specific code paths or one-off fixes.

This shift will likely require refinements to how Slate manages windows and popups. Certain behaviors that were implicit under X11 may need to become explicit and platform-aware. The aim is to enhance Slate itself rather than creating a hard fork, though some breaking changes may be necessary along the way.

SDL will remain useful for certain tasks, such as input handling and possibly notifications along with other things that each distribution handles differently rather than it being just a wayland thing. However, relying on it for windowing and fullscreen management under Wayland has limitations. Native Wayland support provides greater control and avoids dependencies on abstraction layers that may lag behind.

The immediate scope of this work is to get the Unreal Editor running cleanly and predictably on Wayland without relying on compatibility layers or workarounds. That starts with being able to launch the editor with correct input handling and rendering from the outset, followed by establishing proper windowing, focus, and fullscreen behavior that respects the compositor instead of fighting it. From there, the goal is to support multi-window editor workflows in a way that feels natural under Wayland, including detached panels and secondary windows on multi-monitor setups. Because the editor and packaged games share the same underlying platform layer, any fixes or improvements made here should automatically carry over to packaged builds, ensuring games benefit from the same behavior without requiring separate, game-specific logic.

Longer term, this foundation could enable Unreal Engine to be used more effectively for non-gaming applications on Wayland, though that remains a future consideration.

This effort is less about adding new features and more about building the right foundation. Continuing on the previous path would only increase complexity without addressing core incompatibilities. A dedicated Wayland platform layer offers a clearer, more sustainable path forward.

I’ll share further updates as the work progresses.

2 Likes