[Bug] SDL_Window* parameter being NULL when calling SDL_Vulkan_CreateSurface (with -RenderOffscreen)

[Edvard Fagerholm]

When trying to run the engine under Linux with the Vulkan backend and the -RenderOffscreen command line flag, the engine ends up passing a nullptr as the SDL_Window* into SDL_Vulkan_CreateSurface. This happens in VulkanLinuxPlatform.cpp:

void FVulkanLinuxPlatform::CreateSurface(void* WindowHandle, VkInstance Instance, VkSurfaceKHR* OutSurface)
{
    EnsureSDLIsInited();

    if (SDL_Vulkan_CreateSurface((SDL_Window*)WindowHandle, Instance, OutSurface) == SDL_FALSE)
    {
        UE_LOG(LogInit, Error, TEXT("Error initializing SDL Vulkan Surface: %s"), SDL_GetError());
        check(0);
    }
}

[Edvard Fagerholm]

Here's the command that I'm using to launch the engine:

Engine/Binaries/Linux/UE4Editor /home/efagerho/git/UE4Upstream/Samples/StarterContent/StarterContent.uproject -game -MovieSceneCaptureType="/Script/MovieSceneCapture.AutomatedLevelSequenceCapture" -LevelSequence="/Game/Sequence" -MovieFolder="/home/efagerho/ue4/output/" -NoLoadingScreen -MovieFormat=JPG -MovieQuality=100 -ResX=1920 -ResY=1080 -vulkan -RenderOffscreen

I've added a trivial Sequence into the StarterContent project, so that I could record some data. It's pretty clear that this hits a code path that hasn't been tested at all. Basically, if the RenderOffscreen flag is being provided, then in SlateApplication.cpp we see that MakeWindow contains the following:

    if (bRenderOffScreen)
    {
        TSharedRef< FGenericWindow > NewWindow = MakeShareable(new FGenericWindow());
        InSlateWindow->SetNativeWindow(NewWindow);
        return NewWindow;
    }

However, the implementation of GetOSWindowHandle() in FGenericWindow is simply simply "return nullptr;". This is what is given to the CreateSurface() method above.

[Edvard Fagerholm]

I ended up implementing fully headless rendering myself over the last couple of days. In other words, I rewrote some of the Vulkan Linux platform code and made use of the following Vulkan layer:

https://gitlab.freedesktop.org/mesa/vulkan-wsi-layer

Basically, this allows me to create a fake VkSurfaceKHR object that I can pass to the engine, so that headless rendering can be implemented without having to touch larger parts of the Vulkan renderer.