we are currently deploying on Windows only, but want to deploy to Linux in the Future as well. Packaging for Linux works but we have some troubles how to execute the functional tests via Gauntlet.
For windows we let the Unreal Editor run on the same Cloud instance than the packaged project.
Now we want to double check if this is the correct way under Linux:
Compile/Package the Linux game target version with a Windows build machine and the cross compiler
Compile the Linux editor target on a Linux Instance with clang
On a Linux instance start the tests via gauntlet
Is this the way to go?
Another question is how to handle the ground truth images for both platforms / RHIs ? We actually want to keep them in sync with Windows + D3D12 since we expect it to be visually the same. How can we archive this?
Cross compiling and running Gauntlet from Linux should work.
Gauntlet can test a desktop platform only from that platform. So running from Linux to test Linux is the only way.
As for the ground truth, you can use the platform screenshot fallback configuration to make the screenshot comparison use the Win/Dx12 references.
That is by adding these lines in the DefaultEngine.ini file:
[/Script/ScreenShotComparisonTools.ScreenShotComparisonSettings] ; Screenshots for proprietary platforms will be stored under Project/Platforms/Test/Screenshots instead of Project/Test/Screenshots ;Use format +FallbackPlatform=(Child=Platform/RHI, Parent=Platform/RHI) +ScreenshotFallbackPlatforms=(Child="Windows/D3D12_SM5", Parent="Windows/D3D11_SM5") +ScreenshotFallbackPlatforms=(Child="Windows/D3D12_SM6", Parent="Windows/D3D11_SM5") +ScreenshotFallbackPlatforms=(Child="Windows/Vulkan_SM5", Parent="Windows/D3D11_SM5") +ScreenshotFallbackPlatforms=(Child="Windows/Vulkan_SM6", Parent="Windows/D3D12_SM6") +ScreenshotFallbackPlatforms=(Child="Linux/Vulkan_SM5", Parent="Windows/D3D11_SM5") +ScreenshotFallbackPlatforms=(Child="Linux/Vulkan_SM6", Parent="Windows/D3D12_SM6")This will be true for any platform, not just the desktop platforms.
Also this does not prevent having specific alternate ground truth a particular comparison for a particular platform.
The fallback only trigger if no matching platform ground truth is found and the dependencies fallback will resolve in order to find a suitable ground truth.