How to do Gauntlet tests on Linux?

Cross compiling and running Gauntlet from Linux should work.

Gauntlet can test a desktop platform only from that platform. So running from Linux to test Linux is the only way.

As for the ground truth, you can use the platform screenshot fallback configuration to make the screenshot comparison use the Win/Dx12 references.

That is by adding these lines in the DefaultEngine.ini file:

[/Script/ScreenShotComparisonTools.ScreenShotComparisonSettings] ; Screenshots for proprietary platforms will be stored under Project/Platforms/Test/Screenshots instead of Project/Test/Screenshots ;Use format +FallbackPlatform=(Child=Platform/RHI, Parent=Platform/RHI) +ScreenshotFallbackPlatforms=(Child="Windows/D3D12_SM5", Parent="Windows/D3D11_SM5") +ScreenshotFallbackPlatforms=(Child="Windows/D3D12_SM6", Parent="Windows/D3D11_SM5") +ScreenshotFallbackPlatforms=(Child="Windows/Vulkan_SM5", Parent="Windows/D3D11_SM5") +ScreenshotFallbackPlatforms=(Child="Windows/Vulkan_SM6", Parent="Windows/D3D12_SM6") +ScreenshotFallbackPlatforms=(Child="Linux/Vulkan_SM5", Parent="Windows/D3D11_SM5") +ScreenshotFallbackPlatforms=(Child="Linux/Vulkan_SM6", Parent="Windows/D3D12_SM6")This will be true for any platform, not just the desktop platforms.

Also this does not prevent having specific alternate ground truth a particular comparison for a particular platform.

The fallback only trigger if no matching platform ground truth is found and the dependencies fallback will resolve in order to find a suitable ground truth.