How to setup and configure automated screenshot comparison testing in Horde?

Development Pipeline & Plugins
, ,
1

We are attempting to run the screenshot comparison testing against our project. The experience reported so far is that in editor the test captures mostly white screens rather than the environment we are trying to capture, and when run via Horde we don’t get the comparison part. We have fixed one issue which is the need to run the Horde agent in interactive mode, so that is implemented now. However, the jobs continue to fail. No reports are generated, and the screenshot comparison UI in the Automation Hub never populates. At this point, after the various things tried, I can only conclude that our implementation is fundamentally flawed somehow. I am attaching the build graph scripts and a log from running the job, the automated testing code that I know about, and the functional testing map we are using. Unfortunately I only wrote the build graph, so I will have to relay messages about the testing code and assets. The build graph script in question is Graph\AutomatedTesting\TestGameLighting.xml that is using TNM1.Content.TNM1.Maps.Test.QA\LumenScreenshots\TheShed\P_Greenhouse_LumenTest.umap to run the test.

[Attachment Removed]

2

Steps to Reproduce
I’m afraid the repro is a bit too tangled to isolate effectively for this purpose.

[Attachment Removed]

3

Hello! Just so that you’re aware, it looks like there are some unredacted API keys in “++Main+vmain_eng-CL 1458205-Run Lighting Test Agent.log”.

Would it be possible for you to run the test again with the controller’s logging verbosity increased to VeryVerbose, and then share the resulting ClientOutput.log?

You should be able to drop `-LogCmds=“LogTNMAutomatedScreenshotGauntletController VeryVerbose”` into TestGameLighting.xml alongside the existing LogCmds argument.

The main agent log reported 201 errors and 128 warnings from the client, but the client log itself wasn’t attached for review.

[Attachment Removed]

4

Testing file attachment.

[Attachment Removed]

5

Thanks for sending these over! I can’t be 100% sure this is the same case as the first run, but here, there appears to be an argument quoting issue.

If you check RunUnreal/Log.txt at line 3, you can see that a quotation mark comes in before the word “verbose” from the “-LogCmds= all verbose” argument.

The whole section after this quotation mark then appears to be dropped on line 7. Importantly, the “-Map” argument is part of the dropped section.

ClientOutput.log displays the command line arguments that it received on line 2, which doesn’t include the -Map, -RenderOffScreen, or several others.

The default map is likely loaded, which would explain why the test errors out at line 2423 with the following error:

[2026.09.04-21.00.40:462][ 0]LogTNMAutomatedScreenshotGauntletController: Error: UTNMAutomatedScreenshotGauntletController::LateInit tried to collect ACineCameraActors in UWorld and found none! Exiting Test

Would you be able to try switching to the config (CVar) syntax for logging, as shown in this article, so you don’t need to work around the quotes?

-ini:Engine:[Core.Log]:LogTNMAutomatedScreenshotGauntletController=VeryVerbosehttps://dev.epicgames.com/community/snippets/3GoB/unreal\-engine\-how\-to\-set\-log\-verbosity\-via\-command\-line\-with\-logcmds

[Attachment Removed]

6

apologies for the delay, there was an issue with PSO caching to sort through that was breaking the job. Here are the logs from the new run

[Attachment Removed]

7

No worries at all! Unfortunately, it looks like these have the same issue with the log commands breaking the argument parsing.

In RunUnreal/Log.txt, you can see on line 6 that the arguments end abruptly after “-LogCmds=all)”, dropping the map and logging override.

In ClientOutput.log, the logs show that the default map is loaded, which doesn’t appear to have any ACineCameraActors.

[2026.09.11-18.42.34:753][  0]LogGlobalStatus: UEngine::LoadMap Load map complete /Game/TNM1/Maps/FrontEnd/L_FrontEnd
...
[2026.09.11-18.42.36:647][  0]LogTNMAutomatedScreenshotGauntletController: Error: UTNMAutomatedScreenshotGauntletController::LateInit tried to collect ACineCameraActors in UWorld and found none!  Exiting Test

Could you please review TestGameLighting.xml to ensure that you’ve removed the “-LogCmds="all verbose"” from the AdditionalClientArgs option?

[Attachment Removed]

8

I apologize. We have a milestone to hit today, so I’m trying to fit this where I can. I made sure it was removed and run again.

[Attachment Removed]

9

No problem at all from my side, thanks for getting these across.

It looks like a good (failing) run (i.e., the “P_Greenhouse_LumenTest” map was loaded at RunUnreal/Log.txt:#333, and the UTNMAutomatedScreenshotGauntletController logs show that the screenshotting occurs).

Some observations:

1. The ClientOutput logs show that the screenshots are saved to “UserDir/Saved/Profiling/Moonlog/<guid>/screenshot.png”.

2. The attachments don’t include the index.json report that Gauntlet requires, which we should verify is generated correctly.

3. The RunUnreal\Log.txt shows three values for ReportExportPath, set in Graph\AutomatedTesting\TestGameLighting.xml

- “H:\horde\main\Sync\UE5\TNM1\Saved\Automation\Reports\Gauntlet”

- “H:\horde\main\Sync\UE5\TNM1\Saved\Automation\Reports\Editor”

- “H:\horde\main\Sync\UE5\TNM1\Saved\Automation\Reports\Client”

Could you please confirm where on the agent the report is located?

Could you please confirm whether “TNM1\Saved\Automation\” contains “Gauntlet.TestData.json” after a run?

Could you please check whether “Engine\Programs\AutomationTool\Saved\TestData\” is empty?

What I think might be happening is that the data is being generated, but not in the location that is expected.

These checks should tell us whether the data is being created in the wrong place, or if it’s not being created at all.

The last two checks are in there because TNMScreenshotBaseTestNode.cs sets HordeTestDataPath to “TNM1\Saved\Automation”, but Horde by default expects step-testdata artifacts in the UE_TESTDATA_DIR directory.

"step-testdata: Data written to the directory given by the UE_TESTDATA_DIR environment variable by a build step (normally Engine/Programs/AutomationTool/Saved/TestData).

Json files in this directory will also be harvested for test data results and surfaced to Automation Hub."

https://github.com/EpicGames/UnrealEngine/blob/5\.6\.1\-release/Engine/Source/Programs/Horde/Docs/Config/Artifacts.md\#L54

[Attachment Removed]

10

New run attached with job and step logs

[Attachment Removed]