Automation tests failing on 'Initialization Failure' in 5.7.3 because artifacts deleted prematurely

Development Pipeline & Plugins
, ,
1

I found that in E:\Repos\Triton_Unreal\UE5\Engine\Source\Programs\AutomationTool\Gauntlet\Unreal\Base\Gauntlet.UnrealTestNode.cs,

function StopTests,

SaveRoleArtifacts is now deleting the E:\Repos\Triton_Unreal\UE5\GauntletTemp\DeviceCache\Win64\LocalDevice0\UserDir\Saved directory deleted before it is parsed in the CreateRoleResultsFromArtifacts function.

When I debug inside CreateLogSummaryFromArtifacts, I found the ProcessLogFile of the AppInstance had been deleted

(eg. E:\Repos\Triton_Unreal\UE5\GauntletTemp\DeviceCache\Win64\LocalDevice0\UserDir\Saved\Logs\AcousticsGame.log)

The fallback to use the ProcessLogOutput instead is insufficient as the buffer does not contain the required string to validate the EngineInitialized.

public string EngineInitializedPattern = @“LogInit.+Engine is initialized\.”;

The root cause seems to be a change at //UE5/Release-5.7 CL 46399096 to:

E:\Repos\Triton_Unreal\UE5\Engine\Source\Programs\AutomationTool\Gauntlet\Unreal\Base\Gauntlet.UnrealSession.cs,

Function SaveRoleArtifacts

At Line 1155, I believe the refactored logic is incorrect:

if (SkipArchivingAssets && !bRetainArtifacts && SourceDirectory.Exists)

In 5.6, the effect of the nested logic would have yielded the following instead:

if (!SkipArchivingAssets && SourceDirectory.Exists && !bRetainArtifacts)

These are some things that did work for us:

  1. Modifying this line 1155 to the logic allowed our tests to pass.
  2. Reverting to Gauntlet.UnrealSession.cs to CL 45349090
  3. Adding parameter -RetainDeviceArtifacts to RunUAT commandline params (but we didn’t have to do this in previous versions). Is this necessary now?

I also noticed you’ve modified the function at 45733620 on //UE5/Main/; however, these changes don’t address the issue I’m observing. When DirectoryToCleanUp == null, it falls through the same logic.

[Attachment Removed]

2

Steps to Reproduce
After updating to 5.7, our tests started failing on an ‘Initialization Failure’ despite our Unreal clearly initializing and passing most tests.

eg.

RunUAT.bat RunUnreal -Project={uProjectName} -Platform=Win64 -Configuration=Development -build=Editor -test=UE.EditorAutomation -runtest={testGroupName} -ReportExportPath={testOutput} -NullRHI -MaxDuration=1200 -VS2022

Comparing: 5.6 output

[Image Removed]

vs. 5.7 output

[Image Removed]

[Attachment Removed]

3

This case has a fix for this issue:

[Content removed]

Looks like the issue was known and fixed but it did not make it to 5.7.

Here a patch to fix that:

Change 46418278 by Jerome.Delattre on 2025/10/01 09:29:54
 
	Update the process log file path when written to file/copy
	to still reference a valid log file path
 
Affected files ...
 
... //UE5/Main/Engine/Source/Programs/AutomationTool/Gauntlet/Framework/Base/Gauntlet.TargetDeviceDesktopCommon.cs#25 edit
 
Differences ...
 
==== //UE5/Main/Engine/Source/Programs/AutomationTool/Gauntlet/Framework/Base/Gauntlet.TargetDeviceDesktopCommon.cs#25 (text) ====
 
[Content removed]8 @@
 				return new LogFileReader(ProcessLogFile);
 			}
 
+			Log.Warning(KnownLogEvents.Horde_BuildHealth_Ignore, "Log file '{filepath}' is missing. The buffer will be used instead but will most likely lack the beginning of the file.", ProcessLogFile);
+
 			return ProcessLogOutput.GetReader();
 		}
 
[Content removed]16 @@
 			{
 				ProcessUtils.CheckProcessLogReachedSizeLimit(new FileReference(ProcessLogFile));
 				File.Copy(ProcessLogFile, FilePath, true);
+				if (HasExited)
+				{
+					// Follow the new location
+					// SaveRoleArtifacts may clean device cache folder prior parsing the log file
+					ProcessLogFile = FilePath;
+				}
 			}
 			else
 			{
-				Log.Warning("Log file '{filepath}' is missing at the time of making a copy. The buffer will be used instead but will most likely lack the beginning of the file.", ProcessLogFile);
+				Log.Warning(KnownLogEvents.Horde_BuildHealth_Ignore, "Log file '{filepath}' is missing at the time of making a copy. The buffer will be used instead but will most likely lack the beginning of the file.", ProcessLogFile);
 				StreamWriter Writer = ProcessUtils.CreateWriterForProcessLog(FilePath, CommandLine);
 				foreach(string Line in ProcessLogOutput)
 				{

[Attachment Removed]