UE5.3 Does not find automation tests through command line

Development Programming & Scripting
,
1

I may have found a bug on Unreal Engine 5.3. Didn’t find any of this in the Issue Tracker.

I’m currently running automated tests through the command line, using the command:

“%UE5.3Folder%\Engine\Binaries\Win64\UnrealEditor-cmd.exe” “%MyProjectPath%” “%MyProjectPath%*.uproject” -nosplash -Unattended -nopause -nosound -NullRHI -nocontentbrowser -ExecCmds=“Automation RunTests Project.Functional Tests.;quit” -TestExit=“Automation Test Queue Empty” -ReportOutputPath=“%ReportFolder%” -log -Log=RunTests.log

However, the command line results in "LogAutomationCommandLine: Error: No automation tests matched ‘Project.Funcional Tests.’ ". I`m currently upgrading the project from 5.1 to 5.3, so I know this works on 5.1, but not on 5.3. If I issue the following command to see the list of automation tests (-ExecCmds=“Automation List;quit”):

“%UE5.3Folder%\Engine\Binaries\Win64\UnrealEditor-cmd.exe” “%MyProjectPath%” “%MyProjectPath%*.uproject” -nosplash -Unattended -nopause -nosound -NullRHI -nocontentbrowser -ExecCmds=“Automation List;quit” -TestExit=“Automation Test Queue Empty” -ReportOutputPath=“%ReportFolder%” -log -Log=RunTests.log

My tests are not on the list. Hence, UE5.3 does not find it.

To reproduce this bug you can:

  1. Create a brand new project on UE5.3

  2. Create a functional test BP and add to a map

  3. Save the functional test and the map. Close unreal.

  4. Run the command:
    “%UE5.3Folder%\Engine\Binaries\Win64\UnrealEditor-cmd.exe” “%MyProjectPath%” “%MyProjectPath%*.uproject” -nosplash -Unattended -nopause -nosound -NullRHI -nocontentbrowser -ExecCmds=“Automation List;quit” -TestExit=“Automation Test Queue Empty” -ReportOutputPath=“%ReportFolder%” -log -Log=RunTests.log

  5. Note the funcional test is not on the list of tests.

Repeat the same process for UE5.1 and you will see that the test is indeed found by the command. Hence, a bug in UE5.3.

Im missing something? There seems to have no update on this on UE5.2 and UE5.3 release notes. I dug into Unreals engine code for this and it didn`t seem to have big changes to break such thing.

Thanks in advance

2

Any inputs on this?

3

So I did find a solution. It seems that the second argument passed on the command line (“%MyProjectPath%”) does something that confuses the Unreal’s cmd. I noticed that when I ran the command with UE5.3, it was searching for all projects on my computer. Maybe it is trying to find the target project on the second argument, and not on the third. Thus, removing it does the trick.

Kind odd that this works on UE5.1 and not on UE5.2 nor UE5.3. Maybe devs should take a look on what has changed.