Running the Compile Blueprints test via commandlet does not work correctly when using the SetFilter option.

Development Pipeline & Plugins
, ,
1

Context

We are trying to run the Compile Blueprints test, which is part of Unreal’s Test Automation framework, outside of the engine to integrate it into our CI/CD pipeline. While the test runs successfully inside the editor, executing it via a Python script externally does not behave as expected.

Observed Behavior

The test Project.Blueprints.Compile Blueprints is categorized under the Stress Test filter. To locate and run this test, the filter must be explicitly set using Automation SetFilter Stress. However, when both SetFilter and RunTest are included in the same ExecCmds line, it appears that Automation RunTest overrides the filter and resets it to Standard Tests. This causes the test to not be found.

Example Command and Log Output

“-ExecCmds=\“Automation SetFilter Stress, Automation RunTest Project.Blueprints.Compile Blueprints;\””

When run without Automation RunTest, the filter is correctly applied, and the log shows 39 additional tests—close to the expected number seen in the Automation tab. The source of these extra tests is unclear.

Request for Clarification

Could you clarify:

  • How to run a test in the Stress category directly from the command line?
  • Whether this behavior is a known limitation or bug in Unreal’s automation system?
2

Steps to Reproduce

  1. Enable Editor Tests Plugin for project located in the Testing tab in the plugins window
  2. Restart the engine for the plugin to get added
  3. Create a python script inside the project folder by copying the code I sent with your correct path to engine and project
  4. Run python script
  5. The Unreal Project Browser will most likely open, select the project you’ve specified the path to

`import subprocess

engine = “path/to/UnrealEditor.exe”
project = “path/to/your/project.uproject”

cmd=[
engine,
project,
“-editortest”,
“-ExecCmds="Automation SetFilter Stress, Automation RunTest Project.Blueprints.Compile Blueprints"”,
#“-TestExit=Automation Test Queue Empty”, #Has been disabled to not exit the editor after running the test
]

cmd = " ".join(cmd)
subprocess.run(cmd)`

3

Video showing step 4 and 5

4

`import subprocess

engine = “path/to/UnrealEditor.exe”
project = “path/to/project.uproject”

cmd=[
engine,
project,
“-editortest”,
“-ExecCmds="Automation SetFilter Stress"”,
#“-TestExit=Automation Test Queue Empty”, #Has been disabled to not exit the editor after running the test
]

cmd = " ".join(cmd)
subprocess.run(cmd)`Code and video showing “-ExecCmds=\“Automation SetFilter Stress\””, removed Automation RunTest Project.Blueprints.Compile Blueprints

5

The command line argument `-TestExit=` is legacy and should not be used.

Instead use the `Quit` in side `-ExecCmds=Automation`.

Also queuing commands with `-ExecCmds=Automation` is done by using the `;` separator.

For example: `-ExecCmds=“Automation RunTest MyTest;Quit”` is the valid syntax to have the test automation to run tests and then quit.

To go back to the original post, the proper command line to set filters, run tests and quit should be:

-ExecCmds="Automation SetFilter Stress;RunTest Project.Blueprints.Compile Blueprints;Quit"I’m not aware what `-editortest` could be used for.