Greetings,
In BuildTools presentation (link) I found that I can run tests with command line. Can someone provide an example please?
Hello,
To run all of the automation tests in a project:
UE4Editor.exe GAMENAME -Game -ExecCmds=“Automation RunAll”
or
To run a single test:
UE4Editor.exe GameName -game -execcmds=“Automation Run FSetRestTest”
You can find the list of tests by running -execcmds=“Automation List”. In the output log you’ll see a long list of tests. Each test name has the letter “F” in front of it.
Here’s some more info on the automation system:
Creating a test:
Overview:
1 Like
How would you execute a group of tests (or indeed even a complex test like FCompileBlueprintsTest) Obviously don’t want to have to list in the command line all the Blueprints… Alternatively would be a way to trigger a preset collection of auto tests?
To run a preset collection through command line is not currently possible but it is on our list of things to do.
You can run the individual tests from a complex test or you can run 'em all.
To run an individual test use this command argument:
-AutomationTests=“Pretty.TestName”
Example:
-AutomationTests="Blueprints.Compile Blueprints.My Awesome Blueprint, “Blueprints.Compile Blueprints.MyOKBlueprint”
To run all of a complex test then use:
-Execcmds=“Automation Run FCompileBlueprintsTest”
Does this answer your question?
When I run with -Execcmds="Automation Run FCompileBlueprintsTest" I get
ERROR: Failed to find test FCompileBlueprintsTest
I have double checked the name of the test (copied and pasted from a list command). The only way I can get it to work is by specifying a specific BP. We are using UE4.4.
By chance, are you attempting to run the blueprint test in the game? That test is flagged as Editor only so it won’t work if you run it in game.
I am not running with the -game tag and it correctly launches the editor.
I’m not sure how I ever got -Execcmds="Automation Run FCompileBlueprintsTest /some/path/to/blueprint" from the command line to work as even that doesn’t work (even that says can’t find test when the editor launches).
However, if I run Automation Run FCompileBlueprintsTest /some/path/to/blueprint in the command line from the Session Frontend, that works. But even then, running Automation Run FCompileBlueprintsTest says it can’t find that test.
My apologies I was wrong about the -execcmd being able to easily run all of the complex tests for an automation test. 
You’ll wan to run with: -automationtests=“Blueprints.Compile-On-Load”
The couple of issue with that is the editor/game will exit once the automation test que is empty which is fine if that is what you want it to do. This also may or may not run all of the available complex tests.
I’ve logged these issues into our database to hopefully make this easier to use.
To get a full name of a complex test you can run the editor or game with -Execcmds=“Automation List” or in the editors console window entering Automation List. That will give you the full list.
-ExecCmds="Automation Run FCompileBlueprintsTest" does not work. -AutomationTests="Blueprints.Compile Blueprints" does not work. But -AutomationTests="Project.Blueprints.Compile Blueprints" does work. That’s the PrettyName for FCompileBlueprintsTest, as defined in BlueprintAutomationTests.cpp.
You are correct Mike. Recently there have been some minor changes to how automation is run through the command line.
“-ExecCmds=“Automation Run FCompileBlueprintsTest””
In this example you’ll need to also specify the asset that is being tested. You can easily find the whole path for that file by running Automation List either as an execcmd or in the console window. The print out will be in your output log.
“-AutomationTests=“Blueprints.Compile Blueprints””
Even though the automationtests= command line can use partial arguments you’ll need to include everything before the test name itself. For example if I just wanted to run all of the blueprints it would look like this: "-automationtests=“Project.Blueprints” or if I wanted to run compile all blueprints that start with the letter A I would use this: -automationtests=“Project.Blueprints.Compile Blueprints.A”.
Heads up though, this will be changing in the next release. For the better. 
Hey just a heads up, I do not believe that your answer works with the current version of Unreal.
I believe it was updated, but this is what I’ve found to work.
What you want to do is to call the following for specific tests from your Engine/Binaries/Win64 directory (or whatever directory your UE4Editor.app… etc is in):
UE4Editor.exe "[PathFromUnrealEditorBinaryToYourProjectDirectory]/[ProjectName].uproject" -Game -ExecCmds="Automation RunTests [TestPrettyName]" -log
In order to get all of the Pretty names for the tests run the following command:
UE4Editor.exe "[PathFromUnrealEditorBinaryToYourProjectDirectory]/[ProjectName].uproject" -Game -ExecCmds="Automation List" -log
You can call multiple tests using this method if you have a Pretty name that encapsulates multiple tests. for Example:
I have the following tests under my AI code: “TytoDome.AI.2Animals1Plant”,
“TytoDome.AI.Save The Animals”, “TytoDome.AI.Animal Hunting”, “TytoDome.AI.Fish Eating”. As well as the following tests in my Data Tool: “TytoDome.Data Tool.Unique EcoRole Collection”, “TytoDome.Data Tool.Zone Birth and Death and Unlocking”, “TytoDome.Data Tool.Animal Collection General”.
If I wanted to call EVERY test I have, I would use the following command:
UE4Editor.exe "[PathFromUnrealEditorBinaryToYourProjectDirectory]/[ProjectName].uproject" -Game -ExecCmds="Automation RunTests TytoDome" -log
If I wanted to call ONLY my AI Tests I’d run the following:
UE4Editor.exe "[PathFromUnrealEditorBinaryToYourProjectDirectory]/[ProjectName].uproject" -Game -ExecCmds="Automation RunTests TytoDome.AI" -log
If I wanted to call ONLY my Animal hunting Test I’d run the following:
UE4Editor.exe "[PathFromUnrealEditorBinaryToYourProjectDirectory]/[ProjectName].uproject" -Game -ExecCmds="Automation RunTests TytoDome.AI.Animal Hunting" -log
From here anyone should be able to figure out how to get this to work with their project!
Any chance there is some documentation on -AutomationTests? I was hoping to find some details on it here (Command-Line Arguments in Unreal Engine | Unreal Engine 5.1 Documentation).
1 Like
I’ve tested, command line
“Automation RunTests Project.Blueprints.CompileBlueprints”
does work, there’s no space between Compile and Blueprints, hope this can help you
To run unit tests from the command line, log to a file (and have a log popup), and exit after the tests, here is what I use:
UE4Editor.exe Path/To/MyProject.uproject -ExecCmds="Automation RunTests MyProject" -unattended -nopause -testexit="Automation Test Queue Empty" -log -log=RunTests.log
I wanted to also use -NullRHI to avoid seeing the Editor (and the associated Slate load) but this presently crashs UE4.20 at exit
Edit: also, I’ve updated the command line to get a Tests Report using the option -ReportOutputPath.
Here is our current complete commande line:
UnrealEngine\Engine\Binaries\Win64\UE4Editor.exe "C:\workspace\MyProject\MyProject.uproject" -ExecCmds="Automation RunTests MyProject" -Unattended -NullRHI -TestExit="Automation Test Queue Empty" -ReportOutputPath="C:\workspace\MyProject\Saved\TestReport" -Log -Log=RunTests.logDo you manage to see which tests fail?
I purposely made one test fail. When I run the tests in the editor, the logs state the failure.
When I run the tests from the command line, there is no mention of it 
Yes, in case of failure I do get a error log.
I will also make a script to parse the index.json report
Thanks. I found out the issue I had, which is a bug (now fixed in 4.21): when you run the tests and the session frontend is opened, no test is run. Now that it works in the command line, I need to find out why in my Jenkins, the editor tries to compile all the shaders when it did not have to when I cooked the game data…
Ok, good to know!
Also, I would like to know how you parse the results. I am currently using the Warnings Plugin but it does not fail the build as it should using thresholds.
At the end of the day, the real proper method IMO is to convert the json into JUnit test result so that the proper Jenkins plugin can display all appropriate info
Right now I did not have a chance to go that far since I do not have (yet!) the unit tests results in the log but my plan was to use the warnings plugin, with a custom regexp, and to fail the build if any test fails
I am doing that using Jenkins Pipeline with the following config:
warnings failedTotalAll: '0', parserConfigurations: [[parserName: 'UnrealTestReports', pattern: 'Saved\\TestReport\\index.json']]
With a very basic regular expression to catch the global status (so does not count how many failures):
Regular Expression: ^\s*"state": "Fail",$