Hey there! I don’t know the answer to your question, but I’d like to give it my best guess.
The AutomationFramework is an UnrealEngine Subsystem, which communicates through the message bus with other parts of the engine. When you pass along info using “-ExecCmds”, those become messages that are passed in and then parsed. As near as I can tell, this also happens when you use the SessionFrontend GUI for selecting and running automated tests.
There are two kinds of tests in the AutomationFramework. FunctionalTests, and AutomationTests. FunctionalTests are actors, and they get placed in the map somewhere, and have a physical position and more. AutomationTests (Unit Tests, and Spec Tests), and NOT actors, and they aren’t in a map anywhere.
If you just have “pass/fail” tests to run (as opposed to “performance” tests, or other kinds of tests which gather information, then I think the first approach works really well.
REM This is an example in Windows Batch
set UNREAL_CMD_EXE="F:\UE_5.2\Engine\Binaries\Win64\UnrealEditor-Cmd.exe"
set UPROJECT_FILE="F:\GitLab-Runner\builds\rT-3XvJF6/0/gishin2/cisystem/CIsystem.uproject"
set AUTOMATION_FLAGS=-unattended -nullrhi -nosound
set AUTOMATED_TESTS_TO_RUN=DummyGroup.DummyTest.Test+DummyGroup.DummyTest2.AnotherTest
set AUTOMATION_CMDS=-ExeCmds="Automation RunTests %AUTOMATED_TESTS_TO_RUN%; quit"
set AUTOMATION_EXIT=-TestExit="Automation Test Queue Empty"
set AUTOMATION_COMMAND=%UNREAL_CMD_EXE% %UPROJECT_FILE% %AUTOMATION_FLAGS% %AUTOMATION_CMDS% %AUTOMATION_EXIT%
call %AUTOMATION_COMMAND%
There are a couple of caveats about this approach though:
- No networking support – not an issue for a single-player title
- No built in profiling support (unlike when using Gauntlet)
- No built in support for testing on consoles (unlike when using Gauntlet)
- Does not run against a packaged build–I think (unlike when using Gauntlet)
So, when using this approach for something like smoketesting, or for a smaller game, or an offline game, I think it’s pretty decent, and less complicated than using Gauntlet.
RunUAT
set UPROJECT_FILE="F:\GitLab-Runner\builds\rT-3XvJF6/0/gishin2/cisystem/CIsystem.uproject"
set AUTOMATED_TESTS_TO_RUN=DummyGroup.DummyTest.Test+DummyOtherGroup.DummyOtherTest.OtherTest
REM Note the use of quotes here, but not in the variable AUTOMATED_TESTS_TO_RUN
set AUTOMATION_CMDS=-RunTest="%AUTOMATED_TESTS_TO_RUN%"
REM -test=UE.TargetAutomation - This specifies a built in Gauntlet test, which can handle running FunctionalTests, as well as
REM AutomationTests. Other options include: UE.EditorAutomation, and writing your Gauntlet test class
REM
REM -nosplash - don't use the UnrealEngine splash screen
REM -unattended - don't require any kind of user interaction
REM -nopause - when finished, don't pause before the window closes
REM -nocontentbrowser - don't start the content browser, it's not needed
REM -nullrhi - don't use the RHI thread, which transfers data to and from the GPU (IE: don't render anything)
REM
set AUTOMATION_FLAGS=-test=UE.TargetAutomation -nosplash -unattended -nopause -nocontentbrowser -nullrhi
REM You don't want to use all of these at the same time, and for a "headless" (not rendering)
REM test, you might not want any of them
REM
REM -ResX, -ResY - makes the "client" (the game) run with the given resolution. Don't mix with -Fullscreen
REM -Fullscreen - makes the "client" (the game) run in fullscreen mode. Don't mix with -ResX, -ResY
REM
REM -MaxDuration - I think this is how long any individual test is allowed to take
REM -NoTimeout - I think this disables the timeout for "all" of the tests you're running
REM The default timeout is like, 30s or 60s or 10m (can't recall), so using
REM this is more appropriate to performance testing that runs for a while
REM
set CLIENT_ARGS=-ClientArgs="-ResX=1920 -ResY=1080 -Fullscreen -MaxDuration=9999 -NoTimeout"
REM This command is going to be the same as:
REM
REM "F:\UE_5.2\RunUAT.bat" RunUnreal -project="F:\GitLab-Runner\builds\rT-3XvJF6/0/gishin2/cisystem/CIsystem.uproject" -test=UE.TargetAutomation -nosplash -unattended -nopause -nocontentbrowser -nullrhi -RunTest="DummyGroup.DummyTest.Test+AnotherDummyGroup.AnotherDummyTest.AnotherTest" -ClientArgs="" -log
REM
REM Unfortunately, you'll likely just need to "trial-and-error" your way to finding what's going to work for your situation.
REM That's why I recommend using a bat file yourself: it saves on typing and helps prevent spelling mistakes (same thing)
REM
set RUN_UAT_CMD=RunUAT.bat RunUnreal -project=%UPROJECT_FILE% %AUTOMATION_FLAGS% %AUTOMATION_CMDS% %CLIENT_ARGS% -log
REM This needs to run from your UE5 directory. Based on your file paths, it looks like you'd do:
REM pushd "push directory" kinda just means "do the thing inside this directory"
pushd F:\UE_5.2\
call %RUN_UAT_CMD%
REM popd "pop directory" kinda just means "okay done with that directory"
popd
Lastly, you showed an example from the logs. Figuring out what happened is a complex situation. The two most likely situations,
in my opinion, are:
-
The executable crashed, or exited early for some other reason.
a. If this is true, it’s higher up in the log somewhere
-
The AutomationFramework didn’t find all of the tests, and it didn’t find your tests, so it didn’t run your tests
#2 is pretty tricky. I’m trying to figure out how to work around this limitation, and I don’t know if it’s possible to do so, without
writing some CPP and compiling it locally, to use your own version of the editor. If that is happening to you, and you’re not modifying
the editor, then probably consider just using your first approach if it’s working for you.
If you actually need to use the UnrealAutomationTool, (RunUAT), to launch Gauntlet (RunUnreal), then you’re going to have to spend
some hours making a robust build pipeline, and after you do so, it’ll be an easier problem to solve.
I don’t know if this was helpful, but I hope it was (either to you or the next person who finds this thread)
