Included in the UE4 4.21 patch notes is information about the “Gauntlet” automation framework. Is there any documentation available about this? I would love to use this in a project.
Same here. Any external documentation available ?
Same here. Any external documentation available ?
Same here. Any external documentation available ?
Google Translation of woaidaima’s answer into English, for those who don’t speak Chinese. It’s not a complete answer, and there are still things which I don’t understand, but at least it’s a bit more info than what I could find elsewhere.
First of all:
Gauntlet Automation Framework is one of the features added in 4.21 and is a framework for automated testing . The outline is described in the 4.21 release notes , but in summary, you can execute various tests by creating and executing test scripts (.cs, .cpp) according to the project. Please note that the configuration and usage may change in later versions because it is an early access function. This article is confirmed for 4.21.1.
Function:
Gauntlet Automation Framework is a function to execute an automatic test using RunUAT (AutomationTool). In other words, it is also possible to execute a test regularly by linking with Jenkins, etc.
The following figure illustrates the flow of testing using Framework (Gauntlet Test).
- Start the test. Execute the test command from RunUAT to start the test. Although Jenkins is often used as a CI, it is an image similar to creating and executing a test .bat file in the same way.
- Run the application. Start the content (Cooked content) in StagedBuild before running the test. If there is a specification of the device to execute, execute the application based on StagedBuild and start the device.
- Test execution. Start the test based on the test script created in advance. The application works on a test basis.
- Test complete. When the test results are returned from the application, the test ends. Even if the result is not returned, the test script returns the result by itself and completes the test.
- Report. Outputs the report to Log or .csv file based on the test result.
Although it is a brief explanation, the above is a series of flow at the time of test execution. The blue / green color in the figure is the function provided by the main Gauntlet Framework, but this will be explained in detail in the later chapters.
How to use
This chapter mainly explains how to execute Gauntlet Test. Here are the basic steps, such as how to run a simple test script and create a batch file yourself.
Run a sample test
First of all, I will describe the procedure for running GauntletTest. Gauntlet Test comes with a simple test script (ElementalDemoTest.cs), so you can try it right after downloading the engine. Here are the steps to run the sample:
[Execution procedure]
- Create Staging Build
- Test execution
Create Staging Build
Staging Build is created when you create a package, etc. It is created in [PROJECT]/Saved/StagedBuilds. Here we will package the project. If you want to use commands, such as automation, you can create them with the following command:
Build\BatchFiles\RunUAT.bat BuildCookRun -project=ElementalDemo -platform=Win64 -configuration=Development -build -cook -pak -stage
Test execution
After creating the Staging Build, all you have to do is run the tests. Here, the test script created in advance is executed from RunUAT. The following is an example of running ElementalDemo project on Windows:
Engine\Build\BatchFiles\RunUAT.bat RunUnreal -project=ElementalDemo -platform=Win64 -configuration=Development -test=ElementalDemoTest -build=Samples\Showcases\ElementalDemo\Saved\StagedBuilds
If you are successful in running the test, start the application and start the test. A sample test script is a script that simply runs the application and times out after 50 minutes. Therefore, once you start the application, it will stay as it is. If you quit this application (Alt + F4 etc.), you can end the test and get the following result. If you can confirm the operation so far, the procedure to carry out the Gauntlet Test is complete. Test reports are output by default to [ENGINE_ROOT]\GauntletTemp\Logs.
Create a batch file for testing
Next, I will explain how to create a batch file for test execution so that you can run automated tests in your own project. Below is a basic batch file to run the test. This is an edited version of “Example command of test execution” described above. A batch file can be created based on this command.
This batch file is an example of a case where the test of the test sample ElementDemoTest is executed with Win64: Development configuration for the own project GauntletTest, using Engine obtained from Github. You can basically use it as it is by rewriting the path or project name, but you can customize it by your project.
path to rem // RunUAT.bat
set UAT_PATH=G:\GitHub\UnrealEngine-4.21\Engine\Build\BatchFiles\RunUAT.bat
rem // project name
set PRJ_NAME=GauntletTest
rem // Staging directory path
set STAGING_DIR=G:\Projects\GauntletTest\Saved\StagedBuilds
rem // test execution command
set TEST_CMD=RunUnreal
rem // test name
set TEST_NAME=ElementalDemoTest
rem // platform
set PLATFORM=Win64
rem // build configuration
set CONFIG=Development
rem ********************* Start Gauntlet Test *********************
%UAT_PATH% %TEST_CMD% -project=%PRJ_NAME% -platform=%PLATFORM% -configuration=%CONFIG% -test=%TEST_NAME% -build=%STAGING_DIR%
rem ********************* End Gauntlet Test *********************
pause
Variables:
- UAT_PATH: Any specification is acceptable as long as RunUAT.bat can be executed.
- PRJ_NAME: Specify a project (does not require .uproject)
- TEST_CMD: You can specify the test command and BuildCommand’s inherited class (RunUnreal)
- TEST_NAME: Inherited class of DefaultNode
- PLATFORM: Specify the platform (Win64 / Andorid / PS4 etc.)
- CONFIG: Specify build configuration (Shipping / Development etc.)
Framework
- Startup script:
\Engine\Source\Programs\AutomationTool\Gauntlet\Unreal\RunUnreal.cs
Execution class for GauntletTest. We have the ability to notify Test Executor that the test has started and to set up the device to run the test. By inheriting as needed, you can add steps such as device activation before test execution and account authentication.
- Test Execution / Management:
\Engine\Source\Programs\AutomationTool\Gauntlet\Framework\Gauntlet.TestExecutor.cs
This class mainly manages test execution. Basically, it receives the start of the test from the start script, executes the test script, and possesses the function to manage the test execution result.
- Test script:
\Engine\Source\Programs\AutomationTool\Gauntlet\Unreal\Base\Gauntlet.UnrealTestNode.cs
Base class for test script. TestStart and Tick callback functions are provided, so you can control the behavior during test execution by writing the necessary code for the test.
- Test control:
\Engine\Plugins\Experimental\Gauntlet\Source\Gauntlet\Public\GauntletTestController
Base class for performing GauntletTest. It works by enabling the Plugin. Basically, the content is embedded in the application, and is used when accessing the game code directly to control the game. If you want to add an automatic test for each project, create a subclass based on this class and add test code. A base class is a class that provides only basic modules, and you can also implement logic that starts and monitors in derived classes. Use command line parameter (-gauntlet = MyControllerName) to execute TestController.
Customizing sample and source code
If you want to carry out automatic testing in a project, you can create your own test by deriving a class provided by Engine. In the diagram described in “Gauntlet Test Flow”, classes that mainly extend are the following reddish classes. This is just an example, but other classes can be extended or modified to provide more efficient tests or tests in line with the project.
The following are samples of scripts and classes provided by default. If you want to create your own test environment or script you can use it as a reference.
GauntletTest sample test 1:
\Engine\Source\Programs\AutomationTool\Gauntlet\Unreal\Game\Samples\ElementalDemoTest.cs
Inherited class of UnrealTestNode, there is only a 50 minute timeout and the test results fail due to timeout.
GauntletTest sample test 2:
\Engine\Source\Programs\AutomationTool\Gauntlet\Unreal\Game\Samples\FortGPUTestbedPerfTest.cs
Inherited class of UnrealTestNode, detect a timeout of 5 minutes or specific input and end the test.
Startup Test Controller:
\Engine\Plugins\Experimental\Gauntlet\Source\Gauntlet\Private\GauntletTestControllerBootTest.cpp
This is a test sample that performs application launch check in a subclass of UGauntletTestController.
Error Test Controller:
\Engine\Plugins\Experimental\Gauntlet\Source\Gauntlet\Private\GauntletTestControllerErrorTest.cpp
This is a test sample that performs application error detection in a subclass of UGauntletTestController. It is an execution module of FGauntletModuleImpl GauntletTest. It mainly controls GauntletTestController. Since the Type of Plugin is Development, it can not be executed at runtime by default. You need to change Type to “Runtime” to enable this. (Probably a bug!)
This section describes the sample test and controller class behavior presented above. The following code is a modified version of the sample with additional comments. By referring to the contents described here, you will understand how the sample works and I think it will be a guide for expanding from the sample.
Test sample
namespace UE4Game
{
///
Define test class (name to pass as argument at test execution) ///
Define test code as subclass of DefaultNode public class ElementalDemoTest : DefaultNode
{
public ElementalDemoTest ( Gauntlet . UnrealTestContext InContext )
: base ( InContext )
{
}
/// Define the configuration for test
/// For example, MaxDuration specifies the test timeout time
/// ※ There is a comment with "5min" but actually 5 * 600 (seconds) = 50 ( Timeout will occur in minutes)
public override UE4TestConfig GetConfiguration ()
{
/// just need a single client
UE4TestConfig Config = base . GetConfiguration ();
UnrealTestRole ClientRole = Config . RequireRole ( UnrealTargetRole . Client );
ClientRole . CommandLine + = "- unattended " ;
Config . MaxDuration = 5 * 600; // 5min should be plenty
return Config;
}
/// This is a module that creates a report of test results and is called at the end of the test
/// This module describes the process of outputting the result of Health Snapshot (see the next chapter for details)
public override void CreateReport ( TestResult the Result , UnrealTestContext Contex , UnrealBuildSource Build , IEnumerable < UnrealRoleArtifacts > Artifacts , string ArtifactPath )
{
UnrealRoleArtifacts ClientArtifacts = Artifacts . the Where ( A => A . SessionRole . RoleType == UnrealTargetRole.Client).FirstOrDefault();
var SnapshotSummary = new UnrealSnapshotSummary<UnrealHealthSnapshot>(ClientArtifacts.AppInstance.StdOut);
Log.Info("Elemental Performance Report");
Log.Info(SnapshotSummary.ToString());
base.CreateReport(Result, Contex, Build, Artifacts, ArtifactPath);
}
/// This is a module to save the test result, and is called at the end of the test
/// This module performs processing to save the result of FPSChart to the specified Directory with .zip
/// as an argument at test execution time Create a .zip file by specifying the output destination with -uploadfolder = [FULLPATH] "
/// Run StartFPSChart / StopFPSChart during testing to save the results of FPSChart.
public override void SaveArtifacts_DEPRECATED ( string OutputPath )
{
string UploadFolder = Globals . Params . ParseValue ( "uploadfolder" , "" );
if ( UploadFolder . Count () > 0 && Directory.CreateDirectory(UploadFolder).Exists)
{
string PlatformString = TestInstance.ClientApps[0].Device.Platform.ToString();
string ArtifactDir = TestInstance.ClientApps[0].ArtifactPath;
string ProfilingDir = Path.Combine(ArtifactDir, "Profiling");
string FPSChartsDir = Path.Combine(ProfilingDir, "FPSChartStats").ToLower();
string FpsChartsZipPath = Path.Combine(TestInstance.ClientApps[0].ArtifactPath, "FPSCharts.zip").ToLower();
if (Directory.Exists(FPSChartsDir))
{
ZipFile.CreateFromDirectory(FPSChartsDir, FpsChartsZipPath);
string DestFileName = "ElementalDemoTest-" + PlatformString + ".zip";
string DestZipFile = Path.Combine(UploadFolder, DestFileName);
File.Copy(FpsChartsZipPath, DestZipFile);
}
}
else
{
Log.Info("Not uploading CSV Result UploadFolder: '" + UploadFolder + "'");
}
}
}
}
Controller sample
#include "GauntletTestControllerBootTest.h"
/// Called every frame when the test is run
void UGauntletTestControllerBootTest :: OnTick ( float TimeDelta )
{
/// When the boot process is completed Test ends as "Test result: Normal"
/ // Actually, IsBootProcessComplete () always returns false, so it does not pass
if ( IsBootProcessComplete ())
{
EndTest ( 0 );
}
else
{
///
if 300s (seconds) have passed from the start of the test “Test result: The test ends as "abnormal" /// Actually GetTimeInCurrentState () always returns 0 and does not pass
if ( GetTimeInCurrentState () > 300 )
{
UE_LOG(LogGauntlet, Error, TEXT("Failing boot test after 300 secs!"));
EndTest(-1);
}
}
}
Performance report
As described in Release Notes, the function to output performance reports as below from 4.21 has been added. This is also available for Blueprints, but can also be incorporated into tests.
The following is an example of acquiring a report in Blueprint. The ability to capture this performance is called Health Snapshot, which allows you to capture performance logs between specific points. The three main events are described below.
-
Start Performance Snapshots – Starts acquiring Health Snapshots.
-
Log Performance Snapshot – Record between 2 points. Change the capture method of Record by the flag of “Reset Stats”.
-
Stop Performance Snapshots – Stops acquisition of Health Snapshot and outputs it.
If at the time to confirm these output in ElementalDemoTest.cs of the sample, in advance Snapshot and Summly Title need to have to match, and then capture the HealthSnapshot during the execution of the test. The following is an example for acquiring Snapshot after matching with the title “LogPerf”.
- Get Snapshot with “Log Perf” with Blueprint:
- Get Snapshot with “Log Perf” when creating Report:
var SnapshotSummary = new UnrealSnapshotSummary<UnrealHealthSnapshot>(ClientArtifacts.AppInstance.StdOut, "LogPerf");
Hello, buddy, do you know more about gauntlet test writing, such as automatically assigning android devices from a device pool
The current documentation on the Gauntlet seems to be based on this post or at least contain very similar parts (but it’s a lot shorter) and it does not work out of the box for UE4.22.3.
I do hope the Gauntlet will get better documentation for UE4.23 and a bit more examples on how it’s expected to be used.
I tried the command for testing ElementalDemo in UE document:
RunUAT RunUnreal -project=Samples/Showcases/ElementalDemo -platform=Win64 -configuration=Development -build=local -test=ElementalDemoTest
But it cause the error “Can’ not find project file for Samples/Showcases/ElementalDemo/ElementalDemo”.
It should use full project name at the “-project” part, and “-build” should set to the StageBuilds folder:
RunUAT RunUnreal -project=Samples/Showcases/ElementalDemo/ElementalDemo.uproject -platform=Win64 -configuration=Development -test=ElementalDemoTest -build=Samples\Showcases\ElementalDemo\Saved\StagedBuilds
I hope this will be useful to anyone who encounter the same problem.