Gauntlet Performance Testing: how do my friend?

Development Pipeline & Plugins
, ,
1

`// File: RunPerfTest.cs ----

using Gauntlet;
using UnrealBuildBase;

namespace MyGame {
public class RunPerfTest : RunUnreal {

public override string DefaultTestName => “MyPerfTestNode”;

public override ExitCode Execute(){
Globals.Params = new Gauntlet.Params(this.params);
UnrealTestOptions testOptions = new UnrealTestOptions();
AutoParam.ApplyDefaults(testOptions);

testOptions.Project = “MyGameProject”;
testOptions.Namespace = “MyGame.Automation”;

AutoParam.ApplyParams(testOptions, Globals.Params.AllArguments);

return RunTests(testOptions);
}
}
}

// File: MyPerfTestNode.cs ----

using Gauntlet;
using UE;

namespace MyGame.Automation {
public class MyPerfTestNode : UE.TargetAutomation {
public MyPerfTestNode(UnrealTestContext inContext) : base(inContext) {}

public override AutomationTestConfig GetConfiguration(){
AutomationTestConfig config = base.GetConfiguration();
UnrealTestRole clientRole = config.RequireRole(UnrealTargetRole.Client);

// so I don’t have to race the game client to attach the debugger
clientRole.CommandLine += " -WaitForDebugger";

// What is meant to go here? Is it the GauntletTestController, or just some test string:
// eg: Project.MyGameProject.Functional Tests.ThisIsAFunctionalTestIWrote
config.RunTest = “MyGameProject.MyGauntletTestController”;
// config.RunTest = “MyGameProject.MyUnitTests”;

return config;
}
}
}

// File: MyGauntletTestController.h ----

include “Gauntlet/Public/GauntletTestController.h”
include "MyGauntletTestController.generated.h

UCLASS()
class MYGAME_API UMyGauntletTestController : public UGauntletTestController
{
GENERATED_BODY()
public:
virtual void OnInit() override;

virtual void OnTick(float deltaTime_s) override;
};

// File: MyGauntletTestController.cpp ----

include “MyGauntletTestController.h”

// just imagine all of these functions calls are implemented and working

void MyGauntletTestController::OnInit()
{
Super::OnInit();
GetRemainingPositionsFromTestNode();
}

void MyGauntletTestController::OnTick(float deltaTime_s)
{
Super::OnTick(deltaTime_s);

if(state == ETestState::GetNextPosition)
{
currentPosition = GetNextPosition();
}
else if (state == ETestState::ProfilingFinishedForCurrentPosition)
{
Framework::EmitMessage(“ProfiledPosition”, currentPosition);
}
}`

Essentially, my goal is to:

  • Have a simple command which will launch my automated perf testing (For a single player, offline game)

    • Not pictured: during a perf test, I’ll move the APlayerCharacter to different positions, and run the CSVProfiler
      • I want to have MyPerfTestNode.cs read a file of those positions, and then pass them along. (Should the Gauntlet controller be doing this?)

  • If the game crashes, I want some way of retesting the positions I haven’t profiled yet---not of re-running the entire job.

    • Can the Gauntlet controller emit an event back out to MyPerfTestNode? (so it can keep track of the work that’s been performed)
    • Can MyPerfTestNode reboot the game client if it goes down?
      • Also, how would I do that? Does the test node itself need a restart?
      • Previously, I’ve used a flag to restart on a crash, but it restarts the entire job

  • Am I thinking about this the right way?

    • So far, I can do: RunUAT.bat RunPerfTest, and it gets MyPerfTestNode and sets it up
    • I haven’t been able to launch into a Gauntlet Controller though.
      • When using a Gauntlet Controller: What is the “test”? Just the gauntlet controller’s name?

Thank you for your patience with this question. I realize it’s really several questions at once. I’m still trying to get a feel for where different responsibilities are meant to live within these systems.

If any of y’all have thoughts or advice, I would appreciate hearing it. Apologies if I’ve missed anything rudimentary. The code above hasn’t actually been run before, it’s just a generalization of my real code. Thanks!

2

Steps to Reproduce
No repro, mo bug, I'm just trying to wrap my head around the specifics of this system

3

Hi,

So it highly depends on your actual test scenario.

If you use UE.TargetAutomation, Gauntlet will use the Engine FAutomationTestBase registered classes. Enhance the “-RunTest=” argument that will target the set of tests that match the filter input (RunTest can be seen as a filter since it will do partial match by default and run every tests that match the string).

UE.TargetAutomation Gauntlet test node requires the Editor as well to coordinate test execution and it is not what could be considered lean.

But UE.TargetAutomation is able to resume from Engine Test node crashes and resume the execution of the set of tests passing the last failed Engine Test. That is by using the command line argument “-ResumeOnCriticalFailure”

UE.TargetAutomation is not using any Gauntlet Controller. It uses only the FAutomationTestBase classes to drive test execution.

Gauntlet Controller is a mean to drive execution of code in a game client. We suggest to place a Gauntlet Controller in it own UE plugin so it can be enabled modularly.

Gauntlet Controller are summon by Gauntlet if the UnrealTestRole has any controller set.

ie:

var GameRole = Config.RequireRole(UnrealTargetRole.Client); GameRole.Controllers.Add("MyController");see: https://dev.epicgames.com/documentation/en\-us/unreal\-engine/gauntlet\-controller\-in\-unreal\-engine

If you write your own Gauntlet Test Node, you can still leverage the mechanic that UE.TargetAutomation is utilizing to resume on failure; but you will need to implement a few additional elements to make it works.

A Test Node could trigger `SetToRetryIfPossible()` from its `CreateReport()` in order for the test to be rescheduled, a crash could be detected there.

However, your Gauntlet Controller will need a way to know where it left off. If you want to use the Retry mechanic. Make sure to have the `Restart()` method from the Test node properly setup so that anything that needs to be reset for the retry is indeed reset.