Trigger a Functional Test via Gauntlet

Development Pipeline & Plugins
, ,
1

Hey there,

I am trying to run just one functional test on a standalone build (5.5.4) with this commandline (this worked for 5.3, just like that):

RunUAT.bat RunUnreal -CustomModuleRoles=MyGame:Client -test=UE.TargetAutomation -runtest=MyFunctionalTest -map=MyMap -project=myproject -build=pathToPackageBuild -platform=theplatform -configuration=theconfig* I’ve tried PC and PS5.

** The -map parameter isn’t required based on the documentation but I needed it for 5.3:

https://dev.epicgames.com/documentation/en\-us/unreal\-engine/running\-gauntlet\-tests\-in\-unreal\-engine?application\_version\=5\.5

So! with that commandline above, 2 processes start:

- myProject-theplatform-theconfig.exe

- UnrealEditor.exe

The call ends up failing due to: No automation tests matched MyFunctionalTest

Checking the log:

Editor Log:

LogAutomationWorker: Set 270380 tests to D46699434DEFC5FF4EA022AE5C84F1BC

LogAutomationController: Received RequestTestsReplyCompleteMessage from 04154E5E44EF3823FAB051922BB01569

LogAutomationController: 270380 tests available on 04154E5E44EF3823FAB051922BB01569

LogAutomationCommandLine: Ignoring refresh from ControllerManager. NumDeviceClusters=1, CurrentState=2

Client Log:

LogAutomationWorker: Set 69 tests to F913DCC044BB4CA73A9CA7A8F908CF1D

LogAutomationController: Received RequestTestsReplyCompleteMessage from 3EEFC26841FE4560220A589BD4593131

LogAutomationController: 69 tests available on 3EEFC26841FE4560220A589BD4593131

LogAutomationCommandLine: Display: Found 68 Automation Tests

MyFunctionalTest shows up in the 270380 test list from editor.

But the “client test list” with the 69 tests is the one being checked.

MyFunctionalTest has these flags:

EAutomationTestFlags::ClientContext

EAutomationTestFlags::EditorContext

EAutomationTestFlags::ProductFilter

And suddenly on 5.5.4, the only way for me to achieve a simple FTest run in a standalone build is the GauntletController approach from this post:

Any idea? Thank you!

[Content removed]

2

I’m suspecting that the cooking process is not including the map that contains the functional tests anymore.

To force the inclusion of the maps that contain the functional tests you can either force them to be added through the project config file or override the project Asset Manager to make it automatic.

You can achieve that this way:

Add <project>/Source/<project>/AssetManager/<project>AssetManager.h

`#pragma once

include “CoreMinimal.h”
include “Engine/AssetManager.h”
include “AssetManager.generated.h”

/** Overrides cook to make sure all tests get cooked */
UCLASS(BlueprintType)
class MYPROJECT_API UMyProjectAssetManager : public UAssetManager
{
GENERATED_BODY()

public:

#if WITH_EDITOR
virtual void ModifyCook(TConstArrayView<const ITargetPlatform*> TargetPlatforms, TArray& PackagesToCook, TArray& PackagesToNeverCook) override;
#endif

};`Add <project>/Source/<project>/AssetManager/<project>AssetManager.cpp

`#include “AssetManager.h”
include “FunctionalTestingModule.h”
include “Misc/PackageName.h”

#if WITH_EDITOR
void UMyProjectAssetManager::ModifyCook(TConstArrayView<const ITargetPlatform*> TargetPlatforms, TArray& PackagesToCook, TArray& PackagesToNeverCook)
{
Super::ModifyCook(TargetPlatforms, PackagesToCook, PackagesToNeverCook);

TArray TestInfoUnused;
TArray MapAssets;
IFunctionalTestingModule::Get().GetMapTests(false, TestInfoUnused, MapAssets);

// If a map has a runtime test, add to cook list
for (FString MapAsset : MapAssets)
{
PackagesToCook.AddUnique(FName(*FPackageName::ObjectPathToPackageName(MapAsset)));
}

}
#endif`

3

Hey, thank you for such quick reply,

I totally forgot to mention that I made sure all the assets involved were being cooked. I went for the manual change in DefaultGame.ini for the map:

+MapsToCook=(FilePath=“/Game/MyGame/TestMaps/Automation/LVL_FuncTest”

It has to be something else…

Than you anyway Jerome!

4

Yes,

The build is a Test build

Commandline for cooking is:

call Engine\Build\BatchFiles\RunUAT.bat BuildCookRun -project=“MyProject/MyProject.uproject” -noP4 -targetPlatform=Win64 -cook -noiterativeCooking -unattended -clientConfig=Test -serverConfig=Test -noXGE

And the map was loaded with no issue, but the functional test would never run.

Thank you in advance.

5

Hello,

The BP base class in this case is our own implementation of AFunctionaTest (but at the moment it is empty, no extra implementation).

But then I tried with a pure AFunctionalTest BP too and it didn’t work either.

Thanks.

6

Are you reproducing with a Client Development build?

7

I’m in a loss at the moment. I’m unable to reproduce the issue.

Like Ellie mentioned there are several factors that can undermine the test registration.

If the test is not listed in the log when running the Client with -ExecCmds=“Automation List;Quit” compared to the Editor, something is preventing GetTests to list that BP functional tests.

It is usually related to cooking; which is what I proposed at first. But it looks like the map is part of the cooked build since it can be loaded.

In my first post, the proposed change in the project asset manager is the best solution to make sure the BP tests get registered.

Ellie reminded me that indeed world partition or nested level might impact how the test gets registered. But usually it is listed twice, from original map and from the map that nests it.

However with world partition an actor can be saved as external asset. There is code to handle that situation but maybe you are in a failing case that we never found.

Other thing that can impact is if the map is part of Content/Developer folder, there is a project Config to avoid discovering that folder; by default is not enabled.

Also if the map name is prefixed with `FTEST_` it will assume the map to auto starts the BP functional test through the level BP. There is no hook enforced in that case, the user must make sure the BP Functional Test manager is started and trigger the first test. The `FTEST_` map pattern is ancient and comes from the early version of UE4. Changing the prefix for something else will solve the issue. Note that in that case the test name is not including the actor label but end with the map name.

Another potential avenue is to move the assets into their own plugin if that is not already the case. It is easier to control if the tests and related assets are to be included in the Client through the plugin manager. However it adds more boiler plates.

8

As far as I know there was nothing on the test automation side that changed how the BP functional tests are registered or triggered.

We introduced a tagging mechanism that store additional information about C++ and BP tests. That is only remaining aspect that could impact your case that I can see.

9

I will need additional information to investigate further:

What is the build configuration that the test pass is being run on?

What are the arguments used to cook the target configuration?

Can the map be loaded on the packaged game?

10

I was assuming you were using the Blueprint functional test actor to setup your test.

But I’m realizing I did not confirm that we you.

What is the base class of the test?

11

Hey,

That’s right. I am trying to run functional tests in a standalone build. (platforms: Win64 and PS5).

12

Hey! I have some weird suggestions:

A. Can you run this functional test from the SessionFrontend menu in the UnrealEditor?

B. You mentioned the FuntionalTest has EAutomationTest flags. Those flags are for use in AutomationTests (Unit Tests), and not FunctionalTests (Integration Tests). To define an automation test, you need to use a maco, and then implement the RunTest function in the class that macro defines for you.

`IMPLEMENT_SIMPLE_AUTOMATION_TEST(FMyAutomationTestWhichAlwaysPasses, “Project.MyAutomationTests.This test always passes”,
EAutomationTestFlags::EditorContext | EAutomationTestFlags::ProductFilter)

bool FMyAutomationTestWhichAlwaysPasses::RunTest(const FString& Parameteres)
{
return true;
}`

In order to make a CPP functional test, it should likely be in its own file, and then you inherit from it to make a BP object, which is then placed into a game world.

`#pragma once

include <CoreMinimal.h>
include <FunctionalTest.h>

// this has to come last
include “MyFunctionalTestBase.generated.h”

// also declare a log category so you don’t have to use LogTemp
MYPROJECT_API DECLARE_LOG_CATEGORY_EXTERN(LogMyFunctionalTest, Log, All)

// in ONE cpp file, you add
// DEFINE_LOG_CATEGORY(LogMyFunctionalTest)
//
// Then you can log with:
// UE_LOG(LogMyFunctionalTest, Warning, TEXT(“This is the name of some actor: %s”), *AActor::Name);

UCLASS()
class AMyFunctionalTestBase
{
GENERATED_BODY()

// this is called for every test at the same time, when BeginPlay is called
virtual void BeginPlay() override;

// this is for loading and stuff
virtual void PrepareTest() override;

// this is called when a specific instance of this test is allowed to begin
virtual void StartTest() override;

// before a test “starts”, the test framework will give it a chance to being loading
// periodically, it will call AFunctionalTest::IsReady, and if that fn returns true,
// then the test is allowed to proceed. You can use this mechanism to enforce some
// amount of loading time
virtual bool IsReady_Implementation() override { return true; }

// most of an integration test will happen in the tick fn, but if you
// can finish “on the first frame”, then you can probably put all your code in StartTest
virtual void Tick(float deltaTime_s) override;

// after the test ends, ie: after you call FinishTest
// this is a good place to clean up stuff
virtual void CleanUp() override;

}`Sorry if the above is all obvious recap.

C. Does anything change if you capitalize the runtest parameter to say “-RunTest=MyFunctionalTest”

D. Is your packaged build in the `…\UE5\<Project>\Saved\StagedBuilds\<Platform>` directory? (I’ve found it behaves badly anywhere else)

E. When you take your FunctionalTest and place it in a level (extra work if it’s a world partition level), are you taking the name of that specific object from the level, and using that name to launch the test?

ie: if you put your MyFunctionalTest in a map, and renamed it (“LoadTest”, for instance), are you then using “LoadTest” when you try to launch the test, or are you still using “MyFunctionalTest”?

Also, good luck! There’s a lot of process-of-elimination with this system.

13

Hi

A. Yes, without issue.

B. Got it, bit before, on 5.3 I was able to run the without the need of implementing anything. Feels like something changed. in 5.5.

C. I always use caps, yes

D. Yes, I tried from packaged and staged builds, <Project>\Saved\StagedBuilds\<Platform> and <Project>\Saved\Packages\<Platform>

E. I tried many names: the one showing up in SessionFrontend, the internal name of the object, nothing worked out.

Our case is a pretty simple one, just simple zoos with specific gameplay to test with Func. Tests.

But it’s fine, I managed to make it work implementing a Gauntlet Controller, though this wasn’t neccesary on 5.3.

Thank you for your time!

14

Hey Jerome,

It’s ok. I just wanted to confirm if there were changes that would explain why this wouldn’t work in 5.5 anymore.

For the record:

-Cooking is not the problem, levels and assets related are there,

I can run any test via FunctionalTestingModule + UFunctionalTestingManager, but thank you very much for your suggestions!

-Our project is not implementing world partition approach.

-All the levels with functional tests are out of the Dev path. Our current path is Content/Automation/…

-Tests are named like BP_FunctionalTest-

-Didn’t quite follow the “move assets into their own plugin” but it’s fine, it’s sorted out, I have now batches of them running in a row via “Gauntlet Controller + IFunctionalTestingModule + UFunctionalTestingManager”.

Thank you very much for your time, guys. :slight_smile: