Buildgraph, Blueprint Functional Tests, and Horde

Development Pipeline & Plugins
, ,
1

I’m trying to figure out how to use BuildGraph to run Blueprint Functional Tests, with the goal of eventually having the results be run by Horde Templates and show up in the Horde Automation tab.

I have Gauntlet tests that I’ve written and those I can run in my Templates and show up in the Automation tab.

However, for the life of me, I can’t figure out how to get BuildGraph to run my Blueprint Functional Tests. This is what I currently have, but I’ve spent two days on this and tried dozens of variations.

<Agent Name=“RunFunctionalTestsAgent” Type=“TestWin64”>

  \<Node Name\="RunFunctionalTests" Requires\="$(CompileEditorNode)"\>

    \<Command Name\="RunUnreal"

      Arguments\='\-project\=$(ProjectFile) \-build\=Editor \-test\=UnrealEditor \-unattended \-nop4 \-nosplash \-NoSound \-NoVerifyGC \-NoLoadingScreen \-map\=/Game/Maps/TestLevels/FunctionalTests/TM\_FunctionalTests \-ExecCmds\="Automation RunTest Project.Functional Tests.Maps.TestLevels.FunctionalTests.TM\_FunctionalTests;Quit;"' /\>

  \</Node\>

\</Agent\>

My Functional Test is placed in the map (TM_FunctionalTests) and is called FT_DrawStowTest, and is an instance of BP_DrawSwapStowTest.uasset (a Functional Test Blueprint). I can run this test via the Session Frontend and it works perfectly. I can call it directly through something like the following:

“C:\p4\main\game\Engine\Binaries\Win64\UnrealEditor-Cmd.exe” “C:\p4\main\game\OurProject\OurProject.uproject” “/Game/Maps/TestLevels/FunctionalTests/TM_FunctionalTests” -ExecCmds=“Automation RunTests Project.Functional Tests.Maps.TestLevels.FunctionalTests.TM_FunctionalTests.FT_DrawStowTest;Quit;” -unattended -nop4 -nosplash

This also works.

But when I attempt to run it through the BuildGraph above, it loads the map, but never runs the test. I’ve managed to get these variations of outputs:

My test isn’t found, so it runs DefaultTest. (this is the current behavior)

My test is found, “runs”, but none of the code/nodes in my Blueprint get called, so it just times out and fails.

My test isn’t found, so it just launches the Editor and does nothing, waiting forever.

What am I missing? My next attempt is to make a Gauntlet test that just calls my Blueprint, but this doesn’t scale well.

2

Steps to Reproduce

Run the command with the following BuildGraph

"C:\p4\main\game\Engine\Build\BatchFiles\RunUAT.bat" BuildGraph -Script=OurProject/Build/Graph/OurXMLScript.xml -Target="RunFunctionalTests" -set:ProjectFile=OurProject/OurProject.uproject -set:ProjectName=OurProject -set:ProjectPath=OurProject -set:WithWin64Editor=true -set:EditorBootTest=false -set:TargetBootTest=false
 
 
<Agent Name="RunFunctionalTestsAgent" Type="TestWin64">
      <Node Name="RunFunctionalTests" Requires="$(CompileEditorNode)">
        <Command Name="RunUnreal"
          Arguments='-project=$(ProjectFile) -build=Editor -test=UnrealEditor -unattended -nop4 -nosplash -NoSound -NoVerifyGC -NoLoadingScreen -map=/Game/Maps/TestLevels/FunctionalTests/TM_FunctionalTests -ExecCmds="Automation RunTest Project.Functional Tests.Maps.TestLevels.FunctionalTests.TM_FunctionalTests;Quit;"' />
      </Node>
    </Agent>
3

The proper command to run a test through the Engine test automation framework is:

RunUAT.bat RunUnreal -test=UE.EditorAutomation -runtest="Mytest.one" -project=<path to uproject> -build=editorSee https://dev.epicgames.com/documentation/en\-us/unreal\-engine/running\-gauntlet\-tests\-in\-unreal\-engine for more options

You should not need to specify. For Client side the map needs to be cooked with the package so the test can be discovered and the map loaded.

In your buidgraph it should look like this:

<Agent Name="RunFunctionalTestsAgent" Type="TestWin64">
      <Node Name="RunFunctionalTests" Requires="$(CompileEditorNode)">
        <Command Name="RunUnreal"
          Arguments="-project=$(ProjectFile) -build=Editor -test=UE.EditorAutomation -runtest=&quot;Project.Functional Tests.Maps.TestLevels.FunctionalTests.TM_FunctionalTests&quot;" />
      </Node>
    </Agent>
4

I forgot to get the test results to be published to Horde you will need add this commandline argument:

-WriteTestResultsForHorde

5

Thank you for the information. I think there’s something fundamental I’m not understanding here. I’ve made some changes in my many, many attempts to get this to work.

I have a collection of Blueprint Functional Tests. The Blueprints are named in a pattern like:

BP_CrouchTest

BP_JumpTest

… etc

They are added to a map called TM_FunctionTests. In the map, they are named like so:

FTG_CrouchTest

FTG_JumpTest

I changed my BuildGraph to match what you provided.

I’ve added a GauntletTest to also try to get my test to run and have failed.

The GauntletTest looks like so (although I’ve gone through several dozen variations attempting to get this to work):

using Gauntlet;
using UnrealBuildTool;
 
public class SmokeTests : UnrealTestNode<UnrealTestConfiguration>
{
	public SmokeTests(UnrealTestContext ctx) : base(ctx) { }
 
	public override UnrealTestConfiguration GetConfiguration()
	{
		var config = base.GetConfiguration();
 
		var editor = config.RequireRole(UnrealTargetRole.Editor);
 
		editor.CommandLineParams.Add("ExecCmds",
			"Automation OpenMap /Game/Maps/TestLevels/FunctionalTests/TM_FunctionalTests;" +
			"RunTests Project.Functional Tests.Maps.TestLevels.FunctionalTests.TM_FunctionalTests.FTG_*;" +
			"Quit;");
 
		config.MaxDuration = 10 * 60;
		return config;
	}
}

Currently, this also runs into similar problems as before - I can fuss around and get the editor to load different ways, sometimes get the level to load and sometimes not. But my tests never attempt to execute.

I tried running the following command, based on what you provided:

“C:\p4\main\game\Engine\Build\BatchFiles\RunUAT.bat” RunUnreal -test=UE.EditorAutomation -runtest=“SmokeTests” -project=“OurProject/OurProject.uproject” -build=editor

This fails to run any tests.

  No automation tests matched 'SmokeTests'
 
 #### Editor failed: No tests were executed! (TestFailure, ExitCode=-1)

I don’t understand the -runtest parameter. I can’t find any example of what it’s expecting in the documentation, even with searching. I can’t find where it’s defined searching the repo on github for “runtest”. I can’t find “Mytest.one” anywhere to figure out where it’s defined. I’ve tried passing it various parameters, such as FTG_*, SmokeTests, and more, and all have failed to discover any of my tests or error out (it doesn’t seem to accept regex, so the first one errors out for that reason).

Ultimately, my goal is to get the following stack to work:

  • Horde, which calls…
  • Buildgraph, which calls…
  • (optional) Gauntlet, which calls,
  • the Functional Test Blueprints in the given level.

And the results end up on the Horde Automation dashboard.

6

the command line argument -ExecCmds=“Automation <command>” does not have a OpenMap command.

And RunTests command does not need wildcard to run several tests. It is already a partial match.

So you could do this: `-ExecCmds=“Automation RunTests Project.Functional Tests.Maps.TestLevels.FunctionalTests.TM_FunctionalTests;Quit”`

Or more simply:

`-ExecCmds=“Automation RunTests TM_FunctionalTests;Quit”`

But to have Gauntlet interpret the output from the Engine automation test framework I strongly recommend to use UE.EditorAutomation test node.

So from the command line:

RunUAT.bat RunUnreal -test=UE.EditorAutomation -runtest="TM_FunctionalTests" -project=<path to uproject> -build=editorThe runtest argument is the what get feed to `-ExecCmds=“Automation RunTests <test string>;Quit”`

7

That got my tests running locally, thank you so much!

I still have to get them running in Horde, but I’ll try to figure that out on my own and will come back to this thread if I run into further issues.