Getting Blueprint Functional Tests to display in Horde Automation Hub dashboard

Development Pipeline & Plugins
, ,
1

I’m trying to get Blueprint Functional Tests to run in Horde and be displayed in the Automation Hub tab. I’ve got them running in Horde via:

“-set:EditorTestList=UE.EditorAutomation(RunTest=Project.Functional Tests.Maps.TestLevels.FunctionalTests.TM_FunctionalTests)”

This runs my tests correctly, which is great. However, they’re not being picked up by the Automation Hub. The Automation Hub is correctly picking up my Gauntlet tests, fired as follows:

“argumentIfEnabled”: “-set:TargetTestList=MainMenuTest”

When I navigate to my Automation Hub tab in Horde, I have a dropdown labeled “Automation” which only has a single option of “OurProject”, which correctly has every test run by TargetTestList but none of the tests run by EditorTestList.

How do I get these test results to be ingested by the dashboard properly? Every solution I’m investigating seems incredibly hacky. For example, I could make a modification to the Unreal Automation tool to make it change the suite name of Functional Test runs to “OurProject” instead of “Unreal Automation” (which it is now), but this seems very janky. It would also lose the distinction between Functional Tests / Gauntlet Tests, which is not ideal.

I can click on the Job Template / (specific run) / (test step) / Test Reports tab and see the Test Results correctly, so I know Horde is ingesting them properly, they’re just not showing on the dashboard.

2

Steps to Reproduce

  1. Write Blueprint Functional Test
  2. Have Horde job template that runs -set:EditorTestList= targeting test from step 1

Expectation: Test run results show up in Horde Automation Hub dashboard

Actual: Test run results do not display

3

Also I am passing the flag -WriteTestResultsForHorde and have been trying the BuildGraph from https://github.com/EpicGames/UnrealEngine/blob/6978b63c8951e57d97048d8424a0bebd637dde1d/Engine/Source/Programs/Horde/Docs/Config/AutomationHub.md\#L4 as well

4

Are you passing either or both of this command line argument to the UE.EditorAutomation test?

`-ResumeOnCriticalFailure` and/or `-ReportExportPath=`

The both will make sure the Engine tests are generating a json that will be read from Gauntlet and convert it to a format that Horde than can read back/ingest.

Engine tests need to be collected as “Test Suite” in order for Horde to display it in meaningful way on the Automation Hub.