Gauntlet does not seem to detect added devices in horde device management.

Development Pipeline & Plugins
, ,
1

Hi!

So I’m working on enabling/configuring Horde device manager as our automation team begins ramping up Gauntlet tests. I worked off of the Devices.md documentation, and got the plugin running and was able to add some test devices to our Horde instance. For testing purpose, I was using a machine already being used for gauntlet tests that’s also connected as a horde agent.

The Guantlet run for an editor test starts with:

-project=<our uproject> -test="UE.EditorAutomation" -runtest="L_ATS_AssaultCourse" -execcmds="<debug args for the game>" -args=-log -args=-nosound -args=-ResumeRunTest -build=editor -maxduration=3600 -unattended -verbose -ResumeOnCriticalFailure -deviceurl="<https://our horde url>" -devicepool="UE5" -PerfModel=Service' -WriteTestResultsForHordeThis does not seem to have an effect on the test other than adding a line of: “Unable to reserve service device of type: Win64”. I setup the platform to use a Win64 name and a model of service to see if there was some hidden setting in our BuildGraph requiring the two, but this did not change the results. I also got duplicate results when running a test against a full game client.

I’ve been reading through articles here and the code base mentioned in some of those, but can’t help but feel I’m missing a configuration step here. Any assistance would be great!

[Image Removed]

Update: I did get another error on the client build:

Unable to reserve service device of type: Win64
Test <game>.Automation.MinimalGameFlowTest (Win64 Test Client) threw an exception during ready check. Ex: No enabled reservation services were capable of supporting the following constraints:
Win64:Service
The following reservation services were available:
HordeDeviceReservationService
Ensure your environment is configured to enable any necessary services.

[Attachment Removed]

2

Hey there Sean,

Do you happen to have your *.json configuration snippets that correspond to how you’ve setup your device pools? Also, do you happen to have a more complete log for the Gauntlet run that we could review?

Kind regards,

Julian

[Attachment Removed]

3

Desktop platforms do not need to reserve remote devices as they can only be run through the agent.

And anyway Editor build can’t be deployed. Only staged or packaged builds can.

So for your command line to work you need to do:

-project=<our uproject> -test="UE.EditorAutomation" -runtest="L_ATS_AssaultCourse" -execcmds="<debug args for the game>" -args=-log -args=-nosound -build=editor -maxduration=3600 -unattended -verbose -ResumeOnCriticalFailure -WriteTestResultsForHorde

[Attachment Removed]

4

Hi Julian, the guantlet log is pretty scarce. It started with a command line of:

"D:\<game+name>\Sync\Engine\Binaries\DotNET\AutomationTool\AutomationTool.dll" -p4 -NoCompile RunUnreal -project=<game.uproject>-test="<game>.Automation.MinimalGameFlowTest" -platform=Win64 -configuration=Test -build="<game build location>" -maxduration=3600 -unattended -verbose -ResumeOnCriticalFailure  -deviceurl="https://horde.internal.<company>" -devicepool="UE5" -PerfModel=Service -numclients=1 -targetenvironment=<environment>-args=-game -args=-customconfig=noEOS -args=-workspaceRoot=<workspace> -args=-nosound   -WriteTestResultsForHorde
 
 
Initializing script modules...
Total script module initialization time: 0.38 s.
AutomationTool is running as a child instance (uebp_UATChildInstance=1)
Using C:\Program Files (x86)\Microsoft Visual Studio\2022\BuildTools\MSBuild\Current\Bin\MSBuild.exe
Executing commands...
Unable to reserve service device of type: Win64
Added device Win64:default to pool
Mapped Role Unknown to RoleContext Win64 Development Unknown
Mapped Role Editor to RoleContext Win64 Development Editor
Mapped Role EditorGame to RoleContext Win64 Test EditorGame
Mapped Role EditorServer to RoleContext Win64 Test EditorServer
Mapped Role Client to RoleContext Win64 Test Client
Mapped Role Server to RoleContext Win64 Test Server
Mapped Role Host to RoleContext Win64 Development Host
Mapped Role CookedEditor to RoleContext Win64 Development CookedEditor
<LOTS OF GAME FEATURE PLUGINS ETC>
Adding Win64 StagedBuild | Configuration: Test | Flavor: None | Flags: Loose, CanReplaceCommandLine, CanReplaceExecutable | PreferenceOrder: 1
Win64:Service: 1 devices required. Total:0, Available:0
Unable to reserve service device of type: Win64
Test <game>.Automation.MinimalGameFlowTest (Win64 Test Client) threw an exception during ready check. Ex: No enabled reservation services were capable of supporting the following constraints:
Win64:Service
The following reservation services were available:
HordeDeviceReservationService
Ensure your environment is configured to enable any necessary services.
Status: Completed:1, Running:0, Starting: 0, Waiting:0

The json configuration:

"devices": {
    "platforms": [
        {
            "id": "android",
            "name": "Android",
            "models": [
                "Pixel6",
                "Pixel7",
                "Pixel8"
            ]
        },
        {
            "id": "ios",
            "name": "iOS",
            "models": [
                "iPhone12",
                "iPhone13",
                "iPhone14",
                "iPhone15"
            ]
        },
        {
            "id": "PS5",
            "name": "PlayStation 5",
            "models": [
                "Standard",
                "PRO"
            ]
        },
        {
            "id": "XboxSeriesX",
            "name": "Xbox Series X",
            "models": [
                "Standard"
            ]
        },
        {
            "id": "Win64",
            "name": "Win64",
            "models": [
                "Service",
                "Workstation",
                "Build"
            ]
        }
    ],
    "pools": [
        {
            "id": "ue5",
            "name": "UE5",
            "poolType": "Automation"
        },
        {
            "id": "remote-ue5",
            "name": "Remote UE5",
            "poolType": "Shared"
        }
    ]
},

In regards to the below statement about this service not working for desktop platforms, this is definitely a proof of concept for mobile/console testing but I don’t see any documentation about Win64 not being a valid platform for this type of management?

[Attachment Removed]

5

No, it is not documented. But I have a fairly good knowledge about Gauntlet capabilities.

I’m one of the devs that works on this framework. And Desktop platforms are not supported by the Device reservation system.

[Attachment Removed]