Gauntlet: Horde device reservation blocks fail with more than one device (fix in PR #15239)

Development Pipeline & Plugins
, ,
1

Background: this follows up on our earlier case [Content removed] where you confirmed that limiting Horde device reservation blocks to a single device was not intentional and would be fixed in a future release. It is still present in 5.8, ue5-main and ue6-main, so we have written a fix and opened a pull request: https://github.com/EpicGames/UnrealEngine/pull/15239

PROBLEM

A device reservation block (a step annotated DeviceReserve=Begin installs a build; later steps in the batch run with -SkipInstall on the same devices) only works with one device. With two devices the install step is given the same device twice (“Installing on 2: deviceA, deviceA”), so the build is installed twice on one devkit and the second devkit is never reserved. A later step that asks for one device gets both devices of the block back and fails.

CAUSE

HordeDeviceReservationService.ReserveDevicesFromService (Gauntlet.DeviceReservationService.cs) sends one reservation request per device and requires exactly one device back (DeviceReservation.Devices.Count != 1). Horde answers any request made from a step inside a reservation block with the block’s existing reservation (DeviceCollection.TryFindReserveBlockAsync), whatever was asked for.

FIX (PR #15239, one file, +56/-6, against ue5-main)

Grouped requests: constraints without an explicit DeviceName that share the same tags go in one reservation request, and the returned devices are paired with their constraints (by platform, otherwise in order). Named devices keep one request each. Outside a reservation block the result is the same as before: one request for N devices allocates what N single requests did.

Count check: it compares against the number of constraints in the request, and a failed request releases the reservations already made in that call.

Release: ReleaseDevices now uses the DisposeReservation value it already computes, so a shared reservation is released only with its last device.

TESTING

On Horde: a pool of two console devkits, an install step followed by several test steps. Before the fix, two requests and the same devkit twice; with the fix, one request, two different devkits, one install each, and every later step reused both devkits with -SkipInstall.

Locally: the unmodified and fixed service run against a local stand-in for the Horde device endpoints that applies the same reservation-block rule (9 scenarios: install step, follow-up step, one-device request inside a two-device block, different tags, named devices, no tags vs empty tags, shared-reservation release, failed registration, failed 409 request).

QUESTIONS

Question 1: Will you take this change (PR #15239)?

Question 2: Is ue5-main the right base for the PR, or would you rather have it against ue6-main? ue6-main has the same loop; there the new UpdateAndVerifyDeviceOSTags call needs to stay inside the per-device loop.

Question 3: If you already have a different fix planned or submitted, could you share the CL so we can backport it instead?

Thanks!

[Attachment Removed]

2

Steps to Reproduce
Step 1: Set up a Horde device pool with two console devkits and run Gauntlet with -DeviceURL and -DevicePool.

Step 2: In a BuildGraph job, add an install node annotated DeviceReserve=Begin that needs two devices (two client roles), followed by test nodes in the same agent batch that run with -SkipInstall.

Step 3: Run the job.

Result: the install step makes two reservation requests and logs “Installing on 2: deviceA, deviceA” (the same devkit twice; the second devkit is never reserved). A following step that requests one device gets both devices of the block back and fails with “Unable to make device registration with constraint …”.

Expected: one reservation with two different devkits, one install on each, and later steps reuse both devkits with -SkipInstall.

[Attachment Removed]

3

Thank you for the submission! I can confirm that we’ve received that on our side, and it has been routed to the correct team for review.

Unfortunately, I can’t confirm at this time whether the submission will be accepted or when it will be reviewed.

For future submissions, ue6-main would generally be the right base to target, due to minimized code drift and context switching for the team.

For this PR, I wouldn’t worry about retargeting ue6-main, as it hasn’t diverged too much.

There is no fix submitted for this issue internally, so we’ll need to wait on the review.

[Attachment Removed]