Questions Regarding Multi-Device Testing and Reservation Workflows

Development Pipeline & Plugins
, ,
1

Following up the previous one, I have a couple of questions regarding device testing and reservation workflows:

1) Multi-device mobile testing

How is testing across multiple mobile devices typically managed? Specifically:

- How many devices can be connected to a single agent at once?

- Can tests be run sequentially or in parallel across multiple connected devices?

One scenario we are considering is:

- Running performance tests across a large fleet of Android devices (e.g., 30 devices).

- Running similar performance tests across iOS devices (e.g., 31 devices).

We’d like to understand the best practices and limitations around test execution models and device management in this context.

2) Device lease renewal and failover handling

Once devices are reserved, we’d like to maintain those leases reliably. In particular:

- Is there a built-in mechanism to renew device reservations _in the background_ while tests are still running?

- This would help us handle situations such as agent machine failure, where we would prefer to release or reclaim devices proactively.

I also noticed BuildGraph nodes can use annotations like `Annotations=“DeviceReserve=Begin”`, but couldn’t find documentation on how these annotations behave. Specifically:

- Does this annotation ensure that device reservations are maintained for the duration of the node execution?

- Is it possible to explicitly manage reservation lifetimes using annotations or another mechanism?

Thanks in advance for your help and guidance.

2

One can run several tests in parallel using the command line argument `-parallel=n` where n is the number of tests running in parallel.

If there is enough devices provided into the device manager, Gauntlet should be able to handle the coordination.

However, this will put the pressure to the local agent. We don’t recommend using a virtual machine for this as the hardware is already under pressure to serve multiple hosts.

The vanilla device management system in Gauntlet is not meant to scale much, one need to provide at each command line the list of available devices and there is no way to communicate with other Gauntlet instance running on the same network.

Gauntlet is not meant to be a server, it is just a test runner and is highly dependent of the local resources. If the application under test is consuming a lot of resources (which is usually the case for highend games), Gauntlet is not going to be able to follow many instances.

So it is difficult to put a number here. The Agent hardware is a factor, the network also and how the devices are setup.

For more advance device reservation system we use Horde to do that, which is part of the open source code base. Horde is our in-house build system which include Agents and Devices management.

What you have peaked up in the Buildgraph is the pipeline we use to coordinate certain aspect of the device reservation. Gauntlet is able to communicate with Horde device reservation and Buildgraph can get be communicated information about reservation processes it should follow preferably.

You indeed realized that one of the approach that could be used is to try to keep the same device through out the same series of agent job step executions. That is what Gauntlet will use if the necessary information is provided at the command line to connect to the Horde device reservation system.

However, Horde is relatively new, and I have no idea if the documentation for it is explaining in details how to deploy the service. I mainly know the Gauntlet side; I can provide details for that side, but I have no idea how to setup the Horde service itself.