Hi, we currently have a Horde server configured with 3 available agents. Each agent runs on a separate Windows machine. At the moment, we use all of them for incremental builds as well as nightly builds of both the editor and the game project. We have Unreal Build Accelerator enabled and everything works well together.
One of the agents is a powerful machine equipped with an AMD Threadripper CPU. We are considering splitting it into 4 independent agents. We tried an approach where we launched 4 separate Windows services, each running under a different user account, but the system prevented more than one HordeAgent.exe process from running at the same time. Additionally, as part of our attempt to configure 4 separate agents, we also tried downloading separate binaries for each agent into individual folders and launching HordeAgent.exe manually from the Windows command line (cmd). Each instance was configured independently in the appsettings.json file with parameters such as “Installed”: false, “CpuCount”: 16, as well as unique values for “Name”, “WorkingDir”, “ComputePort”, and the bearer “Token” assigned individually to each agent. However, this approach was also unsuccessful.
Is this expected behavior, or could we be missing something in our configuration? What are the recommended or proven approaches for splitting a single machine into 4 independent agents? Would running 4 Linux Docker containers be a good solution? If so, would we still be able to use those agents for the same tasks as our current Windows-based agents without any major issues?
Hello, I believe this is expected, as this isn’t typically how we’d recommend you set up your Horde Agents, which is probably why you’ve run into a few issues.
We recommend that Horde Agents run on dedicated machines. Could you please share your main goals for splitting the machine into separate Horde Agents?
With UBA distributing the compute, you should be able to leverage the powerful computer without needing additional agent processes.
Regarding running Horde Agents on Linux with Wine, you can read more info about this here:
Thank you for the clarification. Our main motivation for splitting the Threadripper machine into multiple Horde Agents was to increase task concurrency and overall agent availability. For example, we imagined scenarios where:
one agent handles an incremental editor build
another runs automated tests
a third packages or builds the game
while a fourth could be used for other build or CI-related tasks
From our perspective, this seemed like a good way to utilize a high-core-count machine more efficiently and avoid situations where one long-running task blocks the entire machine for other jobs.
We also wanted to better understand how this behaves together with UBA. For example, if one developer starts compiling the editor locally and UBA begins heavily utilizing the Threadripper machine as a helper, would another developer compiling at the same time experience significantly reduced performance because the helper machine is already occupied? Or does Horde/UBA handle this kind of resource sharing and scheduling efficiently internally?
We are mainly trying to understand the recommended architecture.
Thank you for the explanation. However, Horde Agent scaling has been designed around horizontally scaling machines, which is why we don’t have any guidance for your setup.
Though with UBA enabled, you should be getting the same throughput due to the high core count of the Threadripper machine.
If you’re encountering instances where a single user is consuming all available resources on your machines, you can use the UBA CLI option -UBAHordeMaxCores to limit the initiating machine’s total request for the session.
It seems to me that we are trying to achieve horizontal scaling specifically by partitioning the resources of a single powerful machine into several Horde Agents. Our goal is to ensure full coverage of the daily workflow for example: incremental editor builds, UBA compilation, asset cooking, final packaging, automated tests, and non-unity builds.
Could you advise on the recommended infrastructure for such a “wholesome” approach? We are now considering using Docker containers, but we would like to understand the limitations of this solution or if there is a better alternative.
Does this approach even make sense, or should we expand our farm with more physical machines instead of partitioning one? Our main priority is to ensure the availability of different agents for various tasks without them being blocked by long processes, such as non-unity code compilation.
The simplest approach would be to scale the farm with additional physical machines, as this aligns with our golden path for support and documentation.
However, that doesn’t mean the Docker approach couldn’t work, but you may encounter issues we haven’t run into before and aren’t easily able to replicate.
I suggest running with your existing setup and monitoring your build health to see if you’re actually hitting limits that need to be addressed now (unless you already are).
At Epic, we run Windows-based Horde Agents on EC2 instances using AMIs containing the Horde Agent and supporting toolchains.
We then run Linux-based agents for remote execution (Unreal Build Accelerator) using Linux and Wine that we can auto-scale independently.
We do run a large Horde setup, but this separation of agents is useful, and you may be able to mirror it as you grow your setup.