Horde Questions/Artifact Copy Redundancy

Development Pipeline & Plugins
, ,
1

We’re a small studio, and don’t have a ton of build agents(only 2 currently)

Up to now, I’ve been managing a team city server, that puts these 2 agents to work building UGS binaries, a client, and dedicated server build.

I’m looking at Horde now, because I think it has some really nice potential upsides, but there are some workflow differences that are still rather fuzzy to me, and it seems to be built for a lot more agent parallelism than what we’re providing it at the moment, and I think the build turnaround is suffering a bit because of it.

First off, some thoughts/questions

  • It looks like Horde does a lot of built in artifact copying that we have not really had to deal with thus far with our team city build pipeline. As a result of this not being needed thus far, we skimped on our bare metal build agent boxes with 1GB NICs, which I think is hurting the Horde performance significantly. Does that seem correct?
    • The only data in/out under our team city build system was
      • UGS binaries pushed to perforce
      • Client/Server binaries pushed to steam.
  • Is there a way to limit the scope of a build target to a single agent, to avoid these costly copy steps?
    • Build client all on 1 agent
    • Build server all on 1 agent
    • Build UGS binaries all on 1 agent
    • Obviously it would eliminate parallelism within each of those build targets, but when using limited agents, I think that is preferrable.
  • How does epic handle these frequent copying of many gigs of data?
    • Doesn’t Epic run their stuff in the cloud?
    • Does Epic pay exorbitant data transfer costs?
  • Is there a way to reduce redundant copying? (Agents redownloading artifacts that it originally uploaded itself, see below)
  • Even worse, suppose you build HLOD with BuilderCount set to something high, say 10, but not enough agents are available, so the same agent ends up processing multiple build jobs. Each build job will needlessly re-downloading the same artifacts for each build job it runs, wasting large amounts of time just in copying the same data multiple times.

For instance, consider the HLOD build script

[Image Removed] I ran this with a BuilderCount of 3, even though I only have 2 agents, to see what it does when an agent has to process multiple build jobs. Sure enough, when CurseAgent2 processes the 2nd build job, it downloads the 60GB Compile Editor Win64 artifacts all over again, despite already having done that when it processed the first job. Surely there is a way to allow the agents to reuse these artifact download caches? The data transfer wastage is off the charts.

I am trying some test runs of HLOD building, as per the BuildWorldPartitionHLODs.xml script, and noticed a couple things.

  • I am using an incremental build workspace for this
  • The Compile Editor Win64 step spends all its time uploading 59Gb artifact, presumably to the server instance, running on CurseAgent2
  • Once this step is complete, the processing begins to “go wide” across multiple agents.
  • CurseAgent1 and CurseAgent2 both then pick up on this workload
  • Each agent spends about 7 minutes re-downloading these artifacts before it runs the actual build job
    • CurseAgent2 is the one that uploaded these artifacts, is it not possible for this agent to use the data it already has?
  • I’m not sure if HLOD can be run successfully under -AllowSoftwareRendering (because neither of our agents has a GPU atm)
  • Are autoSDK folders nuked on non incremental builds and pulled fresh?
  • If so, how do you handle tooling that should be installed once, but not deleted or messed with later? For example steamcmd.exe, the command line tool to push builds to steam. You can’t be nuking the steamcmd folder all the time, or you will have to redo your steam guard authentication.
2

Hey there,

Appreciate the questions - they get at some pretty important concepts here.

Fundamentally speaking, Horde itself doesn’t know ahead of time which Agent will be servicing a particular <Agent> tag. This means it can’t make an assumption that a across <Agent> tags, file dependency shouldn’t be uploaded (as it would just end up on the same machine). As a result, if you know your topology is best served by less parallelization, I’d consolidate within tan <Agent> tag.

I hope this clears things up on how BuildGraph is very much the “brains” in the build execution space.

Kind regards,

Julian

3

Hey there,

Apologies - missed those two in the original reply.

  • No, and they will be updated on regular syncs (just like a regular managed workspace).
    • We use the “base” property in workspaceTypes to reuse a fundamental workspaceType that contains AutoSDK defined items.
  • As an FYI, I know we do place other types of binaries at times in tree - you could also place things in your regular sync (Extras folder, comes to mind)

I hope that clears things up. It’s also important to note that you should be configuring your AutoSDK streams at the global level.

Kind regards,

Julian

4

Thanks for your response.

Can you speak to the last 2 bullets please.

  • Are autoSDK folders nuked on non incremental builds and pulled fresh?
  • If so, how do you handle tooling that should be installed once, but not deleted or messed with later? For example steamcmd.exe, the command line tool to push builds to steam. You can’t be nuking the steamcmd folder all the time, or you will have to redo your steam guard authentication.

A good example of this is steamcmd.exe, the tool used to upload builds to steam. It’s important that the folder structure not be wiped/reset, or you will need to reauthenticate the agent with a steam guard code. So I’m just trying to understand the best approach to implementing this under the build system.

I know I can install steamcmd completely independently and invoke it from the path or something, but I would prefer to do it in a way that doesn’t burden each new agent rollout with custom preparation. I have an autosdk set up with horde already, I am just unclear as to the lifetime guarantees of the data there.

I suppose I could use a setup.bat from the steamcmd autosdk to manage my own “if not exist, install to another folder, out of band with horde/agent delete/repull operations”, but I wanted to ask if there is a better approach before I take that route.