[Horde] Recommended storage strategy for large BuildGraph compile outputs with S3

Development Pipeline & Plugins
, ,
1

We’re currently investigating the amount of storage traffic generated by our Horde jobs after moving our Horde storage backend to S3.

A significant portion of that traffic appears to come from compile outputs that are stored by Horde and transferred between agents.

For example, a typical job currently generates approximately:

  • Compile Editor Win64: ~7 GB
  • Compile Tools Win64: ~5 GB
  • Compile PS5: ~9 GB
  • Compile Win64: ~1 GB

We understand why these outputs need to be stored and transferred. Our compilation and packaging/cooking steps run on different agents, so Horde needs to make those build products available to downstream nodes. This is also an important part of the parallelization we want to retain.

The issue is simply the scale of the traffic. With multiple platforms, configurations, regular builds and preflights, we can generate hundreds of GB of temporary build products in only a few days. With S3 as the storage backend, this can become fairly expensive.

What is Epic’s recommended storage strategy for this type of Horde deployment?

In particular:

  • Is S3 generally intended/recommended as the primary backend for these temporary BuildGraph outputs in a high-volume build environment?
  • Are there recommended ways to reduce the amount of data uploaded/downloaded while retaining BuildGraph parallelization across agents?
  • Is there a recommended approach for keeping short-lived/inter-agent data on local or on-prem storage while using S3 for artifacts that require longer-term storage?
  • Are there caching, retention, agent topology, or storage configuration options we should be looking at specifically for large compile outputs?
  • More generally, how does Epic typically handle this internally for projects where individual compile nodes can produce several GB of data across many builds per day?

Our goal isn’t to avoid the intermediate transfers, we understand why they’re necessary, but rather to make sure we’re using the intended storage architecture for this kind of workload instead of unnecessarily pushing a very large amount of short-lived data through S3.

[Attachment Removed]

2

Hello! Sorry for the delay in getting back to you on this - I wanted to ensure we got a few eyes on these answers to be as accurate as possible.

1. Is S3 generally intended/recommended as the primary backend for these temporary BuildGraph outputs in a high-volume build environment?

Yes, S3 is the recommended storage backend to prevent uploads and downloads from going through the Horde server process, impacting performance.

If you’re on-prem, S3-compatible storage can be used, but is unsupported.

The step output should be short-lived to keep costs down, but if there are a lot of builds, there’ll be a lot of outputs, even if short-lived.

2. Are there recommended ways to reduce the amount of data uploaded/downloaded while retaining BuildGraph parallelization across agents?

Unfortunately, no, beyond reviewing the files that you tag and upload to ensure that they are all required.

3. Is there a recommended approach for keeping short-lived/inter-agent data on local or on-prem storage while using S3 for artifacts that require longer-term storage?

Our recommendation would be to use S3 for this storage because of the advantages over local storage.

S3-compatible on-prem storage could be worth investigating, but isn’t something that we have tried internally.

The “step-output” artifact type is what captures the temporary data moved between agents, which is part of the “horde-artifacts” storage namespace.

Each namespace can be assigned a storage backend (S3, local disk, etc.). However, at the moment, you can’t use a separate storage backend for “step-output” from the other artifacts.

You could experiment with manually using BuildGraph “After” semantics to copy the files as a part of your BuildGraph steps, but you would need to handle the retention and clean-up, too.

4. Are there caching, retention, agent topology, or storage configuration options we should be looking at specifically for large compile outputs?

No, our build farm is fairly simple in this regard. We rely entirely on Horde artifacts to ship things between agents.

5. More generally, how does Epic typically handle this internally for projects where individual compile nodes can produce several GB of data across many builds per day?

This is an ongoing cost that we accept. It’s worth noting that in Epic’s case, our build machines and most of our infra are on EC2 already, so using S3 has not been an issue.

However, if your build machines are on-prem, the story is different (latency, bandwidth, egress costs, etc.), and egress costs will likely vary.

[Attachment Removed]

3

In our case we’ve replaced the tag-based “Produces” nodes for binary compiles with explicit <CreateArtifact and <RetrieveArtifact since we’ve seen that the “Dedupe=true” option helps a lot with them. We’re using a local S3 solution and we’re seeing around 50% effective deduping on 10GB binaries (example bellow is from a PS5 compile, with some changes we’ve made to get stats from all dedupe-writer fork child processes).

Copied 0/8 files (1889.2/9584.1mb, 377.8mb/s, 19%)
Copied 3/8 files (4457.6/9584.1mb, 513.6mb/s, 46%)
Copied 3/8 files (6668.8/9584.1mb, 442.2mb/s, 69%)
Copied 6/8 files (8999.4/9584.1mb, 466.1mb/s, 93%)
Stats:
Cache hits: 90565
Cache hits size: 5958238109
Cache misses: 60278
Cache misses size: 4095594286
Cache adds: 150528
Uploaded artifact 6a9878f59817ce91b9e394da in 22.5s (425.5mb/s, 3404.0mbps)
Artifact creation finished with no exceptions

With this in mind we’re keeping all our step-output small and keep them for 7 days, while keeping binaries artifacts to 1 day. This lets us do retries within the day for failed jobs, and lets us do 7 day retries for most other jobs, but just requires us to be mindful when retrying after a binary has expired

[Attachment Removed]