Reproductibility of Horde artifact upload to investigate timeouts on Agent sending blobs to Storage

Development Pipeline & Plugins
, ,
1

Hi Epic,

We’re currently testing Horde out to perform Packaged Build using a vanilla BuildAndTestProject.xml build graph (the one provided in Engine/Build/Graph/Tasks without modification). It works fine, except we sometimes get random HttpTimeout when artifacts get uploaded at the end of some compilation steps. We’re having trouble finding out the cause of these timeouts as they happen randomly between jobs, CLs, agents, time of the day, etc. See attached file for example of a failing job.

Similar topics here seem to solve this issue by simply getting more firepower (upgrading network/bandwidth, changing ISP, etc), or hardcoding a higher timeout in Horde source code. We cannot/do not want do that so we’re trying to pinpoint the cause of these timeouts.

It feels like a network/hardware issue, but it could come from the agent, server, storage, network between these machines, firewall, or a combinaison of those.

Agents are running on independant machines, Horde server on a Linux VM, and Storage is hosted on intenal hardware (i.e. no AWS). Artifact uploads seems to have to go through the Horde server to end up on Storage.

Monitorings of the server, the agents and the storage do not indicate obvious issue (we do not see 100% usage of CPU, Disk or RAM usage for example).

Something that could help us understanding what is going on would be to be able to trigger artifact upload manually with a dummy of arbitrary size, at arbitrary time, repeatedely and consistently, instead of having to spawn jobs and wait for a whole compilation step to run.

As we’re still kind of new to Horde as a Build Sytem, what would we an easy way for us to create a dummy artifact of several GB and just perform its upload to Horde to trigger these timeouts?

Thank you for your help.

Kind regards

This question was created in reference to: [Horde Http timeout when copying artifact from agent -> [Content removed]

[Attachment Removed]

2

Hello! I have attached a small BuildGraph script (ArtifactUploadTest.xml) and a job template snippet below.

If you drop the script under Engine/Build/Graph and add the template to your .stream.json, you should get an “Artifact Upload Test” job that you can kick off from the Horde Dashboard.

You may need to tweak the agent pools to match your agent/pool setup (I set it to Win64).

The test generates random test data on one agent using the built-in RandomData task, and reads it back on another, forcing the output through the server upload path.

The defaults produce about 10 GB, which you can adjust in the dashboard dialog.

Another option, if you would rather not add a template, is to use the Horde command line tool, which uses the same mechanisms.

If you don’t already have it, you can download it from the Horde Dashboard under Tools > Download > Misc Tools tab.

You’ll need to run it from an agent to replicate the same conditions, though. For example:

horde archive create -Input=<dummy-folder> -Ref=test/timeout -Namespace=horde-artifacts

From your logs, throughput appears healthy right up to the failure, so this does seem like a connection stall on the network side, not bandwidth-related.

Hopefully this test job helps with network troubleshooting. Let me know if there’s anything else we can do to help!

<?xml version='1.0' ?>
<BuildGraph xmlns="http://www.epicgames.com/BuildGraph" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://www.epicgames.com/BuildGraph ../Schema.xsd" >
 
	<Option Name="ProducerAgent" DefaultValue="Win64" Description="Agent type that generates and uploads the data"/>
	<Option Name="ConsumerAgent" DefaultValue="Win64" Description="Agent type that downloads the data"/>
	<Option Name="FileSize" DefaultValue="52428800" Description="Size of each generated file, in bytes (default 50 MB, must be under 2 GB)"/>
	<Option Name="FileCount" DefaultValue="200" Description="Number of files to generate (default 200, for ~10 GB total)"/>
 
	<Agent Name="Upload Test Data" Type="$(ProducerAgent)">
		<Node Name="Upload Test Data" Produces="#TestData">
			<RandomData OutputDir="$(RootDir)/Engine/Saved/TestData" Size="$(FileSize)" Count="$(FileCount)" Different="true" Tag="#TestData"/>
		</Node>
	</Agent>
 
	<Agent Name="Download Test Data" Type="$(ConsumerAgent)">
		<Node Name="Download Test Data" Requires="#TestData">
			<Log Message="Downloaded test data from shared temp storage"/>
		</Node>
	</Agent>
 
	<Aggregate Name="Artifact Upload Test" Requires="Download Test Data"/>
 
</BuildGraph>

// Add this template to the "templates" array of your .stream.json.
{
    "id": "artifact-upload-test",
    "name": "Artifact Upload Test",
    "arguments": [
        "-Target=Artifact Upload Test",
        "-Script=Engine/Build/Graph/ArtifactUploadTest.xml"
    ],
    "parameters": [
        {
            "type": "Text",
            "label": "File size",
            "argument": "-set:FileSize=",
            "default": "52428800",
            "validation": "[0-9]+",
            "validationError": "Must be a whole number of bytes (keep under 2147483647)",
            "hint": "Size of each generated file in bytes (default 50 MB)"
        },
        {
            "type": "Text",
            "label": "File count",
            "argument": "-set:FileCount=",
            "default": "200",
            "validation": "[0-9]+",
            "validationError": "Must be a whole number",
            "hint": "Number of files to generate (200 x 50 MB = ~10 GB)"
        }
    ]
}

[Attachment Removed]

3

Hi,

Thank you for you help, with your setup we’re now able to easily get the timeouts. That’s exactly what we needed!

[Attachment Removed]