Hi,
We’re running Horde 5.8 with on-prem Windows build agents, no AWS anywhere. Every job step had an unexplained ~16 second gap in the log before it launched UAT, so I went digging.
It’s the EC2 instance metadata lookup in JobExecutor.ExecuteCommandAsync (Engine/Source/Programs/Horde/Drivers/JobDriver/Execution/JobExecutor.cs, line 2150 in 5.8, line 2278 in ue5-main):
On a machine that isn’t in EC2 there’s nothing at 169.254.169.254, so each of those blocks until the AWS SDK gives up - 5s timeout x 3 retries plus backoff, about 15.75s per probe. Both values come back empty and get thrown away. ExecuteCommandAsync runs once per step, so on a 10-step incremental build we’re losing 2-3 minutes per job to it.
The part that surprised me: there’s already an agent setting for this, EnableAwsEc2Support, and it defaults to false. CapabilitiesService checks it (line 356), but the JobDriver call site doesn’t - I can’t find it referenced anywhere under Drivers/JobDriver. So the agent is configured not to touch EC2, honours that in its own capability reporting, and then probes IMDS twice per step regardless. I’d guess that’s fallout from the driver being split into its own process, since the setting lives in AgentSettings and DriverSettings has no equivalent.
This looks like the leftover from “Horde Pipeline error indicates trying to use AWS, when ‘EnableAwsEc2Support’ is set to false in the Agent” (Aug 2024), which was closed off by CL 34124349. Comparing 5.4 with 5.6+, that CL wrapped these two lines in a try/catch. It stopped the exception, but the probe still runs - so the crash is fixed and the delay isn’t, and now it’s completely silent because the catch is empty.
Workaround that’s working for us: set AWS_EC2_METADATA_DISABLED=true as a machine-level environment variable on each agent and restart the agent service. The SDK’s IsIMDSEnabled picks it up and the getters return null immediately. It has to be in the agent process environment - setting it per-step or per-agent-type is too late, the probe happens before the child process is spawned.
Might be worth honouring EnableAwsEc2Support at that call site, or just caching the result, since whether the box is in EC2 isn’t going to change while the driver is running. And a debug line in that empty catch would have saved me a fair bit of time.
[Attachment Removed]