We have been experiencing degraded reliability in our Horde ecosystem over the last couple of months and we’re not sure what to try in order to fix our issues. The first is a “Lease Incomplete” issue that kills a running batch and then performs some sort of retry. Sometimes it seems to redo everything, other times not. I’m not sure exactly what it is doing there. But what appears to happen is that the agent and server become disconnected, sometimes for a huge period of time, then horde kills the batch and retries is. It is causing really significant delays, and you know how it is with builds, they’re never fast enough, lol. I’ve attached an example of an audit of one of the jobs that had this issue. My questions are: what kind of error are we looking at, and how do we diagnose the cause? I’ve never seen Horde do this before. We’ve restarted both the containers running the server and the host that the containers are hosted on. We’ve looked at all the telemetry we have and there isn’t a smoking gun. I haven’t been able to find evidence in the Horde server logs either. I see the lease termination and restart but not any reason that it would do so, at least temporally correlated. We’re having a hard time diagnosing this one.
In case it is related, we are also seeing a timeout when blob uploads are happening to the horde server, I’ve attached the log file for one of those. In general, it seems like we have connectivity issues, but none of us have been able to trace it to anything specific.
Thanks for your help, and sorry there’s so little to go on.
Hello! Thank you for the write-up and for sharing these logs up front.
Unfortunately, I think you may have unintentionally left some secrets in the timeoutUploading.log file (lines 6-15), so I’ve deleted the attachment.
On the blob upload timeouts, you might be interested in cherry-picking this change, which came in this week to increase the HttpClient timeout for storage requests to 300s.
This is a self-contained change and should hopefully reduce the occurrence of these timeouts for you.
*"*Horde: Increased the HttpClient timeout for storage requests to 300s.
It currently uses the default 100s but this can cut short the retry policy used by storage requests causing unecessary job step failures where a transient network error was actually recovered from."
Thank you for sharing the log files. There were quite a few attached, so I haven’t yet reviewed all of the lease files outside of cross-file, broad searches.
Could you please confirm that the cancelled job you’re referring to was the cancelation on line 673 of Log-20260811.txt?
2026-08-11 03:58:20.791 -07:00 [ERR] Unhandled exception while executing lease 6a7affbdf1235ee8cbd43979: The operation was canceled.
For this error, the agent appears to have cancelled the lease, rather than the server.
It also appears to be a remote-execution helper for the lease 6a7ae52ef1235ee8cbd362b4, which wasn’t in the attachments, and was running on another agent (build14).
In 6a7affbdf1235ee8cbd43979.txt, you can see that the agent terminates the lease due to a timeout.
Would you be able to check the corresponding logs from the other build machine to see if there are any other indications as to why the data wasn’t sent?
You could also check the dashboard to see the job cancelation reason.
2026-08-11 03:58:10.505 -07:00 [WRN] Terminating compute transport due to timeout (90000 ms with last tick at "2026-08-11T10:56:40.2493549Z")
Aside from this cancelation, the other cancelations that I saw in the leases were “cancelled by server”.
We don’t currently pass the reasons through in these messages, but these can be simple cases such as user-initiated cancelations or pre-flights that are superseded.
You should be able to check the job/step page header in the dashboard to see if these were user-cancelled, if these lease cancellations are of concern.
Thank you for confirming the lease and for sending over the system event logs.
The analysis looks good to me, and checking the event logs, I found Windows system upgrade/restart notifications that line up with the cancellation.
The system restart message comes through at 10:30:06, and the event log service stops at 10:31:08, when the agent logs `Application is shutting down…` in `Log-20260812.txt`.
> Get-WinEvent -Path .\8_12_9am_to_11am_System.evtx -FilterXPath "*[System[EventID=1074 or EventID=6006]]" | Sort-Object TimeCreated | Format-List TimeCreated, Id, Message
TimeCreated : 12/08/2026 18:30:06
Id : 1074
Message : The process C:\WINDOWS\uus\AMD64\MoUsoCoreWorker.exe (BUILD16) has initiated the restart of computer BUILD16 on behalf of user NT AUTHORITY\SYSTEM for the following reason:
Operating System: Service pack (Planned)
Reason Code: 0x80020010
Shut-down Type: restart
Comment:
TimeCreated : 12/08/2026 18:31:08
Id : 6006
Message : The Event log service was stopped.
...
Could you check the automatic upgrade/reboot settings on the agents? It looks like automatic updates are enabled.
If you would like to keep these enabled, you could schedule them at times that wouldn’t affect your jobs.
Due to scheduling I haven’t been able to keep as close an eye on it as I wanted to. I will try this week. We did make another change in the issue tracking that seems to have generally helped our stability. Basically, we excluded jobs that report lots of failures (e.g. - testing jobs) and we segmented the other workflows into smaller subsets. FWIW, that did help overall.
I logged into the machine immediately after the event, so it should be almost the last thing in the logs. I asked claude to summarize it for me, and attached the result. It’s at the end of Lease 6a7c9626f1235ee8cbe5f755.
facepalm. Automatic updates are on. I will take that to IT.
As a note, since splitting up our workflows, we’ve experienced a lot fewer issues in general with the system. That might be valuable information for someone. Basically we went from single digits of workflows to 35 separate workflows.