Horde job timeout at the horde server level

Development Pipeline & Plugins
, ,
1

This question was created in reference to: [Horde job [Content removed]

Hello,

We are dealing with various build system instabilities and would like to use timeouts and retries as a last line of defense.

We have implemented the following:

  • Timeout and retry attributes at the BuildGraph task level
  • Horde job timeouts mentioned in my referenced UDN thread

The issue is that, after upgrading to Horde from 5.7, we started experiencing hangs where the Horde agent finishes execution, but from the server’s perspective, the job still continues. Timeout in JobExecutor does not catch it. (For the thread about root cause of the hang, see [Content removed]

For this thread, I would like guidance similar to the referenced topic, but with an additional hint on where to look if we wanted to move this timer out of JobExecutor higher up the chain to the Horde server, so that it catches any kind of hang.

Thank you,

Ondrej

2

Hello, as I mentioned in the linked case, we’ve discussed this internally, but we don’t have a solution in place today.

One idea we had was a “safe fallback” mechanism: if we see that a lease has been alive for >X hours, the server checks whether any steps in that lease are still executing.

If they aren’t, it properly exits that lease, and we log an event. This is a more involved approach than a threshold check, and is still in discussion internally.

There are a couple of places on the server side that would make good starting points for simpler timeout threshold checks, if you wanted to put something in yourself again.

1. Extending TickAsync in JobTaskSource.cs to also loop over Starting/Running batches and compare elapsed time against a per-batch or per-step deadline.

This would probably be the simplest way to make this change, as the function already runs periodically and queries jobs.

https://github.com/EpicGames/UnrealEngine/blob/5\.7\.4\-release/Engine/Source/Programs/Horde/Plugins/Build/HordeServer.Build/Jobs/JobTaskSource.cs\#L360

2. Creating a new hosted service, similar to JobExpirationService, that periodically checks jobs against a timeout threshold and cancels them.

This is the cleaner approach, but you would need to add a new public function in JobService.cs to set the batch error to JobStepBatchError.Cancelled, update the batch state to Complete, and call CancelLeaseAsync().

https://github.com/EpicGames/UnrealEngine/blob/5\.7\.4\-release/Engine/Source/Programs/Horde/Plugins/Build/HordeServer.Build/Jobs/JobExpirationService.cs

Please let me know if this is helpful, or if you run into any issues implementing either of those.

3

Thank you for the switch reply. We implemented a solution that is very similar to option 2. Additionally, we also send a service notification, similar to what we do for failed config updates. So far, we have tested it only synthetically, and we are now waiting for the next real hanging build to verify it properly. We will let you know how that goes.