[Horde] Job steps spinning idle forever

Development Pipeline & Plugins
, ,
1

Hello and thank you in advance for a moment of your time,

After we upgraded our Horde Server and Agents from 5.6 to 5.7 we started noticing the occasional job step that would just run forever unless manually cancelled. (~2 per week)

The jobs themselves have timeouts significantly shorter than what they ran for here on these example screenshots. So this, we’re assuming, is on the horde server - agent communication losing track of something and dangling.

We’re weary of upgrading to 5.8 and potentially introducing more instability before knowing that this is addressed.

Any pointers or guidance are much appreciated

[Attachment Removed]

2

Steps to Reproduce

[Attachment Removed]

3

We have seen one similar issue, where leases can remain active indefinitely when the agent encounters a problem despite reporting healthily.

In this particular investigation, we found that the lease/session handling seemed to be working, but the agent was stuck due to a corrupt filesystem.

This could very well be an unrelated issue with a different root cause, but in this case, it wasn’t related to the Horde version.

Generally, we don’t have a mechanism today to manage lease timeouts or check-ins, but we have discussed this recently within the team.

To aid in designing and prioritizing this, could you let me know whether you can diagnose the issues with the agents that may be causing this?

In our specific case, the filesystem corruption was triggering a System.IO.IOException that was being swallowed and never logged.

We were able to confirm the problem with the following command: C:\Windows\system32>fsutil dirty query D:

Please see the “Practical Debugging Tips for Horde” article for tips on configuring log verbosity and grabbing agent logs.

https://dev.epicgames.com/community/learning/knowledge\-base/oWG6/unreal\-engine\-practical\-debugging\-tips\-for\-horde\#hordeagent\&serverstate

[Attachment Removed]