Cooks sporadically hanging on Horde

Development Pipeline & Plugins
, ,
1

Hi there,

A few times a week we’ll see a build hanging on Horde on the cook step

Remoting onto the agent, we see the following

  • UnrealEditor process is frozen with unchanging, low, memory utilization
  • CrashReportClientEditor is also running with unchanging memory utilization
  • There’s a CrashReportClient.log that’s timestamped to when the cook started but no useful info there
  • No dmp files are present on the machine
  • Cook log makes it seem like cook completed successfully

After 24 hours, Horde cancels the cook job and agent works just fine on future jobs. Agent also works fine if we cancel the job

Not finding any reports of similar issues on UDN, wondering if anyone else has hit this issue or if you have any advice on how to troubleshoot?

Thanks,Adam

2

Steps to Reproduce
Fire off dozens of jobs with cooks and eventually one will freeze

3

This sounds like a deadlock. We have seen deadlocks in the past, due to e.g. locking two critical sections in different orders from different threads. I can’t remember any in 5.6 though.

The next time this occurs, could you remote into the machine, attach to the process, and capture all-threads callstack, and post it here?

Are your build machines running windows? If so, you can capture callstacks with windbg (.pdbs must be present on the machine):

You might already have windbg installed. In the start menu, type windbg (x64) and run if it exists. If you don’t have windbg it can be downloaded as part of the Windows 10 SDK https://developer.microsoft.com/en\-us/windows/downloads/windows\-sdk/, make sure you install “Debugging Tools for Windows”. Once windbg is running, to attach, do File->Attach to a process. Find UnrealEditor.exe in the list, hit attach, wait for it to load up (this usually takes around 30 seconds), then in the console at the bottom put this in to print out callstacks for all threads: ~*k

4

We are also seeing this. I’ve captured a callstack and added on a private question.

5

We’re still working to verify this, but possibly the hang for that callstack was fixed in 5.7 by CL43606852 aka github commit 869644589dfec1f758b8340244156c03da923b51.