Thanks for the feedback. We went with option 3, and inserted a check() on the line saying “should not occur, panic”. (See attached zip)
That change greatly reduced the crashes -- we’re getting now 1-2 crashes per week across all boxes on the farm, cooks at developer desks, etc. It had been 20-30 per day before integrating 37590157, 10-15 per day with 37590157.
Unfortunately, that check() still occasionally fires -- that 1-2 crashes per week. I haven’t been able to catch this under the debugger. All I can tell offhand is that the EntryIndexPtr is nullptr. The callstack is the same as before; it may have been in garbage collection at the time.
13:08:43,400 INFO - LogWindows: Error: [CookWorker 1]: appError called: Assertion failed: EntryIndexPtr [File:D:\build\streams\main\game\Engine\Source\Developer\IoStoreUtilities\Private\ZenStoreWriter.cpp] [Line: 1063] 13:08:43,400 INFO - LogThreadingWindows: Error: [CookWorker 1]: Runnable thread TAsyncThread 0 crashed. 13:08:43,403 INFO - LogWindows: Error: [CookWorker 1]: begin: stack for UAT 13:08:43,403 INFO - LogWindows: Error: [CookWorker 1]: === Critical error: === 13:08:43,403 INFO - LogWindows: Error: [CookWorker 1]: 13:08:43,403 INFO - LogWindows: Error: [CookWorker 1]: Assertion failed: EntryIndexPtr [File:D:\build\streams\main\game\Engine\Source\Developer\IoStoreUtilities\Private\ZenStoreWriter.cpp] [Line: 1063] 13:08:43,403 INFO - LogWindows: Error: [CookWorker 1]: 13:08:43,403 INFO - LogWindows: Error: [CookWorker 1]: 13:08:43,403 INFO - LogWindows: Error: [CookWorker 1]: 13:08:43,403 INFO - LogWindows: Error: [CookWorker 1]: [Callstack] 0x00007ffa6212dddd UnrealEditor-IoStoreUtilities.dll!FZenStoreWriter::CommitPackageInternal() [D:\build\streams\main\game\Engine\Source\Developer\IoStoreUtilities\Private\ZenStoreWriter.cpp:1063] 13:08:43,403 INFO - LogWindows: Error: [CookWorker 1]: [Callstack] 0x00007ffa62108c61 UnrealEditor-IoStoreUtilities.dll!FZenStoreWriter::BeginCook’::33'::<lambda_1>::operator()() [D:\build\streams\main\game\Engine\Source\Developer\IoStoreUtilities\Private\ZenStoreWriter.cpp:656] 13:08:43,403 INFO - LogWindows: Error: [CookWorker 1]: [Callstack] 0x00007ffa621a7bfc UnrealEditor-IoStoreUtilities.dll!TAsyncRunnable<void>::Run() [D:\build\streams\main\game\Engine\Source\Runtime\Core\Public\Async\Async.h:457] 13:08:43,403 INFO - LogWindows: Error: [CookWorker 1]: [Callstack] 0x00007ffa741953bd UnrealEditor-Core.dll!FRunnableThreadWin::Run() [D:\build\streams\main\game\Engine\Source\Runtime\Core\Private\Windows\WindowsRunnableThread.cpp:159] 13:08:43,403 INFO - LogWindows: Error: [CookWorker 1]: [Callstack] 0x00007ffa74186e23 UnrealEditor-Core.dll!FRunnableThreadWin::GuardedRun() [D:\build\streams\main\game\Engine\Source\Runtime\Core\Private\Windows\WindowsRunnableThread.cpp:79] 13:08:43,403 INFO - LogWindows: Error: [CookWorker 1]: [Callstack] 0x00007ffaa888e8d7 KERNEL32.DLL!UnknownFunction [] 13:08:43,403 INFO - LogWindows: Error: [CookWorker 1]: 13:08:43,403 INFO - LogWindows: Error: [CookWorker 1]: Crash in runnable thread TAsyncThread 0 13:08:43,403 INFO - LogWindows: Error: [CookWorker 1]: end: stack for UAT 13:08:47,761 INFO - LogCook: Warning: CookWorkerCrash: CookWorker 1 failed to read from socket with description: Connection terminated. we will shutdown the remote process. Assigned packages will be returned to the director.