Crash when using multi-core cooking

We’ve recently been trying out multi-core cooking but we’re hitting a crash intermittently. We’re running with 4 cores on a machine that has plenty more cores and plenty of RAM. We had to limit the number of cores because the page file would OOM if we used too many, which isn’t surprising. The problem is that we are intermittently getting a crash that does not appear to be memory related at all.

08:46:50   LogWindows: Error: === Critical error: ===
08:46:50   LogWindows: Error:
08:46:50   LogWindows: Error: Assertion failed: PackageData.IsInProgress() [File:D:\BuildAgent\work\953787b8ae8870bc\UE\Engine\Source\Editor\UnrealEd\Private\CookOnTheFlyServer.cpp] [Line: 2831]
08:46:50   LogWindows: Error:
08:46:50   LogWindows: Error:
08:46:50   LogWindows: Error:
08:46:50   LogWindows: Error: [Callstack] 0x00007ff86058dc76 ***Editor-UnrealEd.dll!UCookOnTheFlyServer::PromoteToSaveComplete() [D:\BuildAgent\work\953787b8ae8870bc\UE\Engine\Source\Editor\UnrealEd\Private\CookOnTheFlyServer.cpp:2831]
08:46:50   LogWindows: Error: [Callstack] 0x00007ff8604b1cef ***Editor-UnrealEd.dll!UE::Cook::FCookWorkerServer::RecordResults() [D:\BuildAgent\work\953787b8ae8870bc\UE\Engine\Source\Editor\UnrealEd\Private\Cooker\CookWorkerServer.cpp:1003]
08:46:50   LogWindows: Error: [Callstack] 0x00007ff8604a28e5 ***Editor-UnrealEd.dll!UE::Cook::FCookWorkerServer::HandleReceiveMessagesInternal() [D:\BuildAgent\work\953787b8ae8870bc\UE\Engine\Source\Editor\UnrealEd\Private\Cooker\CookWorkerServer.cpp:794]
08:46:50   LogWindows: Error: [Callstack] 0x00007ff8604a2659 ***Editor-UnrealEd.dll!UE::Cook::FCookWorkerServer::HandleReceiveMessages() [D:\BuildAgent\work\953787b8ae8870bc\UE\Engine\Source\Editor\UnrealEd\Private\Cooker\CookWorkerServer.cpp:753]
08:46:50   LogWindows: Error: [Callstack] 0x00007ff86044aa06 ***Editor-UnrealEd.dll!UE::Cook::FCookDirector::TickFromSchedulerThread() [D:\BuildAgent\work\953787b8ae8870bc\UE\Engine\Source\Editor\UnrealEd\Private\Cooker\CookDirector.cpp:750]
08:46:50   LogWindows: Error: [Callstack] 0x00007ff86058ecea ***Editor-UnrealEd.dll!UCookOnTheFlyServer::PumpPollables() [D:\BuildAgent\work\953787b8ae8870bc\UE\Engine\Source\Editor\UnrealEd\Private\CookOnTheFlyServer.cpp:2058]
08:46:50   LogWindows: Error: [Callstack] 0x00007ff8605b20bc ***Editor-UnrealEd.dll!UCookOnTheFlyServer::TickMainCookLoop() [D:\BuildAgent\work\953787b8ae8870bc\UE\Engine\Source\Editor\UnrealEd\Private\CookOnTheFlyServer.cpp:1558]
08:46:50   LogWindows: Error: [Callstack] 0x00007ff8605b17fc ***Editor-UnrealEd.dll!UCookOnTheFlyServer::TickCookByTheBook() [D:\BuildAgent\work\953787b8ae8870bc\UE\Engine\Source\Editor\UnrealEd\Private\CookOnTheFlyServer.cpp:1398]
08:46:50   LogWindows: Error: [Callstack] 0x00007ff8602321d0 ***Editor-UnrealEd.dll!UCookCommandlet::RunCookByTheBookCook() [D:\BuildAgent\work\953787b8ae8870bc\UE\Engine\Source\Editor\UnrealEd\Private\Commandlets\CookCommandlet.cpp:631]
08:46:50   LogWindows: Error: [Callstack] 0x00007ff8601f390a ***Editor-UnrealEd.dll!UCookCommandlet::CookByTheBook() [D:\BuildAgent\work\953787b8ae8870bc\UE\Engine\Source\Editor\UnrealEd\Private\Commandlets\CookCommandlet.cpp:584]
08:46:50   LogWindows: Error: [Callstack] 0x00007ff860213e13 ***Editor-UnrealEd.dll!UCookCommandlet::Main() [D:\BuildAgent\work\953787b8ae8870bc\UE\Engine\Source\Editor\UnrealEd\Private\Commandlets\CookCommandlet.cpp:267]
08:46:50   LogWindows: Error: [Callstack] 0x00007ff6a993eb4e ***Editor-Cmd.exe!FEngineLoop::PreInitPostStartupScreen() [D:\BuildAgent\work\953787b8ae8870bc\UE\Engine\Source\Runtime\Launch\Private\LaunchEngineLoop.cpp:4154]
08:46:50   LogWindows: Error: [Callstack] 0x00007ff6a9935566 ***Editor-Cmd.exe!GuardedMain() [D:\BuildAgent\work\953787b8ae8870bc\UE\Engine\Source\Runtime\Launch\Private\Launch.cpp:144]
08:46:50   LogWindows: Error: [Callstack] 0x00007ff6a993589a ***Editor-Cmd.exe!GuardedMainWrapper() [D:\BuildAgent\work\953787b8ae8870bc\UE\Engine\Source\Runtime\Launch\Private\Windows\LaunchWindows.cpp:123]
08:46:50   LogWindows: Error: [Callstack] 0x00007ff6a9939114 ***Editor-Cmd.exe!LaunchWindowsStartup() [D:\BuildAgent\work\953787b8ae8870bc\UE\Engine\Source\Runtime\Launch\Private\Windows\LaunchWindows.cpp:277]
08:46:50   LogWindows: Error: [Callstack] 0x00007ff6a994bd04 ***Editor-Cmd.exe!WinMain() [D:\BuildAgent\work\953787b8ae8870bc\UE\Engine\Source\Runtime\Launch\Private\Windows\LaunchWindows.cpp:317]
08:46:50   LogWindows: Error: [Callstack] 0x00007ff6a994e422 ***Editor-Cmd.exe!__scrt_common_main_seh() [D:\a\_work\1\s\src\vctools\crt\vcstartup\src\startup\exe_common.inl:288]
08:46:50   LogWindows: Error: [Callstack] 0x00007ff8a9984cb0 KERNEL32.DLL!UnknownFunction []

Modified the call stack to obfuscate the binary.

Any idea why this might happen or how to fix/prevent it? We’re on 5.5.4.