Hey
We’re having some trouble with our CI pipelines with Unreal Engine source builds
The cook command gets stuck seemingly at a derived data maintenance step (Though that could be a red herring)
Our builds usually take ~5-10 minutes, but these get stuck for 2 hours and hit our CI build timeout
The end of our log on unsuccessful builds looks like this:
LogInit: Display: Success - 0 error(s), 5924 warning(s)
LogInit: Display:
Execution of commandlet took: 20.48 seconds
LogCore: Engine exit requested (reason: EngineExit() was called; note: exit was already requested)
LogStaticMesh: Abandoning remaining async distance field tasks for shutdown
LogStaticMesh: Abandoning remaining async card representation tasks for shutdown
LogDerivedDataCache: ../../../Engine/DerivedDataCache: Maintenance finished in +00:00:00.000 and deleted 0 files with total size 0 MiB and 0 empty folders. Scanned 0 files in 1 folders with total size 0 MiB.
LogDerivedDataCache: ../../../Engine/DerivedDataCache: Maintenance finished in +00:00:00.000 and deleted 0 files with total size 0 MiB and 0 empty folders. Scanned 0 files in 1 folders with total size 0 MiB.
Our successful builds don’t have any DerivedDataCache maintenance logs, and instead go on to shutdown subsystems
Our CI build command line invocation is:
RunUAT.bat -BuildMachine -NoP4 BuildCookRun -VeryVerbose -Verbose -unattended -distribution -project=‘C:\agent\_work\18\s/Project/[redacted].uproject’ -archivedirectory=‘C:\agent\_work\18\b’ -targetplatform=Android -servertargetplatform=Android -client -configuration=Test -ubtargs=“-NoUBA -NoUBALocal -UBAResetCas -zenstore -cookincremental -UBALog -UBAPrintSummary -UBADetailedLog -UBADetailedTrace -UBAStrict -UBAForcedRetry -UBAStoreCapacityGb=100 -UBAHost=‘0.0.0.0’ -UBAForceRemote -UBAForcedRetryRemote -UBAHordePool=‘Win-UE5’ -UBAHorde=‘[redacted]’ -CookFlavor=Multi -cmdline=”-tracetailmb=20 -trace=Default" -build -cook -stage -pak -package -archive
Any ideas on why our cook step might be freezing - it does seem to occur more commonly on machines which have previously built development build config (Most of our builds are in Test config)