Hi Epic. Since 5.7, we are seeing some cooking failures that we can’t explain:
“LogCook: Error: Content is missing from cook. Source package referenced an object in target package but the target package was marked NeverCook or is not cookable for the target platform.”
The conditions for this are strange. One of our CI machines (let’s call it CIx) seems to repro it every time (some asterisks, see below), and we’ve seen it once on another CI machine’s workspace (that we likely haven’t used since). No other machine has reproed it!
My colleague has managed to link the issue to \{game}\Intermediate\CachedAssetRegistry*.bin containing bad/incomplete data somehow. As if new asset dependencies simply fail to be written into the CachedAssetRegistry.
Here is a timeline / list of data points:
- We didn’t have this with 5.6.
- The target configuration doesn’t matter. We’ve tried with Development, Test and Shipping. CIx reproed the problem with all of them.
- The target platform doesn’t seem to matter either. We’ve tried with two console platforms, and Win64. CIx reproed the problem with all of them.
- No dev machine has ever reproed this, as far as we know. 99% of devs just use PIE/CookOnTheFly, but I ran a few BuildCookRuns locally on my machine, and could not repro this (0/2 I think).
- CIx has 100% repro!
- Other CI machines are happily cooking for all platforms and configurations…
- We tried a ‘p4 clean’ on CIx, which found no corruption. The issue reproed again soon afterwards.
- We don’t suspect hardware issues on CIx, because of the very specific nature of the error, and the consistency of it. Also, see below.
- After some debugging, where we tracked it to the CachedAssetRegistry containing the wrong data, we cleared those files and we tried again. The issue did NOT repro then!
- We then noticed, from looking at the history of logs, that the only assets that gave us the NeverCook error had been created or updated since the p4-clean. Of the 90+ errors we got, there were no patterns in the types of assets affected, except the fact that they were all recently updated. This implies that, after the p4 clean, CIx probably managed one successful cook, so the CachedAssetRegistry got recreated (I haven’t looked for evidence of this), but then, at the next perforce sync, all newly updated assets (at least in terms of them gaining new downstream dependencies (aka “imports”?)) started erroring again during the cook. We have since disconnected CIx from the CI system, and we haven’t done a new p4 sync. As a result, the same 90+ assets were consistently giving us the NeverCook error. Until we deleted the CachedAssetRegistry files (per data point #9).
- With the freshly recreated CachedAssetRegistry, I tried syncing one of the changed assets back to before the p4 clean, and the issue didn’t repro.
- With the freshly recreated CachedAssetRegistry, I then synced the same asset back to the revision that was erroring, and I even edited it to reference some other downstream asset. There was no repro with this either!
- Finally, I restored the “broken” CachedAssetRegistry files, and the issue is now reproing again, with a 100% rate.
I could summarize it all by saying that we have had 3 incidents of CachedAssetRegistry corruption, on 2 machines, since switching to 5.7. The only thing that fixes it is to clear the CachedAssetRegistry*.bin files, but then it can happen again somehow.
Can you give us a summary of when the CachedAssetRegistry*.bin files are updated? Is it when UE.exe starts? Or only when BuildCookRun / CookByTheBook happens?
Any ideas why CachedAssetRegistry*.bin might stop being updated?
We are contemplating scripting the deletion of \{game}\Intermediate\CachedAssetRegistry*.bin before each CI cook, but, will this have an impact on editor startup/cooking times? Also, since we don’t fully understand the problem, this may not solve it anyway. Plus, we’ll be stuck with this hack forever, since we won’t know when the issue might have been fixed. Ideally, we’d get to the bottom of the problem instead.
Thanks in advance for your help.
Kind regards,
Kostas V.
PS Still not a fan of the char limit here!
[Attachment Removed]