"Error: Content is missing from cook. ... marked NeverCook..."

Hi Epic. Since 5.7, we are seeing some cooking failures that we can’t explain:

“LogCook: Error: Content is missing from cook. Source package referenced an object in target package but the target package was marked NeverCook or is not cookable for the target platform.”

The conditions for this are strange. One of our CI machines (let’s call it CIx) seems to repro it every time (some asterisks, see below), and we’ve seen it once on another CI machine’s workspace (that we likely haven’t used since). No other machine has reproed it!

My colleague has managed to link the issue to \{game}\Intermediate\CachedAssetRegistry*.bin containing bad/incomplete data somehow. As if new asset dependencies simply fail to be written into the CachedAssetRegistry.

Here is a timeline / list of data points:

  1. We didn’t have this with 5.6.
  2. The target configuration doesn’t matter. We’ve tried with Development, Test and Shipping. CIx reproed the problem with all of them.
  3. The target platform doesn’t seem to matter either. We’ve tried with two console platforms, and Win64. CIx reproed the problem with all of them.
  4. No dev machine has ever reproed this, as far as we know. 99% of devs just use PIE/CookOnTheFly, but I ran a few BuildCookRuns locally on my machine, and could not repro this (0/2 I think).
  5. CIx has 100% repro!
  6. Other CI machines are happily cooking for all platforms and configurations…
  7. We tried a ‘p4 clean’ on CIx, which found no corruption. The issue reproed again soon afterwards.
  8. We don’t suspect hardware issues on CIx, because of the very specific nature of the error, and the consistency of it. Also, see below.
  9. After some debugging, where we tracked it to the CachedAssetRegistry containing the wrong data, we cleared those files and we tried again. The issue did NOT repro then!
  10. We then noticed, from looking at the history of logs, that the only assets that gave us the NeverCook error had been created or updated since the p4-clean. Of the 90+ errors we got, there were no patterns in the types of assets affected, except the fact that they were all recently updated. This implies that, after the p4 clean, CIx probably managed one successful cook, so the CachedAssetRegistry got recreated (I haven’t looked for evidence of this), but then, at the next perforce sync, all newly updated assets (at least in terms of them gaining new downstream dependencies (aka “imports”?)) started erroring again during the cook. We have since disconnected CIx from the CI system, and we haven’t done a new p4 sync. As a result, the same 90+ assets were consistently giving us the NeverCook error. Until we deleted the CachedAssetRegistry files (per data point #9).
  11. With the freshly recreated CachedAssetRegistry, I tried syncing one of the changed assets back to before the p4 clean, and the issue didn’t repro.
  12. With the freshly recreated CachedAssetRegistry, I then synced the same asset back to the revision that was erroring, and I even edited it to reference some other downstream asset. There was no repro with this either!
  13. Finally, I restored the “broken” CachedAssetRegistry files, and the issue is now reproing again, with a 100% rate.

I could summarize it all by saying that we have had 3 incidents of CachedAssetRegistry corruption, on 2 machines, since switching to 5.7. The only thing that fixes it is to clear the CachedAssetRegistry*.bin files, but then it can happen again somehow.

Can you give us a summary of when the CachedAssetRegistry*.bin files are updated? Is it when UE.exe starts? Or only when BuildCookRun / CookByTheBook happens?

Any ideas why CachedAssetRegistry*.bin might stop being updated?

We are contemplating scripting the deletion of \{game}\Intermediate\CachedAssetRegistry*.bin before each CI cook, but, will this have an impact on editor startup/cooking times? Also, since we don’t fully understand the problem, this may not solve it anyway. Plus, we’ll be stuck with this hack forever, since we won’t know when the issue might have been fixed. Ideally, we’d get to the bottom of the problem instead.

Thanks in advance for your help.

Kind regards,

Kostas V.

PS Still not a fan of the char limit here!

[Attachment Removed]

Steps to Reproduce
It’s a complicated repro… See main post.

[Attachment Removed]

We have seen corruption with the CachedAssetRegistry*.bin files internally on 5.7 and reported by a licensee in 5.6. The corruption seemed to come from multiple UnrealEditor.exe processes, or multiple threads in the same process, attempting to write the files at the same time. The corruption we observed in both cases resulted in the cache being discovered corrupt at load time and discarded, so it does not match your case exactly. But you should try applying the same fix to test whether you’re seeing a different symptom of the same write-collision corruption.

The CachedAssetRegistry*.bin files are created after the AssetRegistry completes its gathering. This happens both when the editor IDE starts and in many commandlets including the cook commandlet.

Please try this code locally and let me know if it prevents further occurrences of the corruption.

Note about the code: In Oct 2025 on 5.7 we encountered on our internal projects the possibility that multiple UnrealEditor.exe would try to save the cache at the same time (e.g. Editor + Commandlet launched from editor). We added A system-wide mutex around the cache file write. If the mutex cannot be entered, we try again later. This was added in these two CLs:

416ad75261e47710bc7533d1f0205c320e7bda90 aka CL 47434330: Writing to the AssetRegistry cache will now attempt to take a system-wide lock…

42e96edc491f0d501bfd39fd0d4de08316911aa2 aka CL 50051915: Avoid possible hang on unix based platforms…

The code used in those changes to retry later if the mutex could not be entered has some complex dependencies on other changes to the AssetDataGatherer in 5.8, so I did not include it. I instead copied just the use of the system-wide mutex. If it cannot be entered, we sleep for 10 seconds, then give an error and abandon the cache write. This will be less robust because it will skip rather than retry, but it will only do that in cases of two editors trying to write at once, which should be infrequent. If you see slow startup in editor or cooker having slow loads, look for this error in the log to see if the mutex is repeatedly failing:

LogAssetRegistry: Error: Timed out trying to write AssetRegistryCacheFiles to %s; another process holds the system-wide mutex \"%s\"

The new code:

Engine\Source\Runtime\AssetRegistry\Private\AssetDataGatherer.cpp:20

#include "HAL/PlatformMutex.h"

Engine\Source\Runtime\AssetRegistry\Private\AssetDataGatherer.cpp:4099

// Line 4099
void FAssetDataGatherer::SaveCacheFile(const TArray<TPair<FName,FDiskCachedAssetData*>>& AssetsToSave)
{
	using namespace UE::AssetDataGather::Private;
	TRACE_CPUPROFILER_EVENT_SCOPE_STR("Save Cache")
...
// Line 4130
			uint64 Hash = CityHash64((const char*)Buffer.GetData(), Buffer.Len() * sizeof(TCHAR));
			int32 Index = (Hash & ShardMask);
			DataPerShard[Index].Add(Entry);
		}
	}
 
 // NEW CODE START
	// Attempt to get exclusive write access to the cache files. If we cannot, sleep until we can.
	// Failing to get the lock here means another process is currently writing. We need to avoid trying to write at the same time to avoid the possiblity of writing corrupted files.
#define CACHEFILE_SYSTEMWIDE_WRITELOCK WITH_EDITOR
#if CACHEFILE_SYSTEMWIDE_WRITELOCK
	FString MutexFilename;
	{
		TStringBuilder<256> Builder;
		Builder.Append(*FPaths::Combine(*GGatherSettings.GetAssetRegistryCacheRootFolder(), TEXT("AssetRegistryCacheLock")));
		MutexFilename = Builder.ToString();
	}
	TOptional<UE::FPlatformSystemWideMutex> CacheFilesLock;
	CacheFilesLock.Emplace(MutexFilename);
	if (!CacheFilesLock->IsValid())
	{
		CacheFilesLock.Reset();
		double StartTime = FPlatformTime::Seconds();
		float MaxDurationSeconds = 10.f;
		float DisplayDuration = 2.f;
		float SleepTimeSeconds = 0.25f;
		double NextDisplayTime = StartTime;
		for (;;)
		{
			double CurrentTime = FPlatformTime::Seconds();
			float Remaining = static_cast<float>(StartTime + MaxDurationSeconds - CurrentTime);
			if (Remaining < 0.f)
			{
				UE_LOG(LogAssetRegistry, Error,
					TEXT("Timed out trying to write AssetRegistryCacheFiles to %s; another process holds the system-wide mutex \"%s\"."),
					*GGatherSettings.GetCacheBaseFilename(), *MutexFilename);
				return;
			}
			if (CurrentTime >= NextDisplayTime)
			{
				NextDisplayTime = CurrentTime + DisplayDuration;
				UE_LOG(LogAssetRegistry, Display,
					TEXT("Trying to write AssetRegistryCacheFiles to %s, but another process holds the system-wide mutex \"%s\". Sleeping for %.0f more seconds before giving up."),
					*GGatherSettings.GetCacheBaseFilename(), *MutexFilename, Remaining);
			}
			FPlatformProcess::Sleep(SleepTimeSeconds);
			CacheFilesLock.Emplace(MutexFilename);
			if (CacheFilesLock->IsValid())
			{
				break;
			}
		}
		return;
	}
#endif
// NEW CODE END
 
 	// If we recently saved or loaded the file then pause for 0.5 seconds before trying to save on
	// top of it, to avoid failure to be able to delete the file we just saved/loaded.
	double LocalLastCacheWriteTime;
...

[Attachment Removed]

Thanks very much for the quick reply! And for the effort to offer a fix compatible with 5.7.

We have submitted this, and we’ll report back if the issue reproes again.

[Attachment Removed]