Excessively long waiting time for discovering asset data

Development Pipeline & Plugins
, ,
1

Typically, this process won’t take too much time. But our project is nearly 20TB in size, and the essential files distributed to each artist’s computer are also around 8TB.

​Once asset data errors or crashes occured, they will be forced to clear the asset data by deleting intermediate folder. Then Too much assets data need to be found again, and it may take several hours.

Most artists use HDD storage; otherwise, they might not have enough storage space.

[Image Removed]

[Image Removed]

[Attachment Removed]

2

Steps to Reproduce
Remove the intermediate folder, and then restart the project. Discovering Asset Data will be started.

[Image Removed]

[Image Removed]

[Attachment Removed]

3

1. We have made improvements to the performance of parsing the AssetRegistry when there is no cache in 5.7; can you update to 5.7? If even that performance is too slow I don’t think there’s much we can do to improve it further.

2. Is the assertion you reported above:

Assertion failed Algo::IsSorted(SerializedChunkIDs)

the only reason that the artists need to delete the Intermediate directory?

We have not seen that assertion fail internally, except in one case of a corrupt workspace, so I don’t have any ideas for what could cause it. Can you add instrumentation to your copy of the engine, to see if it occurs during save?

//UE5/Release-5.6/Engine/Source/Runtime/CoreUObject/Private/AssetRegistry/AssetData.cpp

Line 695:
void FAssetData::SerializeForCacheInternal(FArchive& Ar, FAssetRegistryVersion::Type Version, void (*SerializeTagsAndBundles)(FArchive& , FAssetData&, FAssetRegistryVersion::Type))
{
...
Line 763:
	FAssetData::FChunkArray SerializedChunkIDs;
	if (Ar.IsSaving())
	{
		SerializedChunkIDs = GetChunkIDs();
// NEW CODE START
		if (!Algo::IsSorted(SerializedChunkIDs))
		{
			TStringBuilder<256> ChunkIDText;
			ChunkIDText << TEXT("[ ");
			for (int32 ChunkID : SerializedChunkIDs)
			{
				ChunkIDText << ChunkID << ", ";
			}
			ChunkIDText.RemoveSuffix(1);
			ChunkIDText << TEXT(" ]");
			UE_LOG(LogAssetData, Error, TEXT("ChunkIds are not sorted for AssetData %s.%s: %s"),
				*PackageName.ToString(), *AssetName.ToString(), *ChunkIDText);
		}
// NEW CODE END
		Ar << SerializedChunkIDs;
	}
	else
	{
		Ar << SerializedChunkIDs;
		check(Algo::IsSorted(SerializedChunkIDs));
		ChunkArrayRegistryHandle = UE::AssetRegistry::GChunkArrayRegistry.FindOrAddSorted(MoveTemp(SerializedChunkIDs));
	}
...

void FAssetData::SerializeForCacheInternal(FArchive& Ar, FAssetRegistryVersion::Type Version, void (*SerializeTagsAndBundles)(FArchive& , FAssetData&, FAssetRegistryVersion::Type))

{

[Attachment Removed]

4

That error is a different error than the first one you reported. It looks like an error that would occur if the AssetRegistry’s cache file is corrupt.

The cache files are checksummed, so corruption after it was written or an interrupted write is unlikely. It is more likely that we are somehow writing out an incorrectly formatted cache file due to a bug in the writing code.

Can you run gather some data and report it? Launch the editor normally and wait for the AssetRegistry to finish scanning and write the files out? How many CachedAssetRegistry_*.bin files are present in your Intermediate directory, and what is the size of the smallest and largest of them?

[Attachment Removed]

5

That’s not up against a 2GB boundary, nothing suspicious with the normal sizes.

Larger sizes in the problem case is suspicious, but I can’t think what would cause it.

I’ll research the cache writing code and get back to you with some instrumentation to preport (or temporarily add, if not submittable) that will confirm or rebut the hypothesis that something is going wrong during write.

[Attachment Removed]

6

I haven’t found any leads yet, and while stepping through the code looking for leads I have learned that having a copy of the bad .bin file would be useful. Can you send me a link to one of them? I don’t think you can attach them here because they are over the size limit for attachments.

[Attachment Removed]

7

Thanks, I have downloaded CachedAssetRegistry_0 through 4. Please remove sharing for the folder now, to prevent other accessors.

[Attachment Removed]

8

CachedAssetRegistry_3 has the problem. The other files do not crash.

AssetRegistry gatherer cache files are written in blocks of 2^22 bytes. Each block has a magicnumber signature and a checksum.

All of the blocks in CachedAssetRegistry_3 have a valid signature and checksum.

The final block of CachedAssetRegistry_3 is the wrong block. It has a valid signature and checksum, so it is a valid block, but it is not the block that is supposed to follow the block before it. The bytes in it pick up at a different location in the stream of assets than expected, and it has around 1000 more assets in it than it is supposed to.

I have not yet found any leads for how the final expected block of CachedAssetRegistry_3 could have been replaced with this wrong block. I am still looking.

Some possible leads that might exist outside of the code I’m looking at:

What minor version of 5.6 are you on? 5.6.0, 5.6.1? Some other release branch of UE5 that is around the time of 5.6?

Do you have any local changes to source code in Engine\Source\Runtime\AssetRegistry ?

What operating system are your editors running? Windows, Linux, Mac?

Does this problem occur to most artists or only a small fraction of them?

Are all of the artists running an UnrealEditor.exe built on the same version of 5.6?

[Attachment Removed]

9

1.We’re using 5.6.1 version from github​.

2.Nothing’s changed in Engine\Source\Runtime\AssetRegistry.

3.Windows (both windows10 and 11)

4.It’s difficult to count accurately, at least about 10% of artists have faced this problem.

5.Yes,they’re using the same version

[Attachment Removed]

10

I have one guess so far for a bug that’s causing the issue: writing to the same temp file from two save attempts in the editor when creating the cache file in FAssetDataGatherer::SaveCacheFileInternal occurs and corrupts the file. I’ll submit a change to make the temp files unique and send that to you to try out.

Other possibilities are memory corruption or disk corruption. There was a case a few years back that we found with some intel CPUs that caused rare and random memory corruption. Can you investigate those possibilities on one of the machines that have observed the problem?

  • What is the machine hardware, from Windows’ System Information?
    • Windows Key -> System Information, the `Processor` field in the System Summary page.
  • Launch a memory test
    • Windows Key -> Windows Memory Diagnostic
  • Launch a hard disk test
    • Windows Key -> cmd.exe (Run as Administrator)
      • chkdsk <DriveLetterContainingTheIntermediateDirectory>: /F
        • For example: chkdisk D: /F
      • It’ll ask you if you want to dismount the volume, press N and hit enter.
      • It’ll ask if you want to schedule it for next reboot, press Y and hit enter.
      • Reboot, and it’ll start the scan which may take a while

Also, can you confirm that the Intermediate directories are on a local disk rather than a network drive?

If the bugfix I send you for the temp filepath does not work, and the tests I asked you to run do not provide any leads, then my current plan is to create another change for you in which we modify the file format to include a checksum of the expected blocks in the file, and a validate function that is run immediately after writing the file and whenever we try to read it. That should reduce the cost of the failure (only 1/4 of the cache will be removed) and will also tell us whether the problem occurs during write or sometime later.

[Attachment Removed]

11

Please try this code locally and let me know if it prevents further occurrences of the corruption.

Note about the code: In Oct 2025 on 5.7 we encountered on our internal projects the possibility that multiple UnrealEditor.exe would try to save the cache at the same time (e.g. Editor + Commandlet launched from editor). We added A system-wide mutex around the cache file write. If the mutex cannot be entered, we try again later. This was added in these two CLs:

The code used in those changes to retry later if the mutex could not be entered has some complex dependencies on other changes to the AssetDataGatherer in 5.7 and 5.8, so I did not include it. I instead copied just the use of the system-wide mutex. If it cannot be entered, we sleep for 10 seconds, then give an error and abandon the cache write. This will be less robust because it will skip rather than retry, but it will only do that in cases of two editors trying to write at once, which should be infrequent. If the artists are having slow loads, they should look for that error in the log to see if the mutex is repeatedly failing:

LogAssetRegistry: Error: Timed out trying to write AssetRegistryCacheFiles to %s; another process holds the system-wide mutex \"%s\"

The new code:

Engine\Source\Runtime\AssetRegistry\Private\AssetDataGatherer.cpp:20

#include "HAL/PlatformMutex.h"

Engine\Source\Runtime\AssetRegistry\Private\AssetDataGatherer.cpp:4067

// Line 4015
void FAssetDataGatherer::SaveCacheFile(const TArray<TPair<FName,FDiskCachedAssetData*>>& AssetsToSave)
{
	using namespace UE::AssetDataGather::Private;
	TRACE_CPUPROFILER_EVENT_SCOPE_STR("Save Cache")
...
// Line 4066
			uint64 Hash = CityHash64((const char*)Buffer.GetData(), Buffer.Len() * sizeof(TCHAR));
			int32 Index = (Hash & ShardMask);
			DataPerShard[Index].Add(Entry);
		}
	}
 
// NEW CODE START
	// Attempt to get exclusive write access to the cache files. If we cannot, sleep until we can.
	// Failing to get the lock here means another process is currently writing. We need to avoid trying to write at the same time to avoid the possiblity of writing corrupted files.
#define CACHEFILE_SYSTEMWIDE_WRITELOCK WITH_EDITOR
#if CACHEFILE_SYSTEMWIDE_WRITELOCK
	FString MutexFilename;
	{
		check(GGatherSettings.IsInitialized());
		TStringBuilder<256> Builder;
		Builder.Append(*FPaths::Combine(*GGatherSettings.GetAssetRegistryCacheRootFolder(), TEXT("AssetRegistryCacheLock")));
		MutexFilename = Builder.ToString();
	}
	TOptional<UE::FPlatformSystemWideMutex> CacheFilesLock;
	CacheFilesLock.Emplace(MutexFilename);
	if (!CacheFilesLock->IsValid())
	{
		CacheFilesLock.Reset();
		double StartTime = FPlatformTime::Seconds();
		float MaxDurationSeconds = 10.f;
		float DisplayDuration = 2.f;
		float SleepTimeSeconds = 0.25f;
		double NextDisplayTime = StartTime;
		for (;;)
		{
			CurrentTime = FPlatformTime::Seconds();
			float Remaining = static_cast<float>(StartTime + MaxDurationSeconds - CurrentTime);
			if (Remaining < 0.f)
			{
				UE_LOG(LogAssetRegistry, Error,
					TEXT("Timed out trying to write AssetRegistryCacheFiles to %s; another process holds the system-wide mutex \"%s\"."),
					*GGatherSettings.GetCacheBaseFilename(), *MutexFilename);
				return;
			}
			if (CurrentTime >= NextDisplayTime)
			{
				NextDisplayTime = CurrentTime + DisplayDuration;
				UE_LOG(LogAssetRegistry, Display,
					TEXT("Trying to write AssetRegistryCacheFiles to %s, but another process holds the system-wide mutex \"%s\". Sleeping for %.0f more seconds before giving up."),
					*GGatherSettings.GetCacheBaseFilename(), *MutexFilename, Remaining);
			}
			FPlatformProcess::Sleep(SleepTimeSeconds);
			CacheFilesLock.Emplace(MutexFilename);
			if (CacheFilesLock->IsValid())
			{
				break;
			}
		}
		return;
	}
#endif
// NEW CODE END
 
	ParallelFor(CacheShards, [this, &AssetsToSave, &TotalCacheSize, &DataPerShard, ShardMask](int32 Shard) 
	{
		TArray<TPair<FName, FDiskCachedAssetData*>>& ShardData = DataPerShard[Shard];
...

[Attachment Removed]

12

Yes, I expect it to work in 5.6 with no changes to the code I posted above.

[Attachment Removed]

13

Yes, that will work for testing, you do not need to find a machine where a cache error occurs.

This change is a fix attempt for the occurrence of the corruption; if it works then the corrupted files will stop appearing.

After submitting the change, you should delete all current occurrences of the corrupted files from the intermediate directory on artist’s machine, and wait to see if new instances of corrupted files occur.

[Attachment Removed]

14

Thank you, But we are unable to stably reproduce this error.

An artist is experiencing an AssetRegistry error at present. Attached is the error log, and this error has been occurring more frequently over the years.The codes you provided to me earlier didn’t print out in the log this time. [Image Removed]

[Attachment Removed]

15

Here is the error in log file

2026.01.09-10.57.57:222][ 0]LogInit: Computer: ZHANGYUFENG

[2026.01.09-10.57.57:222][ 0]LogInit: User: zhangyufeng

[2026.01.09-10.57.57:222][ 0]LogInit: CPU Page size=4096, Cores=10

[2026.01.09-10.57.57:222][ 0]LogInit: High frequency timer resolution =10.000000 MHz

[2026.01.09-10.57.57:222][ 0]LogMemory: Process is running as part of a Windows Job with separate resource limits

[2026.01.09-10.57.57:222][ 0]LogMemory: Memory total: Physical=127.9GB (128GB approx) Virtual=149.1GB

[2026.01.09-10.57.57:222][ 0]LogMemory: Platform Memory Stats for WindowsEditor

[2026.01.09-10.57.57:222][ 0]LogMemory: Process Physical Memory: 4404.75 MB used, 4404.75 MB peak

[2026.01.09-10.57.57:222][ 0]LogMemory: Process Virtual Memory: 2909.79 MB used, 2909.79 MB peak

[2026.01.09-10.57.57:222][ 0]LogMemory: Physical Memory: 34618.45 MB used, 96317.14 MB free, 130935.58 MB total

[2026.01.09-10.57.57:222][ 0]LogMemory: Virtual Memory: 35699.34 MB used, 116998.22 MB free, 152697.56 MB total

[2026.01.09-10.57.57:222][ 0]LogCsvProfiler: Display: Metadata set : extradevelopmentmemorymb=“0”

[2026.01.09-10.57.57:222][ 0]LogWindows: WindowsPlatformFeatures enabled

[2026.01.09-10.57.57:222][ 0]LogChaosDD: Chaos Debug Draw Startup

[2026.01.09-10.57.57:222][ 0]LogInit: Physics initialised using underlying interface: Chaos

[2026.01.09-10.57.57:222][ 0]LogInit: Using OS detected language (zh-CN).

[2026.01.09-10.57.57:222][ 0]LogInit: Using OS detected locale (zh-CN).

[2026.01.09-10.57.57:222][ 0]LogTextLocalizationManager: No specific localization for ‘zh-CN’ exists, so ‘zh-Hans’ will be used for the language.

[2026.01.09-10.57.57:222][ 0]LogInit: Setting process to per monitor DPI aware

[2026.01.09-10.57.57:222][ 0]LogThreadingWindows: Error: Runnable thread Background Worker #6 crashed.

[2026.01.09-10.57.57:222][ 0]LogWindows: Error: === Critical error: ===

[2026.01.09-10.57.57:222][ 0]LogWindows: Error:

[2026.01.09-10.57.57:222][ 0]LogWindows: Error: Assertion failed: (Index >= 0) & (Index < ArrayNum) [File:D:\GitRepo\UnrealEngine\Engine\Source\Runtime\Core\Public\Containers\Array.h] [Line: 1067]

[2026.01.09-10.57.57:222][ 0]LogWindows: Error: Array index out of bounds: 1132006277 into an array of size 496403

[2026.01.09-10.57.57:222][ 0]LogWindows: Error:

[2026.01.09-10.57.57:222][ 0]LogWindows: Error:

[2026.01.09-10.57.57:222][ 0]LogWindows: Error: [Callstack] 0x00007ffdf2f14028 UnrealEditor-Core.dll!UnknownFunction []

[2026.01.09-10.57.57:222][ 0]LogWindows: Error: [Callstack] 0x00007ffe5abc8423 UnrealEditor-AssetRegistry.dll!UnknownFunction []

[2026.01.09-10.57.57:222][ 0]LogWindows: Error: [Callstack] 0x00007ffe5aaf0ee8 UnrealEditor-AssetRegistry.dll!UnknownFunction []

[2026.01.09-10.57.57:222][ 0]LogWindows: Error: [Callstack] 0x00007ffe5ab6c00e UnrealEditor-AssetRegistry.dll!UnknownFunction []

[2026.01.09-10.57.57:222][ 0]LogWindows: Error: [Callstack] 0x00007ffe5ab081aa UnrealEditor-AssetRegistry.dll!UnknownFunction []

[2026.01.09-10.57.57:222][ 0]LogWindows: Error: [Callstack] 0x00007ffe5ab0a811 UnrealEditor-AssetRegistry.dll!UnknownFunction []

[2026.01.09-10.57.57:222][ 0]LogWindows: Error: [Callstack] 0x00007ffe5ab0e294 UnrealEditor-AssetRegistry.dll!UnknownFunction []

[2026.01.09-10.57.57:222][ 0]LogWindows: Error: [Callstack] 0x00007ffe5ab1daa1 UnrealEditor-AssetRegistry.dll!UnknownFunction []

[2026.01.09-10.57.57:222][ 0]LogWindows: Error: [Callstack] 0x00007ffdf2bddc55 UnrealEditor-Core.dll!UnknownFunction []

[2026.01.09-10.57.57:222][ 0]LogWindows: Error: [Callstack] 0x00007ffdf2bddb2e UnrealEditor-Core.dll!UnknownFunction []

[2026.01.09-10.57.57:222][ 0]LogWindows: Error: [Callstack] 0x00007ffdf2c03db1 UnrealEditor-Core.dll!UnknownFunction []

[2026.01.09-10.57.57:222][ 0]LogWindows: Error: [Callstack] 0x00007ffdf2bc5fe5 UnrealEditor-Core.dll!UnknownFunction []

[2026.01.09-10.57.57:222][ 0]LogWindows: Error: [Callstack] 0x00007ffdf2de1a33 UnrealEditor-Core.dll!UnknownFunction []

[2026.01.09-10.57.57:222][ 0]LogWindows: Error: [Callstack] 0x00007ffdf3283b3d UnrealEditor-Core.dll!UnknownFunction []

[2026.01.09-10.57.57:222][ 0]LogWindows: Error: [Callstack] 0x00007ffdf327cc8f UnrealEditor-Core.dll!UnknownFunction []

[2026.01.09-10.57.57:222][ 0]LogWindows: Error: [Callstack] 0x00007ffe91f6e8d7 KERNEL32.DLL!UnknownFunction []

[2026.01.09-10.57.57:222][ 0]LogWindows: Error:

[2026.01.09-10.57.57:222][ 0]LogWindows: Error: Crash in runnable thread Background Worker #6

[2026.01.09-10.57.57:230][ 0]LogExit: Executing StaticShutdownAfterError

[2026.01.09-10.57.57:230][ 0]LogWindows: FPlatformMisc::RequestExit(1, FRunnableThreadWin::GuardedRun.ExceptionHandler)

[2026.01.09-10.57.57:230][ 0]LogWindows: FPlatformMisc::RequestExitWithStatus(1, 3, FRunnableThreadWin::GuardedRun.ExceptionHandler)

[2026.01.09-10.57.57:230][ 0]LogCore: Engine exit requested (reason: Win RequestExit)

[Attachment Removed]

16

There’re 4 CachedAssetRegistry_*.bin files in Intermediate folder of the problematic computer. I cannot launch the project unless these files are deleted.

Size of the largest file: 432MB;

Smallest: 355MB

[Image Removed]And I also examined this directory using a machine where the project can be started without any issues.

There’re also 4 files:

[Image Removed]The Largest: 277MB

The Smallest: 255MB

[Attachment Removed]

17

Thank you so much. If it would be helpful, I can provide these problematic .bin files to you.

[Attachment Removed]

18

Here is the google drive link of full intermediate folder:

https://drive.google.com/drive/folders/1q162ihXYt0WNIjWUG_ITJ4A5LF5Kk8nB?usp=sharing

[Attachment Removed]

19

Sorry for the delay – I’ve been occupied with other matters over the past few days. We have completed the hardware testing, and everything is functioning normally. Is the code modification you mentioned above compatible with version 5.6?

[Attachment Removed]

20

I declared the variable

bool IsInitialized() const;

at line 899 of the .h file, and the compilation passed in 5.6.1.

There’s another question:This situation is not easy to reproduce. Assuming I directly delete the intermediate files and let it search for the AssetRegistry again, would that also work for testing? Or I need to find another machine where a cache error occurs

[Attachment Removed]