10+ minutes during cook in FAssetRegistryImpl::SetManageReferences

Hi,

We are seeing 10+ minutes in FAssetRegistryImpl::SetManageReferences during cook or doing certain operations in the editor with some primary assets getting over 50 million entries in their ManageDependencies array (we do not have 50 million assets). This seems related to the change here: https://github.com/EpicGames/UnrealEngine/commit/6225ff72e1268aa286cab8bbda74fd2d49e935f4 and did not occur in UE 5.4.Some FDependsNodes have duplicates in some of their arrays which I suspect is related.

Do you suggestions or ideas to debug this?

Thank you

Steps to Reproduce

  1. Run a game cook

FDependsNodes allow duplicates during startup to avoid spending time on AddUnique, but they are supposed to remove the duplicates when the search finishes; FAssetRegistryImpl::OnInitialSearchCompleted calls SetPerformanceMode(Impl::EPerformanceMode::MostlyStatic) which is supposed to remove duplicates and sort dependencies on each FDependsNode.

What is the callstack to UpdateManagementDatabase and SetManageReferences you are seeing? Maybe it is occurring before the AssetRegistry’s search finishes.

But even with duplicates in the FDependsNodes package dependencies, it is not supposed to be possible to add duplicates in the ManageDependencies, because the SetManageReferences function uses a TMap to record whether each node discovered in the graph search has been added to the manager; this TMap is looked up in the call to FindOrAddNodeData in the loop at the bottom of SetManageReferences, and the loop body is early exited if bModifiedByCurrentManager was cleared by an earlier instance of the loop.

...
			for (FDependsNode* ModifiedNode : CurrentManagerModifiedNodes)
			{
				FSetManageReferencesNodeData& ModifiedData = FindOrAddNodeData(ModifiedNode);
				if (!ModifiedData.bModifiedByCurrentManager)
				{
					// A duplicate of a NodeData we already handled earlier in the list
					continue;
				}
 
				ModifiedNode->SetIsReferencersSorted(false);
				ModifiedNode->AddReferencer(ManagerNode);
				ManagerNode->AddDependency(ModifiedNode, EDependencyCategory::Manage,
					ModifiedData.CurrentManagerProperties);
 
				ModifiedData.bManagedInThisRound = true;
				ModifiedData.bModifiedByRound = true;
				CurrentRoundModifiedNodes.Add(ModifiedNode);
 
				ModifiedData.bModifiedByCurrentManager = false;
				ModifiedData.bVisitedByCurrentManager = false;
				ModifiedData.CurrentManagerProperties = EDependencyProperty::None;
			}
 
...

The first thing to check is whether that for loop is the one adding the duplicates in ManageDependencies.

Turn off optimizations in AssetRegistry.cpp by adding UE_DISABLE_OPTIMIZATION at the top of the file, and add instrumentation in that for loop:

			for (FDependsNode* ModifiedNode : CurrentManagerModifiedNodes)
			{
				FSetManageReferencesNodeData& ModifiedData = FindOrAddNodeData(ModifiedNode);
				if (!ModifiedData.bModifiedByCurrentManager)
				{
					// A duplicate of a NodeData we already handled earlier in the list
					continue;
				}
 
				ModifiedNode->SetIsReferencersSorted(false);
				ModifiedNode->AddReferencer(ManagerNode);
			// BEGIN_INSTRUMENTATION
			if (ManagerNode->ContainsDependency(ModifiedNode, EDependencyCategory::Manage))
			{
				static volatile int HitBreakpoint = true; ++HitBreakpoint;
			}
			// END_INSTRUMENTATION
				ManagerNode->AddDependency(ModifiedNode, EDependencyCategory::Manage,
					ModifiedData.CurrentManagerProperties);
 
				ModifiedData.bManagedInThisRound = true;
				ModifiedData.bModifiedByRound = true;
				CurrentRoundModifiedNodes.Add(ModifiedNode);
 
				ModifiedData.bModifiedByCurrentManager = false;
				ModifiedData.bVisitedByCurrentManager = false;
				ModifiedData.CurrentManagerProperties = EDependencyProperty::None;
			}

If it’s coming from there, we will need to add more instrumentation to find out how. Maybe the same manager is being passed into multiple calls of SetManageReferences from UpdateManagementDatabase, and we are redoing the graph search and repeatedly adding all of its nodes to it.

Also, what is the value of bShouldSortDependencies at the bottom of SetManageReferences?

I added

ensure(!ManagerNode->ContainsDependency(ModifiedNode, EDependencyCategory::Manage));

in there and hit hit when building the database. The last 2 entries in the Referencers array on the ModifiedNode are the same primary asset as well.Here’s the values of ModifiedData:

-	ModifiedData	{bManagedInEarlierRound=false bModifiedByRound=true bManagedInThisRound=true ...}	UE::AssetRegistry::FSetManageReferencesNodeData &
		bManagedInEarlierRound	false	bool
		bModifiedByRound	true	bool
		bManagedInThisRound	true	bool
+		DirectManagersThisRound	Empty	TArray<FDependsNode *,TSizedDefaultAllocator<32>>
+		DebugInstigator	0x0000016c59a3f6c0 {Identifier={PackageName="/Game/TNM1/Materials/Characters/M_Char_Teeth" PrimaryAssetType=...} ...}	FDependsNode *
		bModifiedByCurrentManager	true	bool
		bVisitedByCurrentManager	true	bool
		CurrentManagerProperties	None (0 '\0')	UE::AssetRegistry::EDependencyProperty

bShouldSortDependencies and bShouldSortReferencers are both true at the end of the function

Can you add instrumentation to UpdateManagementDatabase to see if during one execution of the function the same key is present in SetManageContext.ManagerMap in two of the calls to SetManageReferences?