Add primary asset support to DDC commandlet

Development Pipeline & Plugins
, ,
1

For games that are driven by primary assets that are not just maps, the covered set of assets that this commandlet caches is misaligned with what is actually in the game.

We’d like to add a new option to have the starting set of assets populated by querying for primary assets. This also allows customization through game-specific GFT policies. Existing commandlet options can still be used to further refine the list.

I have a PR of this change here https://github.com/EpicGames/UnrealEngine/pull/14077

For sure we’d also be open to feedback on alternate ways to accomplish this.

[Attachment Removed]

2

Thanks, your change looks good to me at a high level; I will integrate the pull request and let you know if I find any changes I need to make to it during testing.

[Attachment Removed]

3

Roger, I have started working on this. I tested with the addition of the AssetRegistry.SearchAllAssets call.

I tested the code on Lyra, and made two changes:

  • Call ModifyCook individually for every TargetPlatform rather than for the editor platform; this fixes the behavior for projects that do some platform-specific primary assets in their ModifyCook.
  • Append the PrimaryAssetNames into CommandLinePackageNames, rather than passing them in as the PackageNames argument to NormalizePackageNames called with the *.uasset,*.umap wildcards. This makes them behave the same as if they were passed in as commandline arguments, which is less surprising behavior.

Can you test it out and let me know if there are any problems with those two changes?

// Moved the block you added on line 616 up to line 585, and changed it to:

...
				for (const FSoftObjectPath& AssetPath : FoundAssets)
				{
					CommandLinePackageNames.Add(AssetPath.GetLongPackageName());
				}
			}
		}
 
// 585: INSERTED BLOCK
		// check to see if we should use the primary asset as the starting set
		if (Switches.Contains("USEPRIMARYASSETS"))
		{
			// SearchAllAssets is required by ModifyCook
			IAssetRegistry& AssetRegistry = FModuleManager::Get().LoadModuleChecked<FAssetRegistryModule>(AssetRegistryConstants::ModuleName).Get();
			AssetRegistry.SearchAllAssets(true /* bSynchronousSearch */);
 
			// Call ModifyCook with one TargetPlatform at a time and take the union of PrimaryAssets. If we have
			// multiple target platforms and they disagree about development objects, ModifyCook produces an error
			// because the cook output is invalid. We're not producing cooked output, so we don't care about that, so
			// call them one at a time to avoid the error.
			TSet<FName> PrimaryAssetNames;
			for (const ITargetPlatform* TargetPlatform : Platforms)
			{
				TArray<FName> PlatformPrimaryAssets;
				TArray<FName> Unused;
				UAssetManager::Get().ModifyCook({ TargetPlatform }, PlatformPrimaryAssets, Unused);
				PrimaryAssetNames.Append(PlatformPrimaryAssets);
			}
			for (FName PrimaryAsset : PrimaryAssetNames)
			{
				CommandLinePackageNames.Add(PrimaryAsset.ToString());
			}
		}
// END INSERTED BLOCK. Note the following code uses CommandLinePackageNames so it needs to come after the insrted block.
 
		// Add defaults if we haven't specifically found anything on the command line 
		if (Tokens.IsEmpty() && CommandLinePackageNames.IsEmpty())
		{
			UE_LOG(LogDerivedDataCacheCommandlet, Display, TEXT("Adding default search tokens for all assets and maps"));
 
			Tokens.Add(FString("*") + FPackageName::GetAssetPackageExtension());
			Tokens.Add(FString("*") + FPackageName::GetMapPackageExtension());
		}
...

// Removed the change to the function call to NormalizePackageNames on line 648 inside the for loop commented with assume the first token is the map wildcard/pathname

// Code starting at line 641 is now unchanged form UE5/Main:
 
		// assume the first token is the map wildcard/pathname
		TSet<FString> FilesInPath;
		TArray<FString> Unused;
		TArray<FString> TokenFiles;
		for ( int32 TokenIndex = 0; TokenIndex < Tokens.Num(); TokenIndex++ )
		{
			TokenFiles.Reset();
			if (!NormalizePackageNames(Unused, TokenFiles, Tokens[TokenIndex], PackageFilter))
			{
				UE_LOG(LogDerivedDataCacheCommandlet, Display, TEXT("No packages found for parameter %i: '%s'"), TokenIndex, *Tokens[TokenIndex]);
				continue;
			}
 
			FilesInPath.Append(TokenFiles);
		}

[Attachment Removed]

4

Submitted in 8017b8d6fc8b2e88cfe40cfd7787c0f5fa8c0b14 aka CL 49628855

[Attachment Removed]

5

Awesome, thanks!

[Attachment Removed]

6

Hey Matt. I added another small change to the PR.

It looks like the ModifyCook will initialize the asset management database, which assumes a registry scan was started, and could hit some warnings if its not. This didnt affect the DDC fill itself, but the warnings make the horde job angry

This is the commit https://github.com/EpicGames/UnrealEngine/pull/14077/commits/bc564c72a1ab829a98e6ff952ae959b935b36509

[Attachment Removed]

7

Looks good to me. I’ll update our implementation to match. Thanks again for taking look at this!

[Attachment Removed]