bLabelAssetsInMyDirectory is set but cooks are pushing everything over to Chunk 0

Development Pipeline & Plugins
, ,
21

Yes, confirmed that “Map” showed up in the primary asset types to scan and that it passed the ShouldScanPrimaryAssetType check.

(Sorry for the delay there)

[Attachment Removed]

22

Looping back around, I did set up a primary asset label for the directory of the super small map that I’m using for my quicker cook, and I toggled bHasBlueprintClasses to false for Maps…

… And it’s still just one chunk.

Note: there’s still some meshes and materials under a different directory based label that are being cooked according to the UFSFiles Manifest that must still also be pushed to chunk 0 that I feel we haven’t gotten close to explaining here either.

[Attachment Removed]

23

I see, ok, I think I’m following.

We do in fact have a child class of the AssetManager, and it does override ShouldSetManager, but it’s just to force recursion off for one of our data asset types that are implicitly set up to chain from one to next.

EAssetSetManagerResult::Type UCUSTOM_AssetManager::ShouldSetManager(const FAssetIdentifier& Manager, const FAssetIdentifier& Source, const FAssetIdentifier& Target, UE::AssetRegistry::EDependencyCategory Category, UE::AssetRegistry::EDependencyProperty Properties, EAssetSetManagerFlags::Type Flags) const
{
	FPrimaryAssetId TargetPrimaryAssetId = Target.GetPrimaryAssetId();
  
 	if (!TargetPrimaryAssetId.IsValid())
 	{
 		// Prevent recursion into any asset references in DialogueAssets
 		if (Manager.PrimaryAssetType == UCUSTOM_DialogueAsset::PrimaryAssetType)
 		{
 			return EAssetSetManagerResult::SetButDoNotRecurse;
 		}
 	}
	
	return Super::ShouldSetManager(Manager, Source, Target, Category, Properties, Flags);
}

The other 4 overridden functions are:

  • ShouldCookForPlatform
  • GetPackageCookRule
  • PostInitialAssetScan
  • PreBeginPIE

I don’t think any of these other ones would affect how we assign chunk ids… considering those first two should just be eliminating things from the cook, not reorganizing, and the other two are intended just as a point to prep a separate game time cache we leverage.

I’m running a separate experiment now, but I’ll prep something I can look at in the morning (I’ve been adding a bunch of logs just in case I miss something… and I’ll just kick a run sometime tonight)

[Attachment Removed]

24

Random additional note: we have an extra branch that was set up to inspect 5.7 viability, and the cooked builds out of there show the same single chunk behavior.

[Attachment Removed]

25

The map shows up in AssetPackagesReferenced because of the Asset Ref.

Nothing seems to come up Bundles associated with that… Nor does the Label Directory data asset seem to come up at all even as an Asset Ref…

Then when it comes to grabbing the chunk id later, it preps 430 managers, and then finds 0 matches -> chunk 0.

Now that I’m looking more closely… If the directory label is supposed to work… Would it need to come up as an Asset Ref, and then it can associate its children via the bundle search? Is that what we’re missing? Or am I misinterpreting something here?

Like… would setting the primary asset labels to always cook make a difference here?

[Attachment Removed]

26

Thank you, this is very in depth.

I think I’ve read through and understand pretty well, and I’ll try to run through a test soon…

I preemptively put some logs into GetPackageManagers, and I think I spammed my computer into oblivion? TBD.

(Obviously didn’t understand well enough at the start)

[Attachment Removed]

27

Ok so the map first comes up in a reference inside ShouldSetManagers with no manager, a data asset referencing it as Source, then… itself as target… and that results in enum value 1 (set but not recurse)

That leads to an added node with no Dependencies according to the ManageProperties variable.

Then it comes up reference by another map that shouldn’t be getting cooked… Odd… But same enum results and add node behavior.

(Looks like that’s referenced by a cinematic)

Then the next mention seems to be an identical repeat of the first scenario written here… And then basically the second scenario again… then the first again… Back and forth…

aaaand I crashed.

[Attachment Removed]

28

Hmm Manager might be off because I was checking Package Name specifically…

[Attachment Removed]

29

Yup you said to check ObjectName there, whoops. Overprepared the wrong way.

[Attachment Removed]

30

… There’s thousands of hits against the Map as a Target…

Should I only be caring about the Map as Manager and Target simultaneously? I have myself set up for each individually, and I’m just completely underwater with the number of times it comes up…

Our basic “Item” data asset is a primary asset, and apparently contributes thousands of managers, and it seems to inevitably hit the specific map from each and every one.

(While I picked the smallest map, it’s also kind of a special one, so maybe it actually webs out to referencing or being referenced by pretty much everything)

[Attachment Removed]

31

Ok, I apparently had reverted the HasBlueprint toggle in the ini and not restored it, so that’s why it was never showing up.

Yes, now it’s first, and it returns enum val 2 Set and Recurse.

[Attachment Removed]

32

Oh and that comes through FAssetRegistryImpl::SetManageReferences with the manage properties value of 8 (direct)

A bunch of other stuff does add the node with a 0 (none).

[Attachment Removed]

33

I’ve got AssetManager printing out the local referencer now, with an added bit of only printing if the local referencer count changes - I’ve got 432 references to my map at the beginning…

Which is quite a lot higher than Lyra’s 2...

At the end of the UpdateManagementDatabase run, I get 432 as well…

And I see no further hits, which indicates it sticks with 432.

One of which is the map itself… None of which seems to be a label.

[Attachment Removed]

34

Ok, I adjusted my specific checks for that map to refer to our dragon themed nightstand mesh, and I’ll have that tested soon…

To address the different sections above…

  • Our items wouldn’t refer to a map directly, but can refer to experience package data assets or placeable blueprints. My assumption is it’s along one of these chains that the manager becomes relevant…
    • Experience Packages will reference a map internally via soft pointer
    • There are some placeables that have teleporting abilities, which will reference data assets that internally specify experience packages.
    • We do use apply recursively and do not have “hard refs only”, so that would make sense to me as a chain. I don’t know if we should be seeking to mitigate that somehow.
    • Both experience package and item types are set to use chunk id -1, so I figured they shouldn’t be critical to why things are ending up in chunk 0, other than maybe making an unfortunate chain from something else.
  • The prop is used by both a map and an item type. So it would be picked up by Item Type scan which would shouldn’t apply a chunk id (-1), and then it also has a label in the folder above it that applies to everything in the folder which should apply chunk id 11 (precisely “/Game/Props/Decor/Dragon” while the nightstand has one subfolder inbetween to separate out the mesh and all its materials)
  • The last point about the root path, I am still parsing… I think I follow… It’s been the root path for a long time, but I assume you’d say it’d be more orthodox to use the /Game folder?
    • Incidentally, I found an old comment via time lapse view that got smashed by the auto adjustment of the ini that says the root path needs to be last in the list… And it is definitely not last in the list now.
    • My understanding is we adjusted to use root instead of game because of a desire to put labels in a game feature folder that was explicitly separate.
    • Edit: extra note that things were added after this entry a year before the chunking situation we’re discussing here started. It’s worth resolving, but I think it’s a separate issue.
      [Attachment Removed]
35

Ok, there’s 1534 referencers to this mesh, amongst them is every map in the game (it should only be placed in 2 levels directly), many Minigame Configs… about Fish… and a Book. And many other odd things.

The book is interesting because all they normally have is text and icons… But this one book also starts a quest, which it refers to by data asset by soft reference… Which references a map location data asset internally (soft), which references a particular level (soft), which references the BP Game State (hard)… which probably leads back to the player inevitably, and leads to a whole dang chain from there.

We’ve been using Apply Recursively on a lot of these asset types… essentially forever… and chunk hard references only has NOT been enabled since at least 2020 according to revision history…

Could there have been a gap there that we were unknowingly taking advantage of during this time?

Edit: 1543, not 1534.

[Attachment Removed]

36

Hmm… There must have been a change… maybe the world as not blueprints, but I’m seeing 1559 known managers here.

LogAssetManager: Display: UAssetManager::GetPackageChunkIds - Kicking off /Game/Props/Decor/Dragon/<Nightstand> with already existing list sized 0 against 1559 known managers

based on

UE_LOG(LogAssetManager, Display, TEXT("UAssetManager::GetPackageChunkIds - Kicking off %s with already existing list sized %d against %d known managers"), *PackageName.ToString(), OutChunkList.Num(), Managers.Num());

[Attachment Removed]

37

I didn’t label these logs well enough around what changes I made for each… 16 shows 0 managers for the nightstand, while 21 shows 446.…

16 was from Friday, 17 was cancelled, 18 was on Monday a few hours after you posted about Maps.

[Attachment Removed]

38

Following line still comes back with 0 even after the 1559 known managers…

Which also that is 25 greater than 1534 16 greater than 1543.

edit: number was incorrect earlier.

[Attachment Removed]

39

Note: I let this complete and the summary from GetPackageManagers shows 1543 the same as the database run, and only one hit, so it stays at 1543 at the end then?

LogAssetManager: Display: UAssetManager::GetPackageManagers - SUMMARY - /Game/Props/Decor/Dragon/<Nightstand> has 1543 referencers:

So the extra 16 would come from elsewhere…

[Attachment Removed]

40

I have an old log line I added that indicates Chunk ID is never higher than 0... It’s usually no rules matched at all, but sometimes we get 1.

Comes back either of the following:

LogAssetManager: Display: UAssetManager::GetPrimaryAssetSetChunkIds - ran to find highest chunk 0 amongst total list size 0

LogAssetManager: Display: UAssetManager::GetPrimaryAssetSetChunkIds - ran to find highest chunk 0 amongst total list size 1

Getting ready to check for actual names soon…

[Attachment Removed]