Content not being found in a packaged application after upgrading to 5.5

Development Pipeline & Plugins
, ,
1

Hey, we upgraded to UE5.5 and now we are unable to load the UMoviePipelinePrimaryConfig-Assets in a packaged project.

We add the folders that contain those assets in “Additional Asset Directories To Cook” in the Project Settings.

In UE 5.3 this worked well, but in 5.5 it somehow doesn’t work for our project.

I tried to create a reproducer, but in an empty project with the same logic as in our project it worked.

Here are some more informations I found out:

In the cooking log it properly lists the assets:

UATHelper: Packaging (Windows): LogCook: Display: Cooking /MyPlugin/MySubfolder/MRQConfigAsset, Instigator: { CommandLineDirectory: D:/*Path to Project*/Plugins/MyPlugin/Content/MySubfolder }After the package process I can also find them in the “Manifest_UFSFiles_Win64.txt”

When trying to load them we basically first use the AssetRegistry.ScanPathsSynchronous on the folders containing the assets with ForceRescan=true.

Afterwards we try to retrieve them with AssetRegistry.GetAssetsByClass. As mentioned, this worked well in 5.3.

While debugging the issue I also used the CommandLine version:

“AssetRegistry.ScanPath /MyPlugin/MySubfolder/ -forcerescan -ignoredenylists”

and

“AssetRegistry.ScanPath /MyPlugin/ -forcerescan -ignoredenylists”

With the verbose logging enabled it shows some issues:

LogAssetRegistry: Verbose: SetPropertiesAndWait called on non-existent path D:/*Path to Project*/Build/5.5/Windows/*Packaged Project Path*/Plugins/MyPlugin/Content/MySubfolder. Call will be ignored. and LogAssetRegistry: SetPropertiesAndWait called on D:/*Path to Project*/Build/5.5/Windows/*Packaged Project Path*/Plugins/MyPlugin/Content which is not in a mounted directory. Call will be ignored.

“PackageName.DumpMountPoints” does properly list the plugin though:

LogPackageName: '/MyPlugin/' -> '../../Plugins/MyPlugin/Content/'

When using “AssetRegistry.DumpState All -log” it lists all assets in our other plugins but not the one with the UMoviePipelinePrimaryConfig-Assets.

What am I missing here or how can I debug further ?

2

Steps to Reproduce

3

Hi,

can you load the affected assets without using the AssetRegistry, e.g. through a softobjectpath, using the regular content path of the asset (/MyPlugin/path/to/asset)?

Is the packaging here a regular packaged build with the plugin included or are you creating separate packages, e.g. through chunking or the DLC workflow?

Do you create IOStore containers or just pak files?

Is there any specific mounting logic or are your containers mounted by Unreal on startup from the content folder?

We’ve had similar issues internally when IOStore was enabled for Editor builds. These were caused by different loading orders for some content, where the path is added to the Assetregistry before the content was mounted.

Does your logs show the mounting (specifically the packagepath being added, not the pak/iostore mount) happening before this message?

In general if you package your content through a normal build your container will include an AssetRegistry.bin that already knows about all the assets inside the container. An additional scan will not be necessary and will not work at all for IOStore containers, since these don’t expose a filesystem abstraction. IOStore Assets need to be discovered via the serialized AssetRegistry.bin right now.

Kind Regards,

Sebastian

4

Hey Sebastian,

I found out several things by following your questions:

1. Can you load the affected assets without using the AssetRegistry, e.g. through a softobjectpath, using the regular content path of the asset (/MyPlugin/path/to/asset)?

- Yes, I was able to load the assets through a softobjectpath.

2. Is the packaging here a regular packaged build with the plugin included or are you creating separate packages, e.g. through chunking or the DLC workflow?

- We create a regular packaged build with the plugin included.

3. Do you create IOStore containers or just pak files?

- We have the IOStore enabled in the Project Settings, but we do no chunking (“GenerateNoChunks”). Unticking “GenerateNoChunks” has no effect, but disabling the IOStore (unticking “UseIOStore”) did work. How could I debug further to find out why it isn’t working with the IOStore?

4. Is there any specific mounting logic or are your containers mounted by Unreal on startup from the content folder?

- There is no specific mounting logic, everything is mounted by Unreal on startup

5. Does your logs show the mounting (specifically the packagepath being added, not the pak/iostore mount) happening before this message?

- I don’t see any mounting of PackagePaths in the log. I gathered all related logs in order:

... LogPakFile: Initializing PakPlatformFile LogIoDispatcher: Display: Reading toc: ../../../MyProject/Content/Paks/global.utoc LogIoDispatcher: Display: Toc loaded : ../../../MyProject/Content/Paks/global.utoc, Id=ffffffffffffffff, Order=0, EntryCount=1, SignatureHash=0000000000000000000000000000000000000000 LogIoDispatcher: Display: Mounting container '../../../MyProject/Content/Paks/global.utoc' in location slot 0 LogPakFile: Display: Initialized I/O dispatcher file backend. Mounted the global container: ../../../MyProject/Content/Paks/global.utoc LogPakFile: Display: Found Pak file ../../../MyProject/Content/Paks/MyProject-Windows.pak attempting to mount. LogPakFile: Display: Mounting pak file ../../../MyProject/Content/Paks/MyProject-Windows.pak. LogIoDispatcher: Display: Reading toc: ../../../MyProject/Content/Paks/MyProject-Windows.utoc LogIoDispatcher: Display: Toc loaded : ../../../MyProject/Content/Paks/MyProject-Windows.utoc, Id=148a5bb5dd32785b, Order=4, EntryCount=6887, SignatureHash=0000000000000000000000000000000000000000 LogIoDispatcher: Display: Mounting container '../../../MyProject/Content/Paks/MyProject-Windows.utoc' in location slot 0 LogPakFile: Display: Mounted IoStore container "../../../MyProject/Content/Paks/MyProject-Windows.utoc" LogFilePackageStore: Mounting container: Id=148a5bb5dd32785b, Order=4, NumPackages=5249 LogPakFile: Display: Mounted Pak file '../../../MyProject/Content/Paks/MyProject-Windows.pak', mount point: '../../../' ... LogAssetRegistry: Premade AssetRegistry loaded from '../../../MyProject/AssetRegistry.bin' ... LogPluginManager: Mounting Engine plugin ... LogPluginManager: Mounting Project plugin ... ... LogAssetRegistry: FAssetRegistry took 0.0001 seconds to start up ... LogFilePackageStore: Updated: NewPackages=5249, OldPackages=0, TotalPackages=5250 ... LogInit: Display: Game Engine Initialized. ...

Regarding: In general if you package your content through a normal build your container will include an AssetRegistry.bin that already knows about all the assets inside the container. An additional scan will not be necessary and will not work at all for IOStore containers, since these don’t expose a filesystem abstraction. IOStore Assets need to be discovered via the serialized AssetRegistry.bin right now.

This information cleared up a lot for me, thanks.

When packaging the project the IOStore logs:

“UATHelper: Packaging (Windows): LogIoStore: Display: Using input asset registry: “ProjectPath”/Saved/Cooked/Windows/MyProject/Metadata/DevelopmentAssetRegistry.bin”

When loading that AssetRegistry into the AssetAudit-Tool I can find all MRQ-PrimaryConfig assets.

So I don’t know what I might be missing here, is the serialized AssetRegistry another one which might not contain those assets?

Kind Regards,

Dominikus

5

Hey,

thanks for the detailed answers and apologies for the delay, I was out of office last week.

Looking at the other info from your last post this looks like an issue with the serialized AssetRegistry. It’s either missing those files or there’s some other reason why it doesn’t find/know about the assets.

The fact that you can load them manually suggests that the containers are loaded and the package paths are set up correctly for those paths.

> So I don’t know what I might be missing here, is the serialized AssetRegistry another one which might not contain those assets?

That is correct, the DevelopmentAssetRegistry.bin contains information about all the assets from the cook, but not the assets in the final pak/iostore files.

The runtime asset registry cache is stored in the “AssetRegistry.bin” file, which is staged in the Saved/Cooked/Windows/ProjectName folder after cooking, or can be directly extracted from the generated pak file (note that non-uasset/ubulk files will always end up in the pak file, not in the .utoc/.ucas files).

Before going that route you can also check the state of the AssetRegistry in your packaged game.

This should also dump the info from the Asset Registry to the log:

AssetRegistry.DumpState -log PackageName Class

One thing that comes to mind is that the assets might use a different class at runtime, after the cooking process (similar to how UBlueprint becomes a UBlueprintGeneratedClass), so this output will also list the class name before listing all the assets of each class.

If the assets are listed here it’s likely your query that is the issue.

If not I’d check the included AssetRegistry.bin file for the presence of those assets, just do double check.

Best,

Sebastian

6

Hey,

Running “AssetRegistry.DumpState -log PackageName Class” also didn’t show the package names of the assets, nor the class. (MoviePipelinePrimaryConfig)

I also hooked the debugger in the cooking process again and checked what gets serialized in the AssetRegistry. I couldn’t see any issue, in the cooking process the function Generator.SaveAssetRegistry() [CookOnTheFlyServer.cpp ln. 10817] also seemed to work properly. In the AssetRegistryState I could find the packages in CachedAssetsByClass & CachedAssetsByPath.

Afterwards I used “UnrealPak.exe -extract” on the Pak file and extracted the content and the AssetRegistry. I used the commandlet AssetRegistryDumpCommandlet on it and it generated “Page_XXXXX.txt” files. I was also able to locate the packages here:

--- begin cachedassetsbypath --- /averendering/mrq_configs : 19 item(s) /myplugin/mrq_configs/mrq-testconfig_inplugin.mrq-testconfig_inplugin ... ... /game/cameras/rendersettings : 1 item(s) /game/mycontentfolder/mrq-testconfig_ingame.mrq-testconfig_ingame ... --- end cachedassetsbypath : 349 entries --- ... --- begin cachedassetsbyclass --- ... /script/movierenderpipelinecore.moviepipelineprimaryconfig : 15 item(s) /myplugin/mrq_configs/mrq-testconfig_inplugin.mrq-testconfig_inplugin /game/mycontentfolder/mrq-testconfig_ingame.mrq-testconfig_ingame ... ... --- end cachedassetsbyclass : 82 entries ---So it seems like the AssetRegistry is properly setup? Then how come I cannot find the assets when I run the game and use “AssetRegistry.GetAssetsByClass(…)”

7

Hey Sebastian,

I finally found the issue!

I could swear that I already tried removing it in my previous tests since you mentioned it, but the culprit is the “AssetRegistry.ScanPathsSynchronous()”.

If I wrap this line with “#if WITH_EDITOR” it works as expected.

I didn’t dig into the code what actually happens in ScanPathsSynchronous at runtime, but I guess the previously serialized AssetRegistry gets overridden with a new one that can’t find any files since they are packaged in the io store containers and not found in the file system?

It is also weird that this worked in 5.3. So maybe there were some changes in the AssetRegistry code inbetween 5.3 and 5.5.

Anyways, I am glad it works now and I was able learn a lot in the process.

Cheers,

Dominikus

8

Glad to hear that, thanks for letting us know!

Does this also happen when not using/setting the ForceRescan option?

From a quick glance at the code using ForceRescan will delete any known assets at the searched paths, so this could indeed cause the behavior you’re seeing.

Without that option I think the AssetRegistry should not drop any already known assets, if that’s still the case for you I’ll have to investigate closer and potentially file a bug report.

Thanks!

Sebastian

9

I tested it with ForceRescan=False and it still worked.

So this is not a bug, but the comment could be more specific. It states that paths will be scanned even if they were previously scanned. This could lead to the idea that there is an optimization with ForceRescan=False, which would not scan paths that were previously scanned.

Or when the functionality changed from 5.3 to 5.5, which caused the issue for us, the comment was not updated.

/** Scan the supplied paths recursively right now and populate the asset registry. If bForceRescan is true, the paths will be scanned again, even if they were previously scanned */ UFUNCTION(BlueprintCallable, Category = "AssetRegistry") virtual void ScanPathsSynchronous(const TArray<FString>& InPaths, bool bForceRescan = false, bool bIgnoreDenyListScanFilters = false) = 0;Anyways, with this we can close the issue. Thanks for your support!

10

Thanks for the extra test and letting us know!

That clarifies that it’s at least specific to bForceRescan.

I’ll create a bug report to improve that behavior and see if we can update the comment in the meantime to make people aware of that pitfall.

I’ll close mark the question as closed!

Best,

Sebastian