External pak files and "use io store" option

Development Programming & Scripting C++
,
1

Hello,

I am working on an application that loads assets from external pak files.
It works well if I package my app and my pak files (separately) by disabling “use io store” in both cases.

This option is not well documented, so it’s not clear how it works and what it improves. There is a tooltip saying “if enabled, use .utoc/.ucas container files for staged/packaged package data instead of pak”. I read on this forum (here) that apparently it is better to keep it enabled for faster loading times.

If I enable the option on my side to package the application and the external pak (which also generates the 2 additional utoc and ucas files), then when I try to load it in the main application, the asset registry fails to correctly read the content of the pak.

I can correctly:

  • mount the pak with “pakFileManager->Mount(*inputPath, 0, *mountPoint);” which returns true.
  • extract the list of files contained in the pak:
 TArray<FString> pakContent;
 FPakPlatformFile::GetFilenamesFromIostoreContainer(FPaths::GetBaseFilename(pakFilePath), pakContent);

followed by some formatting to have paths like this “/Game/[directory_name]/[object_name].uasset” as it is done in this Epic tutorial.

Then I have:

FAssetRegistryModule& assetRegistryModule = FModuleManager::LoadModuleChecked<FAssetRegistryModule>("AssetRegistry");
IAssetRegistry& assetRegistry = assetRegistryModule.Get();
assetRegistry.ScanFilesSynchronous(pakContent, true);

After that, my code which normally loads assets fails (static meshes with textures).
I built Unreal Engine from the sources in debug, to be able to build and package my app from this build, to finally debug Unreal Engine when running my packaged app, and see what happens in ScanFilesSynchronous (the reason being that we can’t debug the loading of pak files in editor mode, neither by attaching Visual Studio to a packaged app by loading the UE debug symbols provided with its installer, which don’t allow to debug packaged builds in depth).
Then I tried to understand where it blocks and what Unreal Engine needs, but I am quite lost. I compared by enabling/disabling “use iostore”. UE finds the files, but at some point, with “use io store”, there seems to be a problem with the “uasset” extension which can’t be identified in FPackageName::DoesPackageExistEx (even if the files in the originally passed list have the uasset extension). I am not sure if this extension problem prevents the completion of the loading process but it might be.

So my question is:
Does anyone know if “use io store” can be enabled in an application that has to load external pak files?

2 Likes
2

I’m experiencing the same issue here. If anyone has any suggestions on how to resolve this, I’d really appreciate your input. It’s truly frustrating that they’ve introduced the Io Store feature without providing any proper documentation. Moreover, its implementation seems to go completely against the dynamic nature of Unreal Engine’s design.
Can’t believe its been out since years…

1 Like
3

@NicolasDev84 Have you found an answer to your question in the meantime maybe? I’m quite curious about this as well.

4

I’ll try to explain what’s happening here.

IOStore and the new container files it creates (.utoc/.ucas) is our new container format for asset data. It was specifically built for the new loading code and stores the asset information/headers separately from the asset data (roughly what was previously stored as .uasset (header) and .uexp/.ubulk (data) files in a pak. It’s more performant and storing assets in pak files is not recommended anymore.

The issue here is that the tutorial is old and was not written with IOStore in mind.
With IOStore enabled there are several things that must be taken into consideration and done slightly differently to obtain the same outcome.

First, we recommend against using/iterating raw file names in .pak files and using paths to .uasset files directly. The Engine has a virtual file system that provides package paths that don’t need to care about where the files/packages are stored (which might not even be on disk in the future).

In the same vein, IOStore containers now longer expose individual files, so the .uasset files do not exist any more when packaging to IOStore. Instead the new containers work on package paths.

The new format provides packages and a assets that would have lived in a pak file under /Game/Content/MyFolder/MyAsset.uasset should only be accessed by its package path ( /Game/MyFolder/MyAsset) when using IOStore. Trying to load raw files like the tutorial does will fail in this case.

Other non-asset files will still be stored inside the pak files, this is the reason that pak files are much smaller but still exist when enabling IOStore containers.

Additionally the AssetRegistry cannot scan a .utoc/.ucas for contents, since there is no folder structure/files to scan anymore. Instead the information about which assets/packages are contained in an IOStore Container are written to a separate AssetRegistry.bin file. This AssetRegistry data will be stored in the pak file and automatically loaded by the AssetRegistry when a pak file with an AssetRegistry.bin is loaded.

If the AssetRegistry.bin is correctly loaded the IAssetRegistry API will know about and correctly find any packages inside the IOStore containers.

So the TLDR for correctly doing this with IOStore containers enabled is:

  • Make when packaging that your pak file contains the serialized AssetRegistry.bin
  • Load the pak file as usual and the AssetRegistry should now about the files inside the IOStore containers.
  • Do not try to use ScanPathsSynchronous to scan IOStore containers, it won’t work and can possibly even clear assets that the registry loaded from an AssetRegistry.bin
  • Instead of searching for files in side the pak file use the AssetRegistry to search for or enumerate your Assets

Hopefully this should clear things up a bit.

Best,

Sebastian