Knowledge Base: Primer: Loading Content and Pak Files at Runtime

Development Platform & Builds
1

Primer: Loading Content at Runtime
This document is intended to give an overview over the various aspects of loading content/assets at runtime. It is specifically focusing on cooked content and native Unreal assets.
Th…

https://dev.epicgames.com/community/learning/knowledge-base/D7nL/unreal-engine-primer-loading-content-and-pak-files-at-runtime

9 Likes
2

Is this also working in UE5?

3

I attempted to inform the author of some issues with the article, but I was unable to reach them, so I’m posting the issues here so those who come across this in the future are aware of them…

I noticed that in the text it seems like some data was lost in the process, likely due to the text being misinterpreted as formatting or discarded by bad sanitization code. There’s another issue as well with the paths as the “parent directory” specifiers which are supposed to be .. (dot dot) are actually a single ellipses character “…” which look like three dots.

The first instance is under Loading Pak Files > Automated Loading
Where it shows “/Content/Paks” I believe that should be along the lines of <ProjectName>/Content/Paks

Other instances like this occurs in the following texts

The paths used for creating a mount point are always relative to the actual game executable, which is usually located in “/Binaries//” (for a packaged project).

I believe that one should be <ProjectName>/Binaries/<PlatformName>/

/Game/ which is mapped to the project’s content folder (…/…/…//Content/)

Should be ../../../<ProjectName>/Content/

Users should be aware that these assume the folder structure of a packaged project, so the project content will be in /Content/ instead of Content/.

Should be “will be in <ProjectName>/Content/ instead of Content/”

Also perhaps wrapping the paths in backticks for markdown inline code block formatting, like I have done in this message, would also be helpful. Actually, I just noticed where I hadn’t done that in this message and it confirms that markdown formatting is messing up the original article a bit! If I remove the code formatting from this text <ProjectName>/Binaries/<PlatformName>/ it becomes /Binaries//

Thanks! Hopefully making these changes will make this article a bit more informative and comprehensive for future readers.

1 Like
4

Hello! Is there a way to generate PAK files form runtime build?

5

I’m the original author of this text, but it was copied over from an internal version on the UDN. Unfortunately a lot of the formatting has been lost in that transition, and since I wasn’t the author here on EDC I didn’t get notifications for your posts.
Just wanted to thank you for the feedback, I’ve finally fixed most of the issues, even if it’s almost a year later :wink:

1 Like
6

Hello! Is there a way to generate PAK files form runtime build?

No, this is not possible at the moment. The tools required for creating .pak and IOStore containers (specifically UnrealPak.exe) only exist in the Editor build and are covered by a different EULA, so you cannot distribute them with a game without an additional license agreement.

7

I’ve been experimenting around this for mod support considerations.

My goal was to set up a BP project that supports mods, relying entirely on default behavior of plugin manager & asset registry, with no C++ involved.

I finally got it to work properly, after running into a few traps and working them out, which I’m gonna document below. This was on UE 5.1

Note that for UE4 projects, a BP-only implementation will be severely restricted due to limited asset registry methods. More precisely, UE5 provides a node GetBlueprintAssets which is designed to find blueprint subclasses, and the ability to cast an UObject to UClass which is necessary after sync/async loading a BP Asset. UE4 blueprints come short on both aspects.

Basegame Setup

I made a base game project, with a stub blueprint base class (BP_Base), a basic level and a material for it.
Then I created BP_Spawner blueprint which uses asset registry to find all blueprints child of BP_Base, and spawn them. This is the spawner BP code :

image
image1787×485 143 KB

I added BP_GameDerived as a child of BP_Base, which doesn’t do much other than confirming my spawner works.
Base game is packaged with default settings (minus IOStore, I haven’t experimented with that yet).

Mod setup

Then I made another project in which I copied BP_Base. This is intended to be the “SDK” for mod support.
Using project launcher I packaged with a first profile “SDK” which will contain all “shared” assets, that are in the base game, and should not be cooked in mods. It will serve as the reference version :
image
also in Packaging settings :
image

Then I created a new Content-Only plugin called Mod1, via the standard plugin creation window.
In plugin contents I created a new material, and another child of BP_Base (BP_ModDerived).
In the new class I set up a timer that selects all meshes and makes them blink between the new material and default material every second.

Using Project Launcher I set up second profile “Mod1” which in turn uses SDK as a reference version to package a DLC

image
image436×558 25.3 KB

note: error in image, should be “SDK” instead of “DLC” in the field above

After packaging, I can confirm the PAK file only contains intended content and does not conflict with any main game file :

Display: Mount point ../../../ModProject/Plugins/Mod1/
Display: "AssetRegistry.bin"
Display: "Content/BP_ModDerived.uasset"
Display: "Content/BP_ModDerived.uexp"
Display: "Content/M_Mat.uasset"
Display: "Content/M_Mat.uexp"

Shipping

All of this was pretty straightforward. After this I ran into multiple “traps” before successfully getting the basegame to detect the mod (and its contents) properly.

The packaged game file structure looks like this

GameProject.exe
Engine/...
GameProject/
  - Binaries/...
  - Content/Paks/GameProject-Windows.pak

The packaged mod file structure looks like this

ModProject/
  - Plugins/
    - Mod1/
      - Mod1.uplugin
      - Content/Paks/Windows/Mod1ModProject-Windows.pak

There are a few things to note.

Assets in the mod pak lie at ../../../ModProject/Plugins/Mod1/Content/... which means they can only be loaded if the base game mounts a custom path towards them. They are not in /Game or any other default virtual path. If you simply drop the PAK alongside main game’s PAK, it will be loaded, but its assets won’t be reachable.

It may cross your mind that, if your ModProject has the same name as GameProject, and you package mod assets in the /Game folder, then you can drop the mod PAK alongside main game PAK and the paths would work out. That is partially true. Assets may be reachable with a StaticLoadObject, but mod pak asset registry is still at /Plugins/Mod1/AssetRegistry.bin and will not be loaded. You might also want to try not using a plugin at all, so that the asset registry ends up in /Game/AssetRegistry.bin. In this case it will conflict with the main game’s asset registry and only one of them will be loaded, so this is not a good option.

PluginManager automatically mounts virtual paths for plugins contents, in the following form :
/PluginName → ../../../GameProject/Plugins/PluginName/Content
So if we follow a proper plugin approach, we should get a proper path towards both the mod asset registry and its assets.

PluginManager only enables plugins specified in the .upluginmanifest file, which is generated and paked automatically for the basegame (not modifiable). However you can manually create additional .upluginmanifest files in the packaged GameProject/Plugins/ folder and they will be read as well.

I failed to get the -DLCPakPluginFile switch to create a .upluginmanifest automatically as mentioned in the above primer.

PluginManager will also automatically load all plugins located in the GameProject/Mods/ folder, however when doing so it will expect asset paths in the Pak to match asset paths on disk. This removes the need to create .upluginmanifest, but you must package the mod as a mod instead of plugin and use the same project name as the base game (more details below).

Mods must be packaged without shared shader archive
image
The base game can have this setting enabled or not, it should not make any difference.

In the end, I’ve got two working approaches, detailed below.

1. Mods as plugins

The advantage of this approach (compared to #2) is that you can work around the ModProject having a different name than the GameProject.

If you simply drop the packaged plugin into the basegame’s plugin folder, it will not load your plugin, as the game relies on a .upluginmanifest file that does not specify to load this specific plugin.

You can manually create one (or multiple) new manifest(s) in the Plugins/ folder. They must not be in a subfolder. They must not conflict with the base game’s upluginmanifest, which is in the base game’s PAK at path GameProject/Plugins/GameProject.upluginmanifest.

Create for example GameProject/Plugins/Mod1.upluginmanifest with the following format :

{
	"Contents": [
		{
			"File": "../../../ModProject/Plugins/Mod1/Mod1.uplugin",
			"Descriptor": {
				"FileVersion": 3,
				"CanContainContent": true
			}
		}
	]
}

Descriptor is essentially a copy of what you have in the .uplugin file. I only included what I believe is necessary. The other fields that are usually present do not seem necessary.

The File field is very important here, as this tells the PluginManager where to find the plugin files, and this is where the plugin virtual path will point to.

If your ModProject has the exact same name as GameProject, then plugin should be in the GameProject/Plugins/ folder.
Packaged file structure will look like this :

Engine
GameProject/
  - Binaries
  - Content/Paks/GameProject-Windows.pak
  - Plugins/
    - Mod1.upluginmanifest
    - Mod1/
      - Mod1.uplugin
      - Content/Paks/Windows/Mod1GameProject-Windows.pak

In this case, the File field can be shortened to ../../Plugins/Mod1/Mod1.uplugin.

If however the ModProject has a different name, then you must create additional folders to match the paths of assets paked into the plugin PAK.
Packaged file structure must look like this :

Engine
GameProject/
  - Binaries
  - Content/Paks/GameProject-Windows.pak
  - Plugins/
    - Mod1.upluginmanifest
ModProject/
  - Plugins/
    - Mod1/
      - Mod1.uplugin
      - Content/Paks/Windows/Mod1GameProject-Windows.pak

"File": "../../../ModProject/Plugins/Mod1/Mod1.uplugin"

2. Mods as mods

As mentioned earlier, this approach does not require to manually create a .upluginmanifest file, however it comes with other constraints.

The PluginManager loads mods in GameProject/Mods folder so your packaged file structure must look like this :

Engine
GameProject/
  - Binaries
  - Content/Paks/GameProject-Windows.pak
  - Mods/
    - Mod1/
      - Mod1.uplugin
      - Content/Paks/Windows/Mod1GameProject-Windows.pak

PluginManager will detect the .uplugin, and automatically mount the pak, and will automatically create the following virtual path
/Mod1 → ../../../GameProject/Mods/Mod1/Content

As seen earlier, when packaging our mod as a plugin we ended up with a PAK with the following contents :

Display: Mount point ../../../ModProject/Plugins/Mod1/
Display: "AssetRegistry.bin"
Display: "Content/BP_ModDerived.uasset"
Display: "Content/BP_ModDerived.uexp"
Display: "Content/M_Mat.uasset"
Display: "Content/M_Mat.uexp"

So there is clearly a mismatch as the PAK contains stuff in ModProject/Plugins/Mod1 while the plugin manager creates a path towards GameProject/Mods/Mod1.

Luckily, fixing this is pretty straightforward :

  • First, the mod project must be exact same name as the game project.

  • Second, Plugins must be Mods instead. Fortunately, the two seem to be interchangeable. Editor and cooker seem to support mods just the same. So after creating your content plugin via editor UI, all you have to do is hop into the folders, create a Mods folder, and move the plugin from Plugins folder into the Mods folder, then reload editor. Nothing should have changed, but now after packaging the PAK will have its contents in ModProject/Mods/Mod1 instead.

After packaging, simply drop the packaged Mods/ folder directly into the packaged GameProject/ folder and it should work just fine.

As a final note, all of this only works at game launch. As stated in the primer above, PluginManager will only look for plugins and PAKs on startup.

For any more dynamic solution, a custom C++ implementation will most likely be required.

But I figured it was a good idea to show what is possible to do with builtin tools already, and where problems can easily arise - ie. virtual path vs. paked path issues for the most part. Most of these traps would likely be present when dealing with dynamic pak loading as well.

If you are struggling with similar issues as I did, here are some debugging methods I used, that will dump info out of shipping builds (no logs) :

#include "Interfaces/IPluginManager.h"
#include "IPlatformFilePak.h"
#include "AssetRegistry/IAssetRegistry.h"
#include "AssetRegistry/AssetRegistryHelpers.h"

void DebugPlugins()
{
	TArray<FString> Output;
	auto& PluginManager = IPluginManager::Get();
	auto AllPlugins = PluginManager.GetDiscoveredPlugins();
	for (const auto& Plugin : AllPlugins)
	{
		Output.Emplace(FString::Printf(TEXT("plugin:%s, enabled:%i, hasContent:%i, mount:%s, explicitlyLoaded:%i"), *Plugin->GetName(), Plugin->IsEnabled(), Plugin->CanContainContent(), *Plugin->GetMountedAssetPath(), Plugin->GetDescriptor().bExplicitlyLoaded));
	}
	FFileHelper::SaveStringArrayToFile(Output, *FPaths::Combine(FPaths::ProjectDir(), TEXT("plugins.log")));
}

void DebugPaks()
{
	TArray<FString> Output;
	auto PakPlatform = (FPakPlatformFile*)FPlatformFileManager::Get().FindPlatformFile(TEXT("PakFile"));
	if (PakPlatform != nullptr)
	{
		TArray<FString> Paks;
		PakPlatform->GetMountedPakFilenames(Paks);
		for (const auto& Pak : Paks)
		{
			Output.Emplace(FString::Printf(TEXT("pak: %s"), *Pak));
		}
	}
	else
	{
		Output.Emplace("PakPlatformFile = nullptr");
	}
	FFileHelper::SaveStringArrayToFile(Output, *FPaths::Combine(FPaths::ProjectDir(), TEXT("paks.log")));
}

void DebugVirtualPaths()
{
	TArray<FString> Output;
	TArray<FString> RootPaths;
	FPackageName::QueryRootContentPaths(RootPaths);
	for (const auto& RootPath : RootPaths)
	{
		FString ContentPath;
		FPackageName::TryConvertLongPackageNameToFilename(RootPath, ContentPath);
		Output.Emplace(FString::Printf(TEXT("RootPath: %s    -->    %s"), *RootPath, *ContentPath));
	}
	FFileHelper::SaveStringArrayToFile(Output, *FPaths::Combine(FPaths::ProjectDir(), TEXT("paths.log")));
}

void DumpAssets()
{
	TArray<FString> Output;
	auto AR = IAssetRegistry::Get();
	auto DumpAssetsInPath = [&AR, &Output](const FString& Path) {
		Output.Emplace(FString::Printf(TEXT("--- %s"), *Path));
		TArray<FAssetData> Assets;
		AR->GetAssetsByPath(*Path, Assets, true);
		for (const auto& Asset : Assets)
		{
			Output.Emplace(FString::Printf(TEXT("Asset: %s"), *Asset.GetFullName()));
		}
	};
	DumpAssetsInPath("/Game");
	DumpAssetsInPath("/Mod1");
	FFileHelper::SaveStringArrayToFile(Output, *FPaths::Combine(FPaths::ProjectDir(), TEXT("assets.log")));
}

void DebugChildBlueprints()
{
	TArray<FString> Output;
	auto AR = IAssetRegistry::Get();
	TArray<FAssetData> Assets;
	FARFilter Filter;
	Filter.TagsAndValues.Add("ParentClass", FString("/Script/Engine.BlueprintGeneratedClass'/Game/BP_Base.BP_Base_C'"));
	AR->GetAssets(Filter, Assets);
	for (const auto& Asset : Assets)
	{
		Output.Emplace(FString::Printf(TEXT("Asset: %s"), *Asset.GetFullName()));
	}
	FFileHelper::SaveStringArrayToFile(Output, *FPaths::Combine(FPaths::ProjectDir(), TEXT("childbp.log")));
}

void DebugStaticLoad()
{
	TArray<FString> Output;
	auto StaticLoad = [&Output](const FString& Path) {
		UObject* Obj = StaticLoadObject(UObject::StaticClass(), nullptr, *Path);
		Output.Emplace(FString::Printf(TEXT("StaticLoad('%s'): %s"), *Path, Obj ? *Obj->GetFullName() : TEXT("NULL")));
	};
	StaticLoad("/Game/BP_GameDerived.BP_GameDerived_C");
	StaticLoad("/Game/Mod1/BP_ModDerived.BP_ModDerived_C");
	StaticLoad("/Mod1/BP_ModDerived.BP_ModDerived_C");
	FFileHelper::SaveStringArrayToFile(Output, *FPaths::Combine(FPaths::ProjectDir(), TEXT("staticload.log")));
}
8 Likes
8

Awesome job @Chatouille. :+1:

9

Hey I just stumbled upon this thread after a long time butting heads with the pak loading systems and coming to an almost similar point, but there’s one specific thing I’m still not managing to achieve properly, which is the AssetRegistry scan.

In a nushell I basically highjacked the ChunkDownloader plugin, I duplicated and renamed it to avoid conflict and then added the extra bits to support loading pak files from external projects. I would have preferred to extend it but sadly it wasn’t made very accesible.

I managed to add most of what I wanted, but I couldn’t figure out how to make the AssetRegistry scan the contents asynchronously.

You mention that AssetRegistry listens to OnContentPathMounted, but from what I’ve managed to understand this only happens at startup. If you want to load the files on demand, you seem to be forced to use ScanPathsSynchronous, and of course this function only works if called from the main thread.

The ChunkDownloader does the mounting in a worker thread, so I thought it would have been super easy to upgrade to what I wanted if I simply added the file system mount and AssetRegistry scan right there and simply report the end of the task when all of it is done and not just the pak mounting, but unfortunately I couldn’t find a way to do this.

Am I missing something?

10

Yeah It is posible, you need to include PakFileUtilities into Build.cs and use ExecuteUnrealPak from PakFileUtilities.cpp. You can extract paks and create paks.

For example I use the following command to create a pak

FString cmd = “-Unrealpack "full path of the pak file to create" -create = "path of txt containing the list of files with its respective mount point of each file"”
ExecuteUnrealPak(*cmd);

And an example of one file described in the txt can be:
“c:\imagen.png” “…/…/…/Game/imagen.png”

11

Hi Sebastian,

Thanks for sharing this detail explanation.
Is it possible to share a example for loading .umap from pak file.
Thanks in advance.

13

I am also facing issue while loading the DLC,

I have a Game Feature plugin called Special which has extra skins which I am storing in primary data assets. When I build a normal project, I disable the plugin as I don’t want that to be included in DLC.

When building DLC I enable the plugin and build DLC, I also make sure version do match between build and DLC.

The problem is after loading the DLC pak, I am unable to find the plugin.
I am following this to load the pak, which is I am able to but not the plugin

14

Thank you! I’ve been trying to achieve this for days now, couldn’t find a solution anywhere else. Found this post from here
This works perfectly.

15

I am having trouble loading assets from .pak files with StaticLoadObject.
The problem seems to be that it’s ONLY possible to manually load assets that have an instance in a map in the .pak file. The problem is not fixed by making a Primary Data Asset reference the needed assets, or by making the assets themselves into Primary Data Assets.

Regardless of the address and structure of the .pak file, and whether I use FAssetData.GetAsset, StaticLoadObject, FindObject, or LoadPackage, ultimately loading assets without an instance in a map is blocked by this message:

LogStreaming: Warning: LoadPackage: SkipPackage: /Game/Content/BP_TestBP (0x62E896F79C30529A) - The package to load does not exist on disk or in the loader

(Obviously message varies depending on filepath)

  • The address/ mount point is not wrong, since map-referenced uassets in the file are loaded successfully.
  • The assets are definitely in the .pak file - I check this with UnrealPak tool and by iterating through the file contents in C++. They have non-zero file size.
  • This is true in multiple different .pak files - it doesn’t matter if I use a .pak file packaged by the game project (so it’s aware of its assets in the Asset Registry), or a file packaged by an empty project; the result is the same.
  • Loading via AssetManager.LoadPrimaryAsset also gives the same error.

I think the most likely explanation is that there’s some wrong setting when I create the .pak files, but I can’t figure out what. Or is it just a requirement of the engine that assets have to be in a level somewhere to be able to be pak’ed + loaded and there’s no way around this?

For context: I’m making a game where players can package and import their own assets and do things like spawn them into a level from a menu window. As well as supporting e.g. paks that contain just a bunch of static meshes and nothing else, I need to be able to create in the “mod SDK”, and then load in the game itself, custom blueprints/ data assets that can define data for individual assets contained in the .pak, such as: UI thumbnails, search tags, categories etc.

16

Hey Robomoo,

As long as your assets are referenced during the cook and part of the pak file you should always be able to load them. I’m not aware of any special settings that could make a difference here, other than the pak file mount point.
My guess is that the package path you are using is wrong in this case, but it’s hard to say without further details.
I’d need the following info to check if everything matches up:

  • the path you use to load the package asset
  • the mount point inside your pak file (UnrealPak.exe -List shows it in older versions of the engine, if you’re on 5.4 or 5.3 you might need to use the .exe from 5.2 )
  • the relative path of the asset inside the pak file
  • your package path mount point for /Game (you can show those with the PackageName.DumpMountPoints console command)

The important point is that pak mount point + asset path in pak file equals package path mount point for /Game + rest of package path used when loading the package.

Are you iterating packages or files in C++?
If the pak file mount point is wrong you might be able to find the .uasset files through the Filesystem API, but you won’t be able to load them as packages.

17

Hi, i have used this tutorial to dynamically load .pak at runtime. It works.
But if i activate “IoStore”, how can i load my assets with .utoc and .ucas ?
Could you update your tutorial for that ?
Does anyone have a solution for that ?

18

In general IOStore should work just the same as pak files.
If you enable the creation of IOStore containers you will always get a pak file and it’s corresponding .utoc/.ucas files.

If you load the pak file in a project that had IOStore enabled when packaging, the engine will detect the utoc/ucas files and load them automatically together with the pak file.

However, some of the more advanced things possible with custom paks (e.g. changing the asset paths or the pak mount point) might not work for IOStore containers.

19

If my .pak loader and my external .pak projects do not use IoStore. It works.

  1. When i active IOStore on the .exe of the pak loader, and i try to load external .pak without IoStore, i have this error:

After:
bIsMounted = PakFileMgr->Mount(*InputPath, PakOrder);
i have:
LogPakFile: Warning: IoStore container “…....\MyPakFiles\MyCars.utoc” not found

I think it’s normal.

  1. When i active IOStore on the .exe of the pak loader, and i try to load external .pak with IoStore, i have this error:

After:
bIsMounted = PakFileMgr->Mount(*InputPath, PakOrder);
i have:

LogIoDispatcher: Display: Reading toc: …....\MyPakFiles\MyCars.utoc
LogIoDispatcher: Display: Toc signature hash: 0000000000000000000000000000000000000000
LogIoDispatcher: Display: Mounting container ‘…....\MyPakFiles\MyCars.utoc’ in location slot 1
LogPakFile: Display: Mounted IoStore container “…....\MyPakFiles\MyCars.utoc”
LogPakFile: Display: Mounted Pak file ‘…....\MyPakFiles\MyCars.pak’, mount point: ‘/’

  1. If i use Loader and .pak without IoStore i have (which works):
    After:
    bIsMounted = PakFileMgr->Mount(*InputPath, PakOrder);
    i have:
    LogPakFile: Display: Mounted Pak file ‘…....\MyPakFiles\MyCars.pak’, mount point: ‘…/…/…/’

Mount point seems to be different. Do i have to change something in my code ?

Here is my code:

`bool MyBFL::MountPakFile(const FString& PakFilePath, const FString& PakMountPoint)
{
int32 PakOrder = 0;
bool bIsMounted = false;

FString InputPath = PakFilePath;
FPaths::MakePlatformFilename(InputPath);

FPakPlatformFile* PakFileMgr = (FPakPlatformFile*)(FPlatformFileManager::Get().FindPlatformFile(TEXT("PakFile")));
if (PakFileMgr == nullptr)
{
    UE_LOG(LogTemp, Log, TEXT("Unable to get PakPlatformFile for pak file (mount): %s"), *(PakFilePath));
    return false;
}

FKeyChain keyChain;
FString EncryptionKeyBase64 = "oIq0gO3Oy1YsMCc0kT1meGVFYeRA5vO9sbPQKhet+Zo="; // My encryption key

TArray<uint8> Key;
bool rsDecode = FBase64::Decode(EncryptionKeyBase64, Key);
check(Key.Num() == sizeof(FAES::FAESKey::Key));
FNamedAESKey NewKey;
NewKey.Name = TEXT("Default");
NewKey.Guid = FGuid();
FMemory::Memcpy(NewKey.Key.Key, &Key[0], sizeof(FAES::FAESKey::Key));
keyChain.GetEncryptionKeys().Add(NewKey.Guid, NewKey);

UE_LOG(LogTemp, Log, TEXT("key decode: %d NewKey.IsValid() = %d"), rsDecode ? 1 : 0, NewKey.IsValid() ? 1 : 0);

KeyChainUtilities::ApplyEncryptionKeys(keyChain);

UE_LOG(LogTemp, Log, TEXT("Try FPakPlatformFile::Mount()"));

bIsMounted = PakFileMgr->Mount(*InputPath, PakOrder);

UE_LOG(LogTemp, Log, TEXT("FPakPlatformFile::Mount() result: bIsMounted = %d"), bIsMounted ? 1 : 0);

return bIsMounted;

}`