Loading assets from a pak file

Development Programming & Scripting C++
1

I have been trying to load an asset from a pak file and after trying a number of different approaches cannot determine how to load an asset from it.
I can successfully mount a package and have managed to get a PakEntry by seaching the pakplatform (which also returns the PakFile it is located in) so I could read files manually ok.
However I cannot find any way to register the .uasset files in the pak with the assetregistry so that I can load them using something like StaticLoadObject.
An example of what I’m trying to do currently:



	IPlatformFile& InnerPlatform = FPlatformFileManager::Get().GetPlatformFile();
	FPakPlatformFile* PakPlatform = new FPakPlatformFile();

	PakPlatform->Initialize(&InnerPlatform, TEXT(""));

	FPlatformFileManager::Get().SetPlatformFile(*PakPlatform);
	const FString pakFilename = TEXT("C:/.../Content/new.pak");
	FString mountPoint(*FPaths::EngineContentDir());
	
	if(PakPlatform->Mount(*pakFilename, 0, *mountPoint))
	{ }
	// RESULT - succeeds
	
	// search for a file from the pak
	FPakFile *assetPakFile = nullptr;
	const FPakEntry *assetFile = PakPlatform->FindFileInPakFiles(*(mountPoint + TEXT("/mymesh.uasset")), &assetPakFile);
	// RESULT - assetFile & assetPakFile both valid

	// enumerate available static meshes
	FAssetRegistryModule& AssetRegistryModule = FModuleManager::LoadModuleChecked<FAssetRegistryModule>(AssetRegistryConstants::ModuleName);
	IAssetRegistry &registry = AssetRegistryModule.Get();
	registry.GetAssetsByClass("StaticMesh", assets, true);
	// RESULT - no static meshes from the pak are visible
	
	UStaticMesh* obj = Cast<UStaticMesh>(StaticLoadObject(UStaticMesh::StaticClass(), nullptr, *(mountPoint + TEXT("mymesh.mymesh"))));
	// RESULT - obj is nullptr

As I say I have tried many different techniques and trawled much of the code and still failed miserably, this must be a relatively common process though as I assume it’s the way that DLC is handled?
Any suggestions greatly appreciated, I have scanned through multiple related forum posts and none of them seem to explain fully how this is done.

2

I have a similar question. Bump

3

OK, so with a bit of help I’ve managed to get a static mesh to load, problems were

  1. asset path needs to be /Engine/mymesh.mymesh.
  2. I had the pak loading in my Uobject constructor which gets called during editor startup when stuff doesn’t quite seem to be working in the same way
  3. You have to use uncooked / cooked for editor / game modes

So code now looks like:



	//// load package
	IPlatformFile& InnerPlatform = FPlatformFileManager::Get().GetPlatformFile();
	FPakPlatformFile* PakPlatform = new FPakPlatformFile();
	PakPlatform->Initialize(&InnerPlatform, TEXT(""));
	FPlatformFileManager::Get().SetPlatformFile(*PakPlatform);
	const FString pakFilename = TEXT("<PATH>/new.pak");
	FString mountPoint(*FPaths::EngineContentDir());

	FPakFile pakFile(*pakFilename, false);
	if(!pakFile.IsValid())
	{
		GEngine->AddOnScreenDebugMessage(-1, 5.f, FColor::Yellow, "Invalid pak file: " + pakFilename);
		return;
	}

	pakFile.SetMountPoint(*mountPoint);
	const int32 PakOrder = 0;
	if(!PakPlatform->Mount(*pakFilename, PakOrder, *mountPoint))
	{
		GEngine->AddOnScreenDebugMessage(-1, 5.f, FColor::Yellow, "Failed to mount pak file: " + pakFilename);
		return;
	}

	// load the mesh asset
	FStreamableManager AssetLoader;
	FStringAssetReference AssetRef(TEXT("/Engine/SM_mymodel.SM_mymodel"));
	mMeshAsset = Cast< UStaticMesh>(AssetLoader.SynchronousLoad(AssetRef));
	mMesh->SetStaticMesh(mMeshAsset.Get());

I am now at the same point that others seemed to get to with asset references all being wrong because you can’t mount to the correct location (e.g. “LoadErrors:Error: Error /Engine/SM_Bush : Can’t find file for asset. /Game/M_Bush”).

Can anybody describe how either the asset references can be repathed after loading the pak or the pak can be mounted to it’s original path so that the references work?

4

Bump, I need this too. I would really like to know if it’s a way to do not lose references when loading a .pak file and mounting assets it on it’s original path.

5

Bumping this too. I’m in the same place.

6

Bump! I’m currently looking into this as well. If I find anything I’ll report it here.

7

So I was able to get downloading assets and replacing loaded assets working with some direction from epic. Here’s what I did:

Make a plugin that is responsible for downloading. I assume you can download the pak to a folder that you can access to mount it. One caveat here is that everywhere on the web people we’re doing what you did above, getting and setting the platform file. However, if you are using a pak file for your in-bundle content ( Setting UsePakFile=true in packaging settings ), you will likely run into a ton of issues using that method ( or at least I did ).

So what I ended up doing was using FCoreDelegate::OnMountPak to actually mount the paks. That way, you don’t need to grab the pak platform file and if you are using UsePakFile, the platform file will be setup the right way by the engine. If you are not using UsePakFile, you should look at how epic sets up the platform file in the LaunchEngineLoop.cpp and try and emulate that. Seems like they may do a few things differently but I haven’t looked into it that far since we are using UsePakFile.

Make sure the mounting point in your pak file matches the relative path for your game content directory. For us, that is: “…/…/…/GAME_NAME/Content”. I ended up having to modify UnrealPak.cpp to override the mounting point because it tries to set the mounting point as the common base directory for all files in the pak but if you are only paking up files in one directory then you lose the path to the file, which is needed if you’re replacing files in your bundle.

After mounting the pak, I go through all of the files in the pak and I build an asset reference string ( “/Game/PATH/TO/ASSET/AssetPackage.AssetName” ) and Create a FStringAssetReference for each. I then reconcile the object and if it reconciles to an actual uobject, I know that this asset is already loaded in the game and I need to replace it. So I get the package of the asset by calling “GetOutermost()” on it and I rename the package by calling Package->Rename( NewName, Package->GetOuter() ). This changes the name of the package so when I async load the newly downloaded version, the engine won’t just find the currently loaded version. I do this for all assets in the new pak that have matching loaded assets. Then I use a streamable manager to async load all of them.

When the async load finishes, I then have to replace references to already loaded ones. I won’t go into great detail here but I will point you at the FArchiveReplaceObjectRef object. It takes in an object to search for references and a map of old refs to new refs to replace. So I use the FObjectIterator to iterate all loaded objects, passing each in as the search object. I build the replacement map once having looped over all of my newly loaded assets.

Finally, in order to make sure your patched assets are loaded over the bundle assets on the next launch of your game, you need to mount the downloaded paks at a higher priority than the bundle pak ( which is priority 4 ) at the launch of your game, before Class Default Objects are created. I do this in the StartupModule function of my pak downloader module. You can’t use a UObject derived class to do it because there are no CDOs yet, so creating a class this early throws some errors at run time. So I have a non-uobject class that is responsible for mounting the paks.

That’s it! If there’s any specific questions let me know. I cannot share source code unfortunately but I’m happy to answer questions or any confusion this post may contain.

8

Hello alk3ovation!

Thank you for this additional info. I did try using OnMountPak as well as the method above but I had even less luck with that. Using the Set Platform File method I was able to load a level into a game, but I would lose all mesh textures in the level. Using OnMountPak I was unable to even load the level even though Execute() was returning true. I think that my mount point was correct, but it’s been a while since I looked at that code. I’ve since put it on back burner and other things are taking my time, but I do expect to revisit it eventually. Thanks again for the additional info. You may get a PM from me down the road when I can get back on this.

9

No worries. I’d venture a guess if the pak was mounting correctly that the mount point was incorrect, probably because of the way UnrealPak works. Good luck!

10

How did you make the code compile?

I have added forward declaration of the class FPakPlatformFile, I also modified the two build.cs files to add module dependencies as
PrivateDependencyModuleNames.AddRange(new string] { “PakFile”, “StreamingFile” });

but I still cannot compile through. The error says ‘FPakPlatformFile’ class has no constructors. I understand that simply forward declaring FPakPlatformFile doesn’t make its constructor available, so did I miss any header files?

11

Hello Xing,

You need these header files:
#include “Net/UnrealNetwork.h”
#include “Runtime/PakFile/Public/IPlatformFilePak.h”
#include “Runtime/Engine/Classes/Kismet/GameplayStatics.h”

And you need these additions to build.cs:
In public dependencies add “Sockets”, “NetworkFile”, “Networking”
In private dependencies add “PakFile”, “StreamingFile”, “NetworkFile”

You may not need all of those but those are the ones I’ve got and mine is compiling… just not working. :slight_smile:

12

Hello alk30vation!

So here’s a couple questions to get this started.

  1. When using OnMountPak I always have to package the project before I can test it. When I run as standalone, OnMountPak.IsBound always returns false. Is there a way around this? It would be much easier to test if I didn’t have to package everytime I change code.

  2. In your post regarding the paragraph that starts “Make sure the mounting point”, can you please spell this process out for me in more detail? :slight_smile: If I was using the old method I would call PakFile.SetMountPoint(), but when using the OnMoutPak method I do not see a way to set the mount point. Should I create a FPakFile for this or is there another way to do this?

My process general process is outlined here:

Thanks!

13

anstontree,

  1. From what I have seen, OnMountPak will not be bound unless you use UsePakFile for packaging because that sets a command line parameter that tells the engine to use the PlatformFilePak, which calls Initialize on IPakPlatformFile and that binds those delegates. I actually added a function to my plugin that checks the current platform file to see if its a IPlatformFilePak. If it is, it does nothing, if it’s not, I basically fall back to creating my own, except I copied code from ConditionallyCreateFileWrapper in LaunchEngineLoop.cpp. I’m currently working on seeing if I can get this to work in the editor. However, I don’t have high hopes for that. The problem is that the editor recognizes all of these changes as modifying the content on disk. So for example, after changing the package name of a loaded asset, the editor thinks you may want to now save that “new” asset. Also, when mounting a pak early so that the engine finds your mounted pak before the base content, the editor can no longer modify the base version of the asset because its looking at the version in your downloaded pak file. I’ll post back here if I figure that out, but so long as it works on mobile I can accept it not working in editor.

  2. For this one, you need to make sure that your mounting point is correct when you create the pak file. If you look at the log when the engine creates a pak file for your base content, it creates a file that looks like this:

“/Full/Path/On/Your/Hard/Drive/GameName/Content/Subdirectory/ToAsset.uasset” “…/…/…/GameName/Content/Subdirectory/ToAsset.uasset”

The UnrealPak tool will take all of the destination files and try to find the common base path of them and set that as the mounting point. I just realized this is actually fine so long as all of your destinations are relative paths to the content directory. I had added a mount point override to the UnrealPak tool but I don’t think I need that anymore. Anyways, if you create a pak file by just passing files in off the command line instead of doing a source to dest file like the above, you will get an absolute path the common directory as your mount point, which is likely why people feel they need to set the mount point themselves. So when you create the pak, use -create=Path/To/SourceDestFile as an argument to UnrealPak.

I hope that’s not confusing, if it is, let me know and I’ll try and clarify.

14

allc3ovation,

Your first answer above gave me an idea. If the IPakPlatformFile.Initialize binds the OnMountPak then I figured why not just call it manually? So I added the following 3 lines of code before I access OnMountPak:


IPlatformFile& InnerPlatform = FPlatformFileManager::Get().GetPlatformFile();
FPakPlatformFile* PakPlatform = new FPakPlatformFile();
PakPlatform->Initialize(&InnerPlatform, TEXT(""));

And it seems to work! As expected I cannot run it in PIE, but when I run Standalone I at least am getting true on IsBound() and Execute(). Hopefully this will save me some time debugging. Thanks!

Now for the pak file itself I have to admit that my knowledge is severely lacking. I’m not quite sure I understand how these paths are used or what exactly I’m looking at when I see one.

I am creating my pak a little differently. I created a brand new project from the TopDownTemplate and then just removed everything except the map and the geometry files. Nuked the PlayerController, GameMode, Character, etc, and then packaged the project for Win64. Once that is done I just copy the resulting pak file (which should be cooked and ready) to whatever folder my main app is running in. Does this make sense?

Since both my main app (built from ThirdPersonTemplate), and the app I’m building the pak from are built from base Unreal templates, they already have common path structures in the pak (I think I understand this but maybe not). It seems like the map would not load at all in my old method if this was not correct?

15

The first thing you do will definitely mount the paks but since you aren’t calling SetPlatformFile with the FPakPlatformFile, I don’t think the file system will actually check the mounted paks when trying to load a file.

For the pak file, if you are doing it that way and the projects are named the same and are in the same location relative to the engine, then it should just work. We are using a customs ource version of the engine and it is contained in the same folder as our project, which i believe affects the relative pathing to your content directory. If you print FPaths::GamContentDir() in both projects and they are the same, then it should work. If the projects aren’t named the same, your pak will end up with a mounting point of “…/…/…/CONTENT_PROJ/Content” while your actual project will look for assets at “…/…/…/ACTUAL_PROJECT/Content” ( these are example paths from our project, your paths may be slightly different ). The engine tries to abstract paths away by replacing pathing with roots. So in the editor you see “/Game/Subfolder/Asset.uasset”, but when you load that asset, the engine actually replaces “/Game” with “/RelativePathTo/ContentFolder”. And the pak system basically gets the first chance at finding a file when it’s enabled. And the first thing it does is check to see if the file the engine is trying to load starts with the Mounting Point.

So your actual project would try and load an asset and the pak system would say does “/…/…/…/ACTUAL_PROJECT/Content” match the mounting path (…/…/…/CONTENT_PROJ/Content) of your downloaded pak? No it doesn’t so don’t even look for the file in this pak.

16

My projects do have the same name and are both in same folder relative to the engine. Well they do now at least. I was hoping that was going to be the fix but unfortunately that did not resolve my issue. I agree, based on your info and on what little I’ve found on the forums, it should just work.

I wonder if it is related to maps. In your implementation are you loading in an entire map from a pak or are you using the paks for other types of assets?

17

Using it for only uassets at the moment and not umaps. If you breakpoint in IPlatformFilePak in OpenRead when you go to load the map, you should be able to tell if it’s at least finding the map in the pak file. If it is, then the problem is in the streaming/replace references part. I don’t think my code handles umaps right now so I can’t really test it out and we’re finishing up a milestone, so I can’t spend the time right now to work on it. But that’s a feature I think we will need so I’ll definitely be looking at it in the near future.

18

Mark!!!

19

Hey everyone,

I am stuck at the same point.

I tried both methods (FCoreDelegates::OnMountPak and FPakPlatformFile) with no success.

My pak file is created using the custom launch profile with the DLC option checked (+ UsePakFile both in the general packaging options and in the custom launch profile).

I can mount it (or can i ??), change its mount point, even list its content using FPakFile::FindFilesAtPath(), but i can’t access any of the content, neither with a system related function nor through StaticLoadObject().

For instance IFileManager::Get().FindFilesRecursive waits 50 minutes (yes, i waited this long :smiley: ) before… doing nothing (but at least this very long time shows something is happening: the function sees or tries to see something).

I have been using IFileManager::Get().FindFilesRecursive for several months to search the main pak file so i know it can work.
Talking of which, the packaging logs for the main pak and the dlc pak seem to be the same (as far as path are concerned) so in principle the dlc pak should be functionnal.

But i can do nothing with the DLC pak file except reading it with FindFilesAtPath, and setting/getting its mountpoint.

I have read (Add Mod-support to Unreal Engine 4 Projects - Tom Looman) that all the pak files put in Content/Paks were automatically mounted at start, but that doesn’t seem to work in my case.

FCoreDelegates::OnMountPak apparently does all the work behind the curtains for some people, but not in my case.

As i can access the pak through FPakFile functions but not file manager functions, i suspect a mount point problem. I tried various mount points (using FPakFile::SetMountPoint() and FPakPlatformFile::Mount()), no success.
By various mount points i mean: Content, Content/Paks, …/…/…/MyGame/Content, etc. each time in relative and absolute path.
I test as much as i can but as i have no solid point to cling on so it’s tedious: not sure about my paths, not sure about the method, not sure about my pak file, not sure about the functions i use, i understand basically nothing.
I feel like a blind cat trying to catch a fly, it would be nice to have some sort of anchor here.

Did anyone understand what exactly SetMountPoint and Mount are doing ? Are they the equivalent of mounting a file system on unix ? Why use both ?
Besides all the functions returning true, is there a way to check if the Pak is correctly mounted and where ?

Sorry bout the wall of infos, just trying to show that i tried to do my homework before posting here :slight_smile:
I have already spent many hours on this, it can’t be that complicated so it’s obvious i’m missing something big or doing something stupid. Any help welcome.

As always, if i understand something or come up with some working code, i’ll post the full solution here.

Cheers

20

Hello ,

All I can say is “welcome to the club” :). I’ve not been able to advance any further than my posts above indicate and I’ve put this part of my app on back burner for now. I do notice that on the Trello board under Platforms there is an entry for “Improved game patching support” with march and april tags. I can only hope that this will include easier ways to load external pak file assets. I know that UE4 has mod support on their todo list but no idea on when it will be fully fleshed out.