Load assets from mounted Pak files

I’ve tried the entire day to load assets from Pak files after mounting them.

Pak file is cooked and I’m running the test code on the same platform in a packaged build.

I think my main problem is figuring out what path to actually use. Been trying all kind of different paths and combinations.

Snippet:



void LoadPak()
{
    FString PakLocation = TEXT("D:\\MilitaryVOL1-WindowsNoEditor.pak");

    if (FCoreDelegates::OnMountPak.Execute(PakLocation, INDEX_NONE, nullptr))
    {
        FStreamableManager StreamableManager;
        FString FileWithPath = "/MilitaryVOL1/Content/Military_VOL1_Tents/Meshes/SM_Tent_01a";
        FSoftObjectPath SoftRef(*FileWithPath);
        UObject *LoadObj = StreamableManager.LoadSynchronous(SoftRef);
        if (!LoadObj)
        {
            // Not ok
        }
        else
        {
            // ok, we could cast now to some type
        }
    }
}


Full code: class FPakFileVisitor : public IPlatformFile::FDirectoryVisitor{public: vi - Pastebin.com

I’ve also read ALL topics about loading pak files but I’m still unable to load any assets from it.

Does anyone have a working pak loader? At this point I’m more than willing to pay for it.


  
 if (FCoreDelegates::OnMountPak.Execute(PakLocation, INDEX_NONE, nullptr))

Delegates should always be called with ExecuteIfBound(…) if you didn’t check the binding earlier with IsBound().


FString FileWithPath = "/MilitaryVOL1/Content/Military_VOL1_Tents/Meshes/SM_Tent_01a";  FSoftObjectPath SoftRef(*FileWithPath);

Seem weired. Asset pathes look like this /Game/NewWidgetBlueprint.NewWidgetBlueprint (the **Content **folder was called **Game **back some time. Its still called Game in the bowels) Also maybe need to make the AssetRegistry aware of the new data, depending on how the StreamableManager works.


IAssetRegistry::ScanPathsSynchronous

@Rumbleball
I’ve added the full source code. class FPakFileVisitor : public IPlatformFile::FDirectoryVisitor{public: vi - Pastebin.com

/Game holds all assets from the current projects pak file right? So let’s say you have a cooked.pak file and load it in another packaged project. What would be the correct path to get to the new assets?

ScanPathsSynchronous does not help I think because when you copy another .pak file to WindowsNoEditor\PROJECT_NAME\Content\Paks it get’s auto-mounted by the engine. I’ve tested that as well.

The if you package from a different project, you have the problem of the MountPath that is within the package. The mount path contains the project name. You need to modify that path to be the name of the project where you mount it.

Output from UnrealPak.exe


C:\Users\vr3> "C:\Program Files\Epic Games\UE_4.19\Engine\Binaries\Win64\UnrealPak.exe" "C:\Users\vr3\Documents\PackagingDir\Mods\WindowsNoEditor\MyFirstModTest\Plugins\FistModPlugin\Content\Paks\WindowsNoEditor\FistModPlugin.pak" -list
LogPakFile: Display: Using command line for crypto configuration
LogPakFile: Display: Mount point ../../../MyFirstModTest/
LogPakFile: Display: "CookedAssetRegistry.json" offset: 0, size: 24560 bytes, sha1: 0CB9C512B052215EA6A82B8685A727B333AEF345.
LogPakFile: Display: "CookedIniVersion.txt" offset: 26624, size: 48481 bytes, sha1: E45F6515C15E4A896DC8B8D65416E3FEC8B5EBD7.
LogPakFile: Display: "Plugins/FistModPlugin/AssetRegistry.bin" offset: 75776, size: 21474 bytes, sha1: 25209C5DB806E659EC39F8BD6FF8D8E80F03D71B.
LogPakFile: Display: "Plugins/FistModPlugin/DevelopmentAssetRegistry.bin" offset: 98304, size: 35984 bytes, sha1: A0594AB0E5A56D9EEE56954351F527D484FE81AA.
LogPakFile: Display: 4 files (130499 bytes), (130499 filtered bytes).
LogPakFile: Display: Unreal pak executed in 0.011382 seconds


LogPakFile: Display: Mount point ../../../**MyFirstModTest**/

is the project name in the mount path. The path can be longer, but that should be the minimum there is.

You need to get the mount point and modify it. You need to mount manually this way.

Some code snippets that should help you out.

It took me a month to get the stuff working for us, so keep it easy.

Get the PakPlatformFile



    if (pakPlatformFile == nullptr)
    {
        // Search for an existing pakPlatformFile
        IPlatformFile* existingPakPlatformFile = FPlatformFileManager::Get().FindPlatformFile(TEXT("PakFile"));

        if (existingPakPlatformFile)
        {
            pakPlatformFile = static_cast<FPakPlatformFile*>(existingPakPlatformFile);
            CM_LOG(Log, "Using existing PakPlatformFile");
        }
        else
        {
            // When mounting a .pak file, we need to store a references to the .pak file in the platform file system.
            // To do this, we need a FPakPlatformFile that allows us to mount our .pak and keeps the information about the package.
            pakPlatformFile = new FPakPlatformFile();
            // But we don't want the engine to lose information of other packages
            // (and what ever other stuff), thus get the current PlatformFile
            IPlatformFile& platformFile = FPlatformFileManager::Get().GetPlatformFile();
            // and place it into our new PlatformFile (PlatformFile-Chain).
            if (!pakPlatformFile->Initialize(&platformFile, TEXT("")))
            {
                CM_LOG(Fatal, "Failed to initialize PakPlatformFile");
            }
            // Not sure about this, see: https://answers.unrealengine.com/questions/574388/how-to-mount-a-pak-file-at-game-content-directory.html
            pakPlatformFile->InitializeNewAsyncIO();
            // Now give the new PlatformFile to the engine.
            FPlatformFileManager::Get().SetPlatformFile(*pakPlatformFile);

            CM_LOG(Log, "Using our own PakPlatformFile");
        }
    }

Get the mount point of the .pak file



// Get the mount point
FPakFile pak(pakPlatformFile->GetLowerLevel(), *file, bSigned);
pak.GetMountPoint();


Now modify the mount point retrieved.

Mount the .pak file manually



pakPlatformFile->Mount(*file, readOrder, *mountPointFinal)


1 Like

@Rumbleball
Thanks for that! I’m still trying to figure how to set the mount point in the pak file.

After mounting the pak. What path do you use for the new assets? Do you use “LoadSynchronous” or “StaticLoadObject”?

Thanks!

You don’t set the mount point in the pak, you give the modified mount point to the



 pakPlatformFile->Mount(*file, readOrder, *mountPointFinal) 

function.

We update the asset registry



FAssetRegistryModule& assetRegistryModule = FModuleManager::LoadModuleChecked<FAssetRegistryModule>(TEXT("AssetRegistry"));
IAssetRegistry& assetRegistry = assetRegistryModule.Get();
assetRegistry.ScanPathsSynchronous({ properContentDir }, true);


and retrieve the assets with UObjectLibrary



void UContentMountLibrary::GetAssetsInDirectory(const TSubclassOf<UObject> assetClass, const FString path, const bool bRecursive, TArray<UObject*>& assets)
{
    // Instead of loading the assets directly via the library, we only get the AssetData
    // and load the objects from those.
    // Reason: The ObjectLibrary seems to load objects (when we want the objects) that it shouldn't
    // which takes much longer than loading the AssetData.

    TArray<FAssetData> assetData;
    TSharedPtr<UObjectLibrary> objectLibrary = MakeShareable(UObjectLibrary::CreateLibrary(assetClass.Get(), false, true));
    objectLibrary->bRecursivePaths = bRecursive;
    objectLibrary->LoadAssetDataFromPath(*FContentMountUtility::MakeProperContentDir(path));
    objectLibrary->GetAssetDataList(assetData);

    assets.Reset(assetData.Num());

    for(FAssetData& data : assetData)
    {
        assets.Add(data.GetAsset());
    }
}


@**Rumbleball **

I too am interested in this,
Does the “ScanPathsSyncronous” end up rescanning the whole games content? or is “properContentDir” not the games primary content dir that FPaths::GameContentDir() would get you.

The scan paths sync also is giving me a error
LongPackageNameToFilename failed to convert ‘D:/Game/Releases/WindowsNoEditor/Helios/Content/’. Path does not map to any roots.

we have lots of code and functions mapping around that stuff, so I just posted some snipps.

A content path looks like /Game/… (Game is the old name for Content). You should also be able to scan /Game

Interesting, didnt know the /game/ was a legitimate folder, thought it was people just refering to their game folder, ie, MyGame/content/

Made a method for rebasing the mountpoint of the PAK while creating it so the asset paths match the /game/content/… path correctly. But for the life of me i can get them to load at run time.

Im trying it with “AssetLoader.LoadSynchronous(TargetAssets*)”
And your object library method too.

Yah… thats some annoying stuff. I gues Unreal did that change cause they want to stop beeing a pure game engine (architecture, film, …) and thus it was renamed to Content.
In the bowels of the engine it still says Game, though.

You also must be carefull where you work and what function you call.
When you want to access files on disk, you need Content, as that what the folder name is…
The internal pathing for assets start with /Game.

When you right click an asset and select “Copy Reference”, the asset path is copied to the clipboard. The reference for a material looks like:
Material’/Game/M_RectecBG.M_RectecBG’
For a Blueprint:
Blueprint’/Game/NewBlueprint.NewBlueprint’
Note that the path is prefixed with **AssetType’. **Not sure whats that for exactly but you can also skip it. When loading an asset via C++ you can take the whole reference or skip the type prefix. Both will work, but maybe you get more performance when keeping the type prefix, I dunno.

It seems to work, read the log. It tryed to load your asset, but the asset has a dependency (“Error: Found 1 dependent packages…”) that is not existing (“Error: /Game/Content/Avatars/New/BP_Bader”). Make sure** /Game/Content/Avatars/New/BP_Bader **is part of the pak file and is also known to the asset registry.

Btw: Make sure the project you package from does have the **EXACT **same settings (at least for rendering). Otherwise you will get problems with your assets (materials).

I’m working on a plugin for the marketplace that loads .pak files via Blueprints. It also includes docs on how to package desired content to a .pak file without any dependencies and breaking references.

@BlueMountainsIO
Take my money!, any possibility of a early version, this is vital to our project.

I have been trying to add mod support into UE4 as a plugin, without requiring a custom-built Mod Editor. I found this ModSkeleton example from 4.15 that was updated to newer build rules:

It successfully mounts the pak as DLC and finds added AssetRegistry.bin file, but due to engine changes since then, UE4 crashes on load in a packaged build:

“AsyncLoading.cpp [Line: 6963] LoadPackageAsync failed to begin to load a package because the supplied package name was neither a valid long package name nor a filename of a map within a content folder”

EDIT: Fixed crash by changing the “Loading Phase” in the plugin from “Default” to “PreEarlyLoadingScreen”
(Alternatively, can just disable “Event Driven Loader” in project settings instead)
Pak mounts in expected location but is still unable to load MOD_SKELETON file…

Another development is mod.io, which seems to have a plugin method working for UE4 that uses authentication/deployment of mods:
mod.io API v1
Example project:
GitHub - modio/modio-ue4-example-legacy: Very simple project using the mod.io UE4 plugin.

Then comes the problem of “Packaging Profiles” to build a mod as DLC. I am not sure if that is solved there-in, or if the profiles have to be released also to cook/build a mod plugin from somewhere else…If anyone can offer help it is much appreciated,

You shouldn’t need a custom built editor version for this, what im doing currently sounds like the same idea as bluemountains, gathering asset dependencies and references, generating a response list and running the pak from cmd.

I Had the same issue earlier with the loadPackageAsync, resolved it by manually creating the pak and rebasing the contents paths to /game/content/ instead of their absolute paths that they initially had, and using “FPackageName::RegisterMountPoint(*MountPoint, *AbsoluteContentFolderPath)” + “assetRegistry.AddPath(MountPoint)”
though im unsure if i need the addpath one, still needs further work.
There seem to be weird things with the Streamer system, where it says it cant find a asset at


Couldn't find file for package /Game/Avatars/New/Badger requested by async loading code. 

but then says it found a dependencies at


LogStreaming: Error: Found 1 dependent packages...
LogStreaming: Error: /Game/Content/Avatars/New/BP_Bader

But you notice with the /Content/ compared to the last that just had /Game/Avatars/.

I can now get the assets showing up the in the runtime asset registry, still those weird path issues to sort…

As for packaging profiles, ive supplied a stripped down duplicate of my project for mod creators without the source to produce mods. But the way i intend to do it dosnt use the launch/packaging system.

It all really depends on how much you expect your mod creators to have to do, creating a simpler system for the end user or using the built in methods.

We never tried assets that have a dependency yet.
What code you use to generate that error?
Is **BP_Bader **part of the .pak file and where should it be located?
Do you have a …/ProjectName/Content/Content/Avatars folder?

Our modding approach is to not give the Project to the modder, Only a ModProject that serves as modkit with a package button, as our Project relies on Marketplace plugins that we are not allowed to share. We have a folder …/ProjectName/Content/SharedWithModKit that contains some stuff that is not dependent on the marketplace plugins, that the modder can use. That comes with lots of limitations to the modder, which we don’t want but don’t know how to solve them either. As of now, our target for a modder is to provide Materials/StaticMeshes/Maps/ActorBlueprints that can be loaded in the our project (we do not have a game project, but a creative application).

How would modifying an asset work at all? If a Blueprint /Game/Characters/BP_Car is provided in the project and a mod does also provide that asset, is it about load order (last loaded is relevant only)? Or will it result in a conflict?

BP_Badger is in the Pak and is a character blueprint, its a child class of the main character class thats packaged in the game.
Its located in /Game/Content/Avatars/New/ Along with all the other things that relate to that character
The Asset thats just “badger” is its skeletal mesh, so i presume its saying when it tries to load the character that it cant find the mesh, but the streaming log says its looking in the wrong place, even though i tell it to load from the specific path that it should be in. The bp_badger and the Badger mesh along with the other pak’s, are all in the asset registry when i do a all asset dump.
Ill try messing with the asset registry to try and get the FassetData from that instead, see if i can get around that streaming issue.

The idea modding for our project is that the parent classes that people can build off, are all contained in the main project as c++ classes and example BP ones.

As for load order and such, im not sure yet, this is still in the early stages of getting it working properly, maybe there is a way of generating a unique bp id with the pak file that only associates its contents with it so it wont conflict with others that may have the same name.

Here you go! Pak Loader Plugin in Code Plugins - UE Marketplace

Would it be easy to create the plugin in a way that the path is not /TestDLC/Stuff but /Mods/TestDLC/Suff ?

Here is also a few links that could work nicely alongside the plugin above:

For modding, someone updated the ModSkeleton plugin, which doesn’t require a custom engine build or the original project files, just drop .pak in folder:

Mod.io has also released a plugin and example project for using their mod system, which uses authentication but only hosts mods:
mod.io API v1

Here is an html .pak downloader for in-game downloading and loading .paks, just need to update the build.cs for later UE4 versions:
https://github.com/emrahgunduz/unrea…/master/Caller

This hole mod.io thing is just like the Steam workshop. It handles downloads but the entire modding thing is up to you.