our game currently supports PC texture mods, implemented by parsing a structured folder of png files at runtime and calling FImageUtils::ImportFileAsTexture2D to import them as UTexture2D. To improve memory footprint and performance of the system, we want to enable texture streaming also for external loose files. To that end, we designed an offline cooker that produces uasset/uexp/ubulk from png files, the idea being that with a valid FSoftObjectPath and relative TSoftObjectPtr, these external textures can now stream. After cooking the textures offline, we mount the mods folder at runtime and call AssetRegistry.ScanPathsSynchronous. The system works well in editor, but in a packaged game ScanPathsSynchronous can’t find any external uasset. We noticed that the macro EXCLUDE_NONPAK_UE_EXTENSIONS enables to check an array of extensions to be excluded inside FPakPlatformFile::IsNonPakFilenameAllowed. By setting this macro to zero we are able to stream loose textures even in a packaged build. We have two questions about this workflow:
1) Are there any pitfalls in setting EXCLUDE_NONPAK_UE_EXTENSIONS to zero? we are particularly concerned about performance for packaged game, now that every uasset/ubulk/uexp file can skip IsNonPakFilenameAllowed, perhaps some logic that relied on parsing only pak files before is now slowed down by all the additional files to check (uasset, ubulk, etc.);
2) The macro is defined in IPlatformFilePak.cpp, and there is a comment: “Use .Build.cs file to disable this if the game relies on accessing loose files”. We assume the intended place to change the macro is a Target.cs file, not a Build.cs, doing something like this: GlobalDefinitions.Add(“EXCLUDE_NONPAK_UE_EXTENSIONS=0”); However, if we set this in our Game.Target.cs the game won’t compile since the change would force to recompile the engine, but we use a retail (pre-compiled) version of the engine internally. If we set it in UnrealGame.Target.cs instead, on the engine side, the macro doesn’t seem to change even after rebuilding engine and game. We can simply mod the macro itself, but we were wondering what would be the intended way to do such a change;
> 1) Are there any pitfalls in setting EXCLUDE_NONPAK_UE_EXTENSIONS to zero?
I don’t think there are any pitfalls as you imagine them. The main reason for skipping the load of loose files is to prevent modding and tampering with the game.
As that’s your intention, it should be fine to enable. Usually, the pakfiles are checked first, so it should not be possible to override built-in assets with this approach.
Performance-wise, both loading assets from pak files and from loose files is not something we recommend. The newer loading system relies on IOStore containers (.utoc/.ucas) and other approaches will have performance drawbacks or might even be unsupported in the future.
A recent case showed that falling back to pak/loose files can also affect your GC times significantly compared to IOStore containers.
So if loading/streaming performance is important IOStore containers would be the best path forward. This depends mostly on the amount/size of external content you are planning to load, though and you might need to test the actual impact for your game.
If you create IOStore containers in your cooker then be aware that AssetRegistry.ScanPathsSynchronous will not work for them (if you are forcing a rescan it might even forget about IOStore assets from your basegame). Instead you will need to load the AssetRegistry.bin from the pak file that comes along the IOStore containers into your AssetRegistry by calling AppendState(). You can check UAssetManager::OnAssetRegistryAvailableAfterInitialization() for an example.
> 2) […] However, if we set this in our Game.Target.cs the game won’t compile since the change would force to recompile the engine, but we use a retail (pre-compiled) version of the engine internally. If we set it in UnrealGame.Target.cs instead, on the engine side, the macro doesn’t seem to change even after rebuilding engine and game. We can simply mod the macro itself, but we were wondering what would be the intended way to do such a change;
Changes to these macros (and the engine code it causes) do require a custom build of the engine, so I’m afraid this is not possible with a pre-built/launcher version of the Engine. You would need to create your own custom builds internally if you need the change for all your engine users.
thanks for the quick answer. It seems safe to set the macro to zero for what we want to achieve, but we will keep an eye for GC spikes! As for IOStore containers, we know it’s the intended way to load assets but our offline cooker doesn’t use editor code, and it was already a nightmare to produce fully functioning uasset/ubulk/uexp from png. We’ll spare the extra development time if changing a macro can avoid it. Thanks for the support!