Question about PAK files, chunking

Hello,

We have a question regarding PAK files, their chunking, and how updates are downloaded.

We already have a version of the game installed on a user’s device. Is it possible for a new version’s PAK file to be downloaded partially, i.e., only the parts that have changed, rather than the entire file?

For example, we have three versions of the game: A, B, and C, where A is the oldest and C is the newest.

Is it possible to update from version B to C by downloading only the delta in the PAK file (not the whole file)?

Is it also possible to update directly from version A to C with only the changed parts of the PAK file being downloaded?

More generally, is it possible to update any older version of the game to the latest version by downloading only the differences in the PAK files?

We want to minimize the amount of data mobile users need to download when updating.

Is such functionality supported in UE4, and if so, how can we implement it?

Thank you in advance!

Hi,

there’s a few different ways to go about patching, each with pros and cons.

I’ll be answering first how the built in patch feature works, but in general it is probably better to use an external solution, since the built-in patching has some limitations.

> Is it possible for a new version’s PAK file to be downloaded partially, i.e., only the parts that have changed, rather than the entire file?

Partially. The built-in patching system allows you to create a “Patch Pak” that will only contain the changed assets.

This new pak would need to be dropped next to the existing ones and the assets in the patch pak would override the ones in the base pak.

This approach is somewhat limited in that it can only replace full assets and doesn’t support any binary delta patching (i.e. changing one pixel in a texture asset will include the full texture in the patch).

This also requires that you use extra parameters when creating the original release, if your release is just the regular output of the package process then you are likely missing the metadata about your release that is necessary to create patches.

> Is it also possible to update directly from version A to C with only the changed parts of the PAK file being downloaded?

You can create a full patch C based on the base release A. This would require deleting the patch paks from B and adding the newly created ones for C. As an alternative it is possible to create Patch C based on patch B, in which case you would need the patch paks from b and c to update A.

> More generally, is it possible to update any older version of the game to the latest version by downloading only the differences in the PAK files?

If your game also contains C++ code then likely not. If your game logic is blueprints only and the engine version does not change then it should be possible to replace only the pak files.

The built-in patching would require you to download additional pak files and keep the original ones.

In general the built-in patching is a fairly simple implementation, it can create patch paks but it’s based on per asset changes and is not intended to work over a long time. If you’re doing many updates to a game at some point you would want to create a new base release with new pak files to get rid of the patch paks (which will add up in install size if certain assets are replaced multiple times).

The docs for your built-in patching can be found here.

Overall we do recommend external binary patching solutions over the built-in patch paks.

A binary patcher can generate a binary delta patch that only replaces the parts of a pak file that have actually changed.

There is no built-in tool that does this, usually every platform has it’s own patching approach.

I.e. Steam or the BuildPatchTool for EGS will automatically create delta patches between two full releases, without requiring explicit support from the engine.

To make those binary patches effective it is important to minimize the changes in the source pak files and keeping the order of files inside the pak consistent.

Our packaging docs have a section on “ordering your pak file”.

In many cases it also helps to not compress the pak file and let the platform handle compression for the generated patches/downloads.

This will result in larger install sizes so you might still want to use some compression for the data on mobile at the expense of larger patches.

Hopefully this gives you a starting point for choosing a direction, I’m happy to go into more detail on the built in features if there’s further questions.

For external tools we unfortunately don’t have much advice to give, you would need to investigate the options that the Android/Apple tools provide.

Kind Regards,

Sebastian

Apologies for the wait!

> Will this pak file remain binary identical across builds during that period, or will it still vary?

If the engine version does not change and the object itself does not change then we aim to make sure that the cook is deterministic.

Repeated cooks of the same object in the same engine version should produce identical output.

However, since this was never enforced until very recently. With the experimental iterative cook introduced in 5.6 it was necessary to enforce and guarantee that assets cook deterministically, so we’ve added tests and automated verification runs to make sure that is the case for all Engine types.

However, for older Engine versions it did happen quite often that some of the Engine classes were not deterministic. Which classes are and are not might vary per version and we do not have a list on the status in each engine version unfortunately.

This is the main cause for in-determinism in pak files.

> If it does vary, is there a way to ensure that it remains unchanged as long as the actual content doesn’t change?

Unfortunately not. You could use chunking to separate your content into multiple pak files, which would cause only a smaller part of your data to be affected, but this depends a lot on the actual assets causing the indeterminism.

Apart from that any changes in how an assets is serialized/cooked due to Engine changes will also change the binary representation, so between Engine versions compatibility is usually not guaranteed and we always recommend to recook all content between engine changes.

> As far as I understand, the number of pak files may affect game loading times, particularly asset lookup during startup. Approximately how many pak files can be used without noticeably impacting performance?

We don’t have any hard data, but 10s of paks should be fine and I would definitely start to profile once you reach the mid double-digits, say 30+. This is mostly guesswork, though, so measuring the actual impact in your game will always be the most reliable way.

I have a couple of questions regarding pak files in Unreal Engine:

Suppose a pak file contains only static content such as textures, meshes, shaders, particles, etc., without any Blueprint code, and this content remains unchanged for, say, a year. Will this pak file remain binary identical across builds during that period, or will it still vary? If it does vary, is there a way to ensure that it remains unchanged as long as the actual content doesn’t change?

As far as I understand, the number of pak files may affect game loading times, particularly asset lookup during startup. Approximately how many pak files can be used without noticeably impacting performance?

Looking forward to your guidance.