Issue with Chunking during Packaging for Unreal Engine

A.RADIX · October 27, 2023, 5:19pm

Hello,

This topic is related to the previous one I made : Chunk in package are not the same as in Editor
And this is the next step :

We are using Pixel Streaming on Windows machines to deliver our solution. The .pak file allows us to save disk space and loading time, making it an important aspect of our project.

In our solution, we have around a hundred programs, where each program consists of a set of 3D assets. These programs are independent of each other but share the same base code and common assets. They are packaged at different times. Once packaged, these programs may receive updates to their code or 3D assets. Therefore, it is crucial that the part containing the code/DLL/UProject/Plugin is completely independent of the 3D assets of a program in the packaged version.

Objective: To have a separate 3D Asset chunk for each program (for example, having “Chunk10” containing all the 3D assets of Program A). Within this chunk, we need to include :

1 - All engine assets used by the 3D assets of Program A.
2 - All assets specific to Program A, without exceptions.

I’ve encountered some issues:

1 - By default, all engine assets are placed in “Chunk 0.” I managed to move the engine assets used by Program A to “Chunk 10” by modifying the Unreal Engine source code. However, I had to comment out the part of the source code that says “this avoids errors.”

Is there a way to duplicate assets across multiple chunks? Can we have an engine asset “X” in both “Chunk 0” and “Chunk 10”?
This might only be possible with assets that are not part of the engine, as the source code seems to place engine assets in “Chunk 0” in all cases.

2023_10_27_19h03m49821×80 8.06 KB

2 - I’ve managed to place 99% of the assets of Program A in “Chunk 10,” but a few assets did not end up there. I have no idea why. Here are the assets:

F:\52\ModerApp\Saved\Cooked\Windows\ModerlabApp\Content\01_Application\02_Common\CAR\Mesh\Common\SM_Car_PASSAGER-3 F:\52\ModerApp\Saved\Cooked\Windows\ModerlabApp\Content\MSPresets\Foliage_Material\MPC_Wind_ML

Their references seem correct, and they are not used in my C++ code.

To identify the reason, I need to set a breakpoint in the functions that assign the chunk number for an asset. However, when I set breakpoints in the packaging classes (for example, FAssetRegistryGenerator), they are never triggered during packaging.

I can successfully trigger all the breakpoints related to runtime. When I look at the loaded modules, I can see UnrealEd with its debug symbols. FAssetRegistryGenerator is within the UnrealEd module.

Does anyone know how to trigger a breakpoint in this function? This would help me find the cause of this issue. I haven’t found any information on this topic.

If anyone has any information or suggestions, please don’t hesitate to chime in. I’m looking forward to your responses. Thank you!

A.RADIX · November 7, 2023, 4:58pm

I’m still struggling to trigger breakpoints during the packaging process, but I’ve made some progress on another issue. I discovered why one of my assets ended up in chunk 0 instead of chunk 10.

I never paid attention to it, but in the reference viewer, we can identify if a reference is an editor reference or not. Editor references are shown as grey links instead of the standard white ones. Thanks to this helpful video by Mathew Wadstein: Link to Video.

If you take a look at the reference graph in my previous post, you’ll notice that the links are grey. It turns out this asset was referenced through sockets:

But I’m still stuck with the issue where an asset that should be in chunk 2 is not there. The Asset Audit shows it should be in chunk 2, but it’s not.

2023_11_07_17h20m40
This asset is a Collection parameter inside a Material function. What’s puzzling is that it’s marked as an editor reference for MF_Wind_ML: