Mutable out of memory crash with Bake Instance

Development Programming & Scripting
, ,
1

Howdy!

I’m testing baking out COIs to bring into Maya for certain parts of our pipeline. With our main CO for the player, baking out any COI crashes with an out of memory error. I have 128 GB of RAM, and VS showed it spiking well above that. Task Manager also shows it hitting the ceiling quite quickly.

I tested with a simpler CO using the same meshes in the COI (but ONLY supplying exactly those meshes), and it worked fine. No crash. This makes me wonder if our main CO, which contains several tables referencing many outfits and meshes, crashes because of all the extra data, even if unused?

For a clear picture:

Working CO: References a simplified morph skel mesh (for using reshape mesh), a body skeletal mesh, and two outfits (each containing 3 skeletal meshes). That’s eight skeletal meshes total.

Crashing CO: References a body skeletal mesh, a metahuman head skeletal mesh, 15 outfits (each containing 3 skeletal meshes), and then several hair assets (10 skeletal meshes). That results in 57 skeletal meshes, but the COI only utilizes 6 of them (body, head, outfit01, outfit02, outfit03, hair).

Our engineer repro’d this and she tried changing CoincidentSimpVerts.Reset() to CoincidentSimpVerts.Empty() to free up memory. This got further but still crashed, saying UE was taking up 90+GB of memory.

Any idea on a fix here? Is the fix just stripping out excess outfit meshes from the parent CO graph when baking out a COI?

[Attachment Removed]

2

Hey there,

You have a couple of options.

  1. You can try enabling the disk to be used as memory. You can turn this on in the 3-dot menu in the toolbar. It’s intended for this type of scenario where you are baking, not doing something at runtime, with large or complex meshes/materials/textures. [Image Removed]
  2. If you’ve not set up your data tables with soft references to your meshes, I would recommend switching them; this should prevent loading every mesh when those data tables are loaded.
  3. Try breaking up your extra-large CO into Child Object Groups. This is the method we use in the sample to help with content segregation and expansion, but it should help with memory too.
  4. Convert to a dateless method where your graphs become more like “functions” you can run on any mesh passed in as a variable.

The latter two suggestions are really meant for runtime. But it could be useful for you in the future.

Dustin

[Attachment Removed]

3

Hi, Mutable dev here.

Sadly, some of the previous options Dustin mentioned may not work for Bake since it follows a slightly different code path.

  • Note for option 2. You can turn data table rows on/off by adding a bool column and using it as a filter in the table nodes. Mutable will not load or compile meshes within a disabled row. This will work, but it is not practical.
  • Note for Option 4: The Dataless method is not fully functional in UE 5.7, but it is production-ready in UE5 Main (UE 5.8 plus several bug fixes). With the Dataless workflow, you define the customization logic rather than the inputs. This means you can set up a graph to merge, reshape, and perform other operations, while specifying which Skeletal Meshes to use directly in the COI—without requiring a recompile. If you’re using Mutable as a pipeline tool to export to Maya, I recommend this new workflow.

Since this seems to be pretty content-related, it will be difficult to reproduce on our end without a content project. Would it be possible for you to run some tests?

  1. Could you try to reproduce the OOM issue in UE5 Main/Release 5.8? We fixed some bugs there that could lead to OOM in the editor.
  2. Could you try opening all 58 Skeletal Meshes before compiling/baking the COI? This could help determine whether the cost falls within the SKM compilation process or the CO compilation.
  3. Could you get an Unreal Insights trace of the bake process, with memory channels enabled, to identify what is consuming the bulk of memory? Mutable has its own category for memory tracking (MutableRuntime) in insights and LLM.
  • If the memory stays within the MutableRuntime category, then there’s a potential leak or maybe resources are kept in memory longer than they should. That said, a fresh compilation of one of our COs with ~14k meshes and textures keeps our tracked memory usage under 20 GB.
  • If the memory is due to the Skeletal Mesh compilation, then the issue could be with loading/compiling all meshes at once. Although compiling 58 Skeletal Meshes shouldn’t use 90 GB.

One question. Do you focus on mesh operations, or do you also do texture packing or any other texture operations with non-passthrough textures?

[Attachment Removed]

4

Hey there,

Apologies, I was away at Unreal Fest.

I can try this soon! Will copying our COs and content directly from 5.7 cause issues?

You may run into issues with how skeletal meshes are constructed in 5.8 when using a copy, but ideally, it automatically catches this and updates the CO for you. We’ve introduced a flow for constructing a skeletal mesh component that is more akin to blueprint, so it is two nodes instead of one now.

I’m curious what 5.8 features are in the works as well. Not to derail this convo, but I would love to know if more work is being done around the reshape mesh node. Is there a good avenue to chat with Mutable devs about planned work here? Would love to chat with y’all.

There have been significant optimization improvements with parallelized generation, and the new skeletal mesh node flow has introduced new, similar nodes to the old flow that interact with the meshes. Those flows have also gotten some performance improvements.

With insights and memory profiling on, crashed right when clicking the save button. Here’s a screenshot of the memory tags. MutableRuntime was only ~16MB.

Yeah, looking at that profile capture, you have a significant amount of texture/mesh usage, but if you could share that utrace I’ll be able to sift through the views and see what I can pick a part. Overall, though, you’re hitting the limits of your machine’s memory with 108 total usage.

Dustin

[Attachment Removed]

5

Yes,

I’m going to set up a quick email chain with your BD, and we can get you a location, then continue the conversation once they’re uploaded.

Dustin

[Attachment Removed]

6

You should have received an email for a box.com invite to upload your uTraces.

[Attachment Removed]

7

Thanks for sending us that! We took a look, and there is definitely something there, but we’re not quite sure what the issue would be. We think it is a content-bound issue that might be addressable. It would help us a lot to get a repro case from you in some way.

Either way, I will reiterate that you would benefit from dataless mutable if you are able to upgrade the plugin/engine to Unreal 5.8.1.

Dustin

[Attachment Removed]

8

Thank you both for the replies! Much appreciated.

From option 1, I enabled disc to be used as memory and actually got a logged crash. It created all the materials and the SKEL_ but crashed and did not generate the SK_. The log shows it attempts to save the generated SK_ file and crashes there.

---

[2026.06.10-22.09.09:448][450]OBJ SavePackage: Finished generating thumbnails for package [/Game/[AssetNameSnip]_h255799693]

[2026.06.10-22.09.09:448][450]Cmd: OBJ SAVEPACKAGE PACKAGE=“/Game/[AssetNameSnip]_h255799693” FILE=“../../../[DirSnip]/Content/[AssetNameSnip]_h255799693.uasset” SILENT=true

[2026.06.10-22.09.09:464][450]LogSkinnedAsset: Display: Waiting for skinned assets to be ready 0/1 ([AssetNameSnip]_h255799693) …

[2026.06.10-22.09.26:259][450]LogSkeletalMeshReduction: Reducing skeletal mesh for LOD 1

[2026.06.10-22.09.26:260][450]LogSkeletalMeshReduction: Reducing skeletal mesh for LOD 3

[2026.06.10-22.09.26:260][450]LogSkeletalMeshReduction: Reducing skeletal mesh for LOD 2

[2026.06.10-22.09.43:589][450]LogWwiseMonitor: Warning: Voice Starvation

[2026.06.10-22.10.12:718][450]LogOutputDevice: Warning:

Script Stack (0 frames) :

[2026.06.10-22.10.12:733][450]LogWindows: Error: appError called: Fatal error: [File:Z:\UEVFS\Root\Engine\Source\Runtime\Core\Private\Containers\ContainerHelpers.cpp] [Line: 8]

Trying to resize TArray to an invalid size of 2147483648

---

We’ll hopefully be moving to 5.8 soon and going the dataless method. Right now we utilize data tables with hard refs.

1) I can try this soon! Will copying our COs and content directly from 5.7 cause issues?

I’m curious what 5.8 features are in the works as well. Not to derail this convo, but I would love to know if more work is being done around the reshape mesh node. Is there a good avenue to chat with Mutable devs about planned work here? Would love to chat with y’all.

2) Opening all the skel meshes in a session didn’t fix this. I think both this and the log point to it failing after the CO compilation perhaps?

3) With insights and memory profiling on, crashed right when clicking the save button. Here’s a screenshot of the memory tags. MutableRuntime was only ~16MB.

[Image Removed]

We only focus on mesh operations and material assignments. No texture operations are currently in our CO and everything is a passthrough (or should be).

[Attachment Removed]

9

One of the OOMs we fixed in 5.8 was caused by thumbnails, which do show up in the call stack. These are the commits that fixed the OOMs we found.

https://github.com/EpicGames/UnrealEngine/commit/bc245994b0cf442e828cb70808c4a83390349b98

https://github.com/EpicGames/UnrealEngine/commit/8664e71c3796652b9e77f3c78172f08951e80f13

I would love to know if more work is being done around the reshape mesh node.

Nothing major. The core of the reshape operation will stay the same for a while. As a side note, we’ve been exploring the use of RBF interpolation to achieve a higher-quality reshape at a performance cost, but got sidetracked with other projects.

[Attachment Removed]

10

Thanks for the reply! Sad we missed each other at Unreal Fest. Hope you had a great time!

I’m attaching the utrace here if you see anything that might help. Sounds like Pere’s suggestion above with the 5.8 fixes might be helpful as well. I’ll pursue that since we may be upgrading soon.

[Attachment Removed]

11

Thanks Pere, I appreciate the reply. I’ll check those 5.8 changes out and see if I can integrate them.

If you ever need folks to work with on testing that RBF interp work, let me know! :grinning_cat_with_smiling_eyes: We’re leaning heavily on Mutable, and any way to cut morphs out of our armors would be rad for our pipeline.

[Attachment Removed]

12

It looks like the utrace may be failing to attach. Maybe because it’s ~200 MB? Let me know if there’s a better way to send it over.

[Attachment Removed]

13

Thanks Dustin. I’ve uploaded the utrace example I mentioned above. Let me know if there are any issues with it!

[Attachment Removed]