Problems cooking large project

Hey! I have a project with content around ~150gb and many little objects, like 200,000 of them. I have 32GB ram and good powerful PC. I get no errors while I cook, but unreal randomly stucks up on some files, like little meshes I have, and cooks them like forever. CPU usage is high, memory grows, disks writes are made, but this can go on for many hours without any progress. Log just shows typical output, no errors or anythings.

Besides that, I am trying to use iterative build to track the issue, but Unreal always starts from scratch if I use packaging option. If I use ‘File - > Cook Content for Windows’, iterative seems to work, but oddly, it often processes same files over again.

This is so confusing and frustrating!

Is that only me? Can be the issue that I am running out of memory or something, and that eventually makes it run forever? Also any docs on iterative builds? I could not google literally anything.

I would just try packaging part of it, like 50gb, then add on another 50gb as a patch build, and so on…if it is using a single .pak file in the cook settings it could cause it (because it compresses it all)? I have never cooked that much at a time before so I am probably not much help…

Yah, that is what I will try to do eventually. Is there some way to actually debug what is going on? Like, I guess cooker source is opened source too, maybe I can see what it is actually doing all the time? I wonder should I look for it inside Editor tools sources. It is a part of UnrealBuildTool maybe?

Are you talking about building lighting or compiling the final game files? For something like that it sounds like you wouldn’t be able to build lighting even with 32GB of memory.

I am talking about Cooking only. Game has no lights, I actually have ‘use static lights’ disabled

I could debug it probably, but the problem is that after I click ‘Cancel’, cooking starts again and takes about 5-6 hours to cooking around 110GB of data, then it stucks, and this makes very hard to debug the issue with such waiting times. Still, I finally noticed a pattern. It actually gets stuck on same few files, so I am trying to isolate now. Now that I have something to work with, this question probably needs no answer. Hopefully I’ll get to the bottom of this now.

No, in empty projects, those files cook just fine. And if I delete them from original once, others just take their place. Hmmmm, so this must be related to RAM somehow or something that overflows due to large size. Any idea how could I debug the issue?

EDIT: Randomly breaking the project, I suspected this is distance-field related. Disabling them now and trying again
EDIT2: Yes, true, distance field meshes were causing it. I will continue looking into the issue after building the project fully. Maybe then it will start iterate properly, hence I will be able to recook only meshes, but not whole project