Download

How to get the build system to use more threads? (4.27)

I just upgraded to a new i9-12900k based system with 32gb ram and some fast ssd’s - on my first build of the engine from source I see that it is only maxing out 8 of the 24 hardware threads. I am using windows 10 pro, not 11, so I was wondering if this is some byproduct of the new processor’s asymmetric core architecture but either way - I’d like to convince the build system to peg at least 8 more threads.

I’ve tried messing with the buildconfiguation.xml to set max processor count to 24, and that gets the output to say building with 24 processes, but resource monitor still just show 8 cpu’s maxed out.
v4.27.1
ideas?
Dave

And did you try → Menubar->Tools → Options → Projects and Solutions → Build and Run → maximum number of parallel project builds (what Version of VS is is it u’re using?)

no – will try that now. Thanks, DR
(using VS community 2019)

nope-says 24 processes, but according to resource monitor only 8 cpu’s are maxed…

Strange… :thinking::thinking::thinking::thinking:

Might be an issue with the core architecture of these CPU’s. You can try having a look through some of the settings here. UBT uses it’s own settings independent from the settings within Visual Studio, but by default IIRC it is setup to use all available cores.

As an addendum, 32GB of RAM is definitely not enough for 24 threads anyway, so opening up more threads may result in occasional build failures due to running out of heap space. Every instance of cl.exe has it’s own copy of the PCH in memory, so once it grows above a certain size (which it frequently does in large UE projects) heap space becomes a real problem. (I’m no compiler expert but that’s my understanding of it).

My advice currently is at a minimum, 2-3GB per-thread, leaving you some overhead for the OS too.

thanks for the reply! I’ve been watching the memory and in-use doesn’t scrape up over 18 currently. I do see how if all 24 threads fired it could be a problem. Good safety tip. I guess I should be shooting for something more like 16 in flight at once. That would be excellent too.

The link you referred to is where I got the information on the buildconfiguration.xml – and the print outs in the build output show that it is reading it, I just don’t see a difference in what it really does. Maybe a problem inherent to the new CPU architecture with P-cores and E-cores confusing the issue.

Will try some more tinkering later.
DR
(may require pulling out the shovel and digging into the build system itself to see what’s what)

Update for anyone else finding this thread. I took my build time from 3700 seconds to 1500 seconds by going into the bios and disabling all e-cores. After that the build system ran at 16threads maxed out.

I hope we get some updates and scheduler work to make that performance available with the entire cpu operational. Maybe in windows 11? I’m not ready to beta-test that with my work yet.
DR