UE5 C++ Compile Time is Substantially Longer

Development Programming & Scripting C++
, , ,
1

I’ve upgraded a medium size project to UE5 and we’re seeing substantially higher build times. I’m controlling for as many variables as I can and still seeing significantly worse performance in apples-to-apples comparisons to UE4.

I’m opening the same project in both versions, using Visual Studio 2019 to compile, setting build process count to 22 (12 core / 24 thread machine), and unity builds are enabled. I’m observing two undesirable aspects, one of which seems like a serious regression.

UE4: 2242 actions,         975 seconds,        22 processes
UE5: 2828 actions (+26%), 1820 seconds (+87%), 22 processes

Issue 1: Number of build actions
UE5 is building an additional 600 items compared to UE4 for the exact same project. Is this expected? 26% is a massive increase and it seems reasonable to expect 26% longer compilation times in UE5 simply because of file count increase.

Issue 2: Build time per action
Even after accounting for the higher number of build action, the time to compile is massively increased. I’d expect a roughly 26% increase in compile times to match the increase in file count, but instead I’m seeing 87%. Build times for the team have gone from 16 minutes to 30 minutes.

Memory is almost never maxing out during the build, so I don’t think that’s the issue.

We can look at the amount of time spent building a single action:

UE4:  975 seconds / 2242 actions = 0.43 seconds per action
UE5: 1820 seconds / 2828 actions = 0.64 seconds per action

So it takes 50% longer, on average, to compile a single file. This seems like a severe regression of some sort.

1 Like
2

I’m still looking for insights or solutions to this problem.

I’ve done a little more sleuthing:

  • I’ve disabled many built-in plugins we don’t need. This reduced the number of build actions to 2622, but yielded no measurable improvement to compile time.
  • I’ve tested without our corporate antivirus disabled. This had no measurable effect. Our UE directory was already excluded, so this only would have affected some in-memory scanning.
  • I’ve tested with both VS2019 and VS2022 with similar results.
  • I’ve diffed a few response files from before and after the upgrade and I don’t see any impactful changes to compiler flags.

Having compile time straight up double in UE5 is frustrating.

3

I’ve had the same experience but considerably worse. The potential for compile time reduction in UE is obscene and always has been - but Epic’s solution is the same as it’s always been really, throw it into a massive build farm.

UE4 built in about 22-26 minutes for me on average (5950X, 64GB, NVME). In UE5 that time has increased to 55-70 minutes, so more than double. UBT does now automatically reduce the number of cores that are used based on how much memory is available at the start of the build process to avoid the “out of heap space” issue when the PCH grows too large. On a 32 thread machine, 64GB apparently only gives me enough for 16 threads which is far too aggressive, so I’ve bumped it back up.

The main increases for me seem to stem from most of the core math types being templated now on account of double-precision support. To be honest though, the general poor hygiene when it comes to #includes doesn’t help, and is becoming particularly irksome in many of the new game feature plugins that have been added.

4

What are you using to measure header impact? That’s my next step when I get some time to poke around. MSVC has some built-in timing, I think UBT can dump chrome tracing formatted timing info, and there are some third party extensions to VS. Have you found one of those particularly useful, or some other tool?

5

Apologies, as I don’t have any insight into the issue, I’m merely curious as to a couple of statements/observations from purely a case study of things I might need to look out for in the future and what to avoid in my own development habits.

When you guys mention the increase in build times, what size projects are we talking about here? I realize “size” is a bit ambiguous but I’m trying to understand at what point do you get to where this really becomes an issue? Are the number of actions the best measure of size? Also, are we talking incremental builds, nightly builds, fully packaging solutions, etc.? I guess in my day to day workings with my current project I’m only ever dealing with incremental builds against the files I’m working with so I haven’t been bitten by this just yet.

Also, @ThJamsh you mentioned “…the general poor hygiene when it comes to #includes…”. What do you mean by that? Are you referencing the use (or lack thereof) of forward declarations? Again, asking from more of a learning perspective so I might have a sense of what to avoid within my own projects.

Thanks.

6

Compile Score is a pretty good one, and shows the hierarchy of includes building up to a single file too. Take this example (posted by somebody else after I complained about this elsewhere recently)

FZZMjnTWQAEIh8n
FZZMjnTWQAEIh8n1889×680 71.1 KB

At the risk of hijacking this thread to just have a good old moan, I just find myself becoming increasingly frustrated with how a lot of engine stuff remains incomplete or in an untidy state for long periods of time, sometimes permanently. It feels like it’s becoming increasingly common, and UE5 would have been the perfect time to do a general clean up.

The issue compounds over time because Epic are very excited to push all these new feature plugins before they are really production ready, then release them out of “experimental” still full of TODO comments, and it’s too late to clean them up because they’ll break everybodies projects. EnhancedInput, CommonUI, GameFeatures, even Gameplay Abilities is guilty of this. I mean just look at this CommonUI header, which is a pretty typical example:

FZYxa9LWQAM27FV
FZYxa9LWQAM27FV680×833 54 KB

I only wanted to use the Enum - but now that header is dragging in Streaming Types, Slate Types, Data Tables, Platform Data etc.

It’s a loosing battle trying to keep your project compile times low because all of this is done at the engine level, and it’s particularly a problem for reflected types where forward declaration is not always possible due to UHT requirements etc. Before you know it, your project has to adopt the same patterns.

Apologies, I’m just ranting now - but there is in general, enormous scope for compile time reduction in UE if Epic are prepared to invest some real energy into it. Any large codebase is going to have these problems of course, but we’re long overdue a good hardening pass.

3 Likes
7

what size projects are we talking about here?

Medium size AA project. ~77,000 lines of code, ~17,000 assets.

at what point do you get to where this really becomes an issue?

This is an engine problem, so it’s immediately relevant to any project of any size. Our actual game compiles very quickly. It’s entirely the engine that’s sucking up time.

are we talking incremental builds, nightly builds, fully packaging solutions, etc.?

Full project rebuilds. i.e. right clicking your game project in VS and clicking rebuild. This doesn’t build the entire engine, but it does build all the parts your game depends on which is still going to be most of the engine. We aren’t doing rebuilds in our normal dev flow, but they are fairly common for more systems oriented folks like me who are also learning Unreal or making modifications.

Are the number of actions the best measure of size?

No, just a reasonable tool to help compare compile time changes in Unreal.

8

I’m not sure if this will help you since my project is small, but it definetly helped me (went from 3 minutes compiling time to 3 seconds in my first project (just started learning)):

Go to:
C:\Users\YOUUSERNAME\AppData\Roaming\Unreal Engine\UnrealBuildTool.xml

edit it with the following code:

I found that in this threat:
Compilation taking twice as long in UE5: Number of processes being limited? - Programming & Scripting / C++ - Unreal Engine Forums

1 Like
9

Ah, yes this seems like the same issue I posted about here: Compilation taking twice as long in UE5: Number of processes being limited?

Check the above post for more info and details regarding this. It seems like they might have changed something regarding to how many processes are used (or some other factor that affects overall CPU usage) when compiling in UE5.

The only workaround at the moment for some people seems to be the above BuildConfiguration xml change, although it doesn’t lead to any improvement on my environment.

I think it’s up to Epic to fix this one, but as there hasn’t been an official response yet, the current status of this issue is unclear.

It majorly affects productivity, so fingers crossed.

10

If you read the OP carefully you’ll see I’ve already accounted for the number of processes. I’m using the same number of processes in UE4 and UE5.

The issue you two are running into is that Epic made 2 changes:

  • Use one compiler process per physical CPU core, rather than one per logical CPU. Essentially they disabled hyperthreading.
  • Require 1.5 GB of RAM for each compiler process.

UBT will take both of these into account and only spawn as many processes as allowed. In practice this means a lot of people are compiling with roughly a third of the threads they were before. I’m very surprised they made this change, personally.

You can re-enable hyperthreading using ProcessorCountMultiplier in BuildConfiguration.xml. And you can reduce the RAM requirement with MemoryPerActionBytes. You can also keep a couple of threads free so your PC doesn’t grind to a halt while compiling using MaxProcessorCount.

On my machine I’m able to effectively disable the RAM requirement. I have 24 threads (limited to 22) and 32 GB of RAM. Memory usage sits around 60% average with occasional spikes. It’s very rare the spikes exhaust my physical memory and end up hitting the page file, so it’s genuinely faster than Epic’s new defaults.

1 Like
11

Unfortunately, ProcessorCountMultiplier in BuildConfiguration.xml doesn’t seem to be effectively re-enabling hyperthreading on my CPU, or in any case CPU usage seems to be getting limited at around 20-40% capacity, with no build-time improvements from any permutations of said parameters in BuildConfiguration.xml in my situation

Intel(R) Core™ i7-6700 CPU @ 3.40GHz (Cores: 4, Logical processors: 8)

Example:

<?xml version="1.0" encoding="utf-8" ?>
<Configuration xmlns="https://www.unrealengine.com/BuildConfiguration">

	<BuildConfiguration>
		<MaxParallelActions>16</MaxParallelActions>
	</BuildConfiguration>
	
	<ParallelExecutor>
		<MaxProcessorCount>16</MaxProcessorCount>
		<ProcessorCountMultiplier>8.0</ProcessorCountMultiplier>
		<MemoryPerActionBytes>512000000</MemoryPerActionBytes>
	</ParallelExecutor>	
	
</Configuration>

Although the OP in this post also mentions an increase in build time per action, so it’s difficult to separate these factors and determine how much of this is simply due to an increase in engine bloat.

(Again, I was building “4 actions with 8 processes” in UE4, and the most I can get in UE5 is “4 actions with 4 processes,” assuming this actually means anything)

12

Running the same setup and have noticed the same issue with builds taking much longer on UE5. How did you adjust the max thread count back up?

13

A way to improve live coding compilation times is to go into task manager, right click live coding and press Go To Details. Once In details, right click live coding again find “Set Priority” and enable Realtime. If compilation speed is still slow, go into all subcategories of the task and repeat the process.

14

I’ve pretty much seen the same doubling in build times between my UE4 and UE5 customized engines. On my old machine UE4 would take 3+ hours but on my 7950X with 64GB RAM UE4 only takes 5 minutes and UE5 about 10. The customization doesn’t matter though since UE5 main takes about the same 10 minutes to build. I didn’t even bother testing UE5 on my old one machine, that likely would have taken 6+ hours. This is obviously a massive improvement for me so I couldn’t really complain. Other than that I do agree with @Jambax UE5 could have really benefited from a major cleanup, this somewhat feels like Unreal is slowly drifting down the path Unity has been on in my opinion.

How do you structure your projects? Is all your code in a single game module or do you separate it over multiple ones or use plugins?

My current project is only about 20 lines from my FDefaultGameModuleImpl and I do have a Server.Target.cs file. All other C++ code is written as (engine) plugins. Some really tiny, others potentially with several thousands of lines of code. Most of the time these plugins are very stable and rarely need to change. I try to keep my headers as small as possible and use forward declarations if UHT allows for them. The rest of the project is basically pure blueprint that consumes as many of the plugins as it wants and combines them into whatever game I’m working on which essentially eliminates the need to compile anything unless I package the project. Sometimes a game comes up with a feature in blueprint that may be useful in other projects. At that point I go ahead and write another plugin for it and then update the blueprints to use that instead.
Took me a couple of iterations to get to this approach but at the moment it works really well for me. Of course this may not be a viable approach for large projects, since I’m currently working on my project alone in my free time and haven’t used it in larger environments.