We’re noticing that compiling is taking ever longer – up to 36 seconds for our C+±based project. All of our engineers have fast PCs (Intel i7-4770K cores with 16 GB of RAM).
36 seconds is certainly bearable for now but it seems surprising that it would take this long when our codebase is relatively small (our team is tiny and we’ve only been writing code for a little over a month), and we’re concerned about how this will scale as our code base grows much larger and we have many more engineers working on it.
I’m seeing that on my own PC, if I do a full clean & rebuild, only 4 out of my 8 available cores are utilized, and those are only utilized between 15-45% (according to the Windows Task Manager). Overall utilization seems to be around 15%.
There must be something we can do to improve compile times. We’ve got concurrent compilation turned on (Tools / Options / Projects and Solutions / VC++ Project Settings / Maximum concurrent C++ compiles = 0 i.e. use all processors). But I don’t see any other viable compiler options, presumably because you’re using a separate compiler.
I was hoping that using the in-editor recompilation feature would help, since surely it would be faster to recompile in-editor than go back to Visual Studio, recompile, and relaunch the editor. But several tests of recompiling a small change showed that it took 49-51 sec to recompile a trivial change inside the editor, compared to a consistent 35 sec in Visual Studio.
And that in itself is surprising, considering that a full rebuild takes 36 seconds – why is a trivial change (changing a few floating-point values on a color parameter in a function in a non-Unreal object) taking almost as long to compile as a full rebuild?
So, there are really 4 questions here:
- Why isn’t the compiler using all 8 cores?
What can we do to improve our own compile times? ( ← this is the big one, of course)
- Why is in-game recompilation slower than compiling in Visual Studio?
- Why does there seem to be no incremental compilation going on, such that a trivial change takes almost as long as a full clean & rebuild?
I also would like to know how to improve compile times
It is a big deal for me since I am changing code constantly
and I am dealing with 34 second compiles times currently.
My code is now subdivided by VS into 3 parts during compiling, to give you sense of the amount of code I am using.
Specs: i7-4770K, 64GB RAM, 128GB SSD.
Rocket is installed on the SSD, but the project is located on a 1TB HDD (Western Digital Black).
Relatively small usage of C++ at this time, due to switching a number of functions and child classes from C++ to blueprints.
VS2012 (Rebuild, Development Editor config): 26.68 seconds.
Notepad++ : 27.25 seconds.
As far as I am aware, the CPU is the biggest limiting factor and you have that covered. If you haven’t already, ensure that Windows isn’t running crap in the background that you don’t need. Cycles wasted and all that.
In my case, the SSD may/may not help. It depends on who/what you believe. Given the cost of a decently sized SSD, improved load times and the like, I figured what the hell.
A RAM drive is another way you could possibly squeeze out an extra second or two. This is not something I have tried, despite having a motherboard with a built-in RAM drive feature (Asus ROG).
Here’s an example of what I mean – note the 12% overall usage and only every other core being utilized.
why is a trivial change (changing a few floating-point values on a color parameter in a function in a non-Unreal object) taking almost as long to compile as a full rebuild?
There is a known issue here that we’re planning to address for Rocket but haven’t gotten to it yet. Basically, UE4 is designed for HUGE games that have hundreds of source files, so the out-of-the-box settings are configured to get the best possible full recompile times with huge game code bases.
What’s happening with your one-file change is that Unreal Build Tool is globbing together that source file into a combined file with the other code files in your module, and also not bothering to generate (and cache!) a precompiled header for your game source since they all fit within a single combined source file.
As a workaround, to get the faster iterative times with small changes, please try the adding the following lines inside your game module’s Build.cs (MyGame.Build.cs):
MinFilesUsingPrecompiledHeaderOverride = 1;
bFasterWithoutUnity = true;
This will turn off combining of source files, and also tells Unreal Build Tool to always create a precompiled header, which will yield very fast iteration times.
Beyond this, we are also making changes to make Unreal Build Tool startup much more quickly.
Let us know how it goes.
Thank you – THAT’S the answer I was looking for
I will give that a shot ASAP!
OMG thats hilarious, so much pun intended
“Beyond this, we are also making changes to make Unreal Build Tool startup much more quickly.”
Thank you Mike and Epic!!
Mike, thanks so much! This is EXACTLY what we were looking for, and perfectly solves our problem.
We had to make some very minor include file changes, but it’s well worth it – a minor one-line change now takes only ~6 seconds to compile.
Basically, UE4 is designed for HUGE
games that have hundreds of source
files, so the out-of-the-box settings
are configured to get the best
possible full recompile times with
huge game code bases.
This is great – our code base will definitely get to that point, and we’ll be able to use either method in the long run depending on what works best.
#Oh My Thank You Mike Fricker
My compile times when from
Reduce amount of code (;.
But now seriously. You can’t do much. It’s really not you or Epic can really improve, besides very small optimizations of how much code and files there are to compile.
It is entirely dependant on compiler you are using. We are locked in, single one for now, so options are even more limited at this point. It’s early in the morning here, now, but when I get back I will look for compiler flags that might improve compilation times (but that is usually at the cost of code optimization, but that shouldn’t be a problem during development).
Another thing to bear in mind is not to use Rebuild All option, as this remove everything and recompile code from scratch.
Use Build instead, this tracks changes in your code, and recompile things only that changed. It should reduce build times.
Well the thing is, this IS something Epic can help with since Visual Studio is set up to route everything through UnrealBuildTool.exe (look in your project properties → Configuration Properties → NMake).
I’ve asked for documentation on UnrealBuildTool command line flags and options before but no dice. I don’t know whether it just forwards everything to Visual Studio’s compilers in the end…
I’ve also tried using MSBuild.exe directly (as per here, that’s for C# but it should work for C++ as well), but never could get it to work. About 27 sec for a decent-sized project on my rig, would like to get it down to sub-10.
“It’s really not you or Epic can really improve”
Lukasz: nonsense. You don’t have any way of knowing that at all.
Do you know what compiler Epic is using or what the build flags are for that compiler?
“Another thing to bear in mind is not to use Rebuild All option … Use Build instead.”
Please re-read my original post. I provided compile times for Build vs Rebuild All, which are nearly identical.
Yes. They are using VC++ compiler along with VC++ Libraries. In other case, you wouldn’t be using visual studio tool-chain (;.
In any case I was only referring to compilation part, not the prebuild part.
I was referring to the compilation part, too.
And yes, you can use other compilers under the hood with Visual Studio – for example, LLVM: Getting Started with the LLVM System using Microsoft Visual Studio — LLVM 17.0.0git documentation
I asked them about possibility of using LLVM/Clang for compilation, and there is no option for that:
- Entire tool-chain is build around VC++ and Visual Studio.
- List item
- Third party libraries are made in mind with VC++ compiler as well, it is not practical to force use, something else with them.
The only part I can see real improvements are possible is the prebuilding part, that build all needed C++ headers, and doing… whatever it is doing.
I don’t feel the compilation times are that big for now. Compiling game code for CryEngine, took on my machine from scratch took about 15 mins (probably closer to 20).
The other possible issue is that there is lots of headers included for files. Which means quit a bit of traversing and possible linking. But… you either have to wait a bit longer, or you loose the nice, convenient structure of accessing engine classes.
Having experience with CryEngine game code, I would never sacrifice the UE4 convince. It makes things getting to work sooo much easier.
I asked them about possibility of using LLVM/Clang for compilation
That’s nice. It’s a completely separate question from mine, though. I am trying to open a dialogue with Epic and get their thoughts about all of this.
I don’t feel the compilation times are that big for now.
Err – what? I don’t mean to be rude, but how is that helpful?
I DO feel like it has the potential to be a concern for my project in the future, based on the reasoning I outlined in the original post. And if it does have the potential to be a problem in the future, wouldn’t you agree that it would only make sense for me to bring up the problem as early as possible to give us more time to discuss it with Epic?
It sounds to me as if you are trying to imply that my concerns about compile time are not valid.
I have a hard time with that.
Having experience with CryEngine game code, I would never sacrifice the UE4 convince.
I think you are missing the point. No one is bashing Unreal here, and no one is comparing it to any other engine or even discussing the possibility of abandoning it for any other engine. I am trying to open a discussion with Epic on these issues, and that’s it. There’s nothing more to it than that.
It sounds to me as if you are trying to imply that my concerns about compile time are not valid.
I’m not. It’s valid concern, but having my self compiled hundreds of C++ applications, I just don’t see room for big improvements in that regard.
But that’s me. I just probably learned to live with it already.
It would be cool to hear from Epic though (;
Please see Mike Fricker’s answer below.
I don’t care that you’ve compiled hundreds of C++ applications; I’ve been programming for over 20 years and have compiled hundreds myself. That doesn’t mean either of us knows the whole story.
It’s not a good idea to dismiss other people’s concerns when you don’t have a good enough basis to dismiss them.
I wonder how can I save an answer…