How do I work with this engine without having to rebuild the whole thing all the time?

Development Programming & Scripting C++
1

I’ve spent the past two weeks coming up to speed on UE4 and I’m pretty sure 90% of that time has been just waiting for engine rebuilds that shouldn’t be happening.

I’ve built the engine from source with a couple minor modifications, and am working in a C++ project created with that source build. Here’s how I’d expect that to go:

  • In Visual Studio, I make a change to game code (not engine code).
  • I hit Build (not Rebuild).
  • Game code recompiles incrementally (not from scratch).
  • Engine code does not recompile at all (it was already built in order to get the editor I used to generate the .uproject, and has not been changed since then).
  • I launch my source build of the engine and open my .uproject, which loads normally and is ready for editing.

Here’s what actually happens:

  • In Visual Studio, I make a change to game code (not engine code).
  • I hit Build (not Rebuild).
  • Flip a coin. Heads: game code recompiles incrementally. Tails: entire engine source is rebuilt from scratch, ETA ~40 minutes.
  • I launch my source build of the engine and open my .uproject.
  • Flip a coin. Heads: the project loads normally. Tails: I get a dialog about my game assembly being out of date and asking to build it (even though I literally just did that). If I click no, the editor closes. If I click yes, the entire engine is rebuilt, ETA ~40 minutes.

This morning’s incident was the worst I’ve encountered so far. Last night I built the engine (again), built the project, loaded the project, and verified everything was working. Then I shut down and went home. Came in this morning, launched the editor, and opened the project – having made zero changes to anything in the interim – and the “out of date” dialog popped up and kicked a full engine rebuild again. Because… reasons? And then after ~30 minutes that engine rebuild failed with a non-specific error and told me to try building again from scratch, leading to another ~40 minutes of building in Visual Studio before I could even open the project at all.

How am I supposed to get anything done in this engine when it decides to rebuild itself every time I sorta-kinda look in its general direction?

I’ve pored over documentation and forums and tried a bunch of different experiments to figure out how the buildchain works and why my previous builds are getting invalidated. Things I’ve checked:

  • In Visual Studio I’m hitting Build, not Rebuild. It should be doing incremental builds. Sometimes this works. Often, it doesn’t.
  • I’m building the same solution configuration as before. (Naturally, changing solution configurations would cause many things to be rebuilt. I’m not doing that.)
  • Nothing is ******** with the Intermediate folder. In fact, I’m often getting full engine rebuilds even when Intermediate is already populated with the up-to-date object files I just built the night before.
  • I’m not making new engine code changes. (I have a small set of engine changes from early last week and no new engine changes since then.)

Are there other things that can invalidate my build artifacts? Is there some way I can debug or log this buildchain to hopefully shed some light on why it’s deciding (wrongly) that all this stuff needs to be rebuilt all the time? Is there some magic technique everyone else is using that I’m just missing?

FWIW: I’ve tried this with both VS 2015 and VS 2017 (currently using 2017) but gotten the same results. My engine build is branched from release (4.15.1) and my engine changes are minor UnrealEd interface tweaks that have nothing to do with building. And I’m on Windows 10.

2

Hmm, I’ve never (personally) seen this issue.

When using VS, have you installed the Unreal VS Extension and simply added your UProject to your commandline parameters, so the engine will just instantly launch the editor with your project opened?

(You can ignore most the setup states. The newer version of the installer just works).

3

I’m not sure if this is the case for you, but a little tip: don’t run the project from an external HDD or any other slow storage.

I run mine on an SSD scratch disk, and find build times are short and compiling takes seconds.

Is this in one project, or have you encountered it across multiple, different projects?

4

I’m building on an NVMe SSD with a little over 3.0 GB/s read and 2.0 GB/s write using an i7-7700K at 4.2 GHz. Visual Studio builds still take 40 minutes. Installed builds take even longer.

And this happens on every project I try. I’ve spun a bunch of different bare-bones test projects to try different things on. They all trigger near-constant engine rebuilds.

5

C++ architecture seems kind of bound that way or maybe not I don’t know for sure.

6

Happens to me all the time - until now!

My advice is, make your source code changes - then package up a binary build of the engine. You get consistent fast build times, and you can easily distribute it to a team. You can also make engine changes, set off a build machine to build it and keep working in the meantime.

Usually when you make an engine change, you make it then never touch it again for a while - so this is a great workflow for me at least.

7

Yeah, I’m leaning pretty strongly in this direction now.

8

Right-clicking on the game project and selecting “Build Only” works fine for me.
You can also create a short-cut for it or similar:
Build Current VS Project With Single Key Press

9

Some people have had this issue many times, others seem to never experience it. I’m pretty sure for the most part it’s caused by some bugs in UBT that only rear their head under certain circumstances. I know at times I’ve pulled an engine version from Github and had variations of this happen, other times it’s been fine.

As TheJamsh says, by far the best approach is to create an installed build, which basically tells UBT “Don’t bloody touch me, I’m built already”. That said, I’ve seen UBT clean lib files from launcher engine installs at times, so you’re never fully safe!

1 Like
10

I just tried making an installed build from a folder where I’ve previously built the engine using Visual Studio. The buildchain scanned all my source files, saw that they were up-to-date, and moved on to deployment within like 90 seconds… which is what I would’ve expected given I omitted the -clean flag in the call to RunUAT.bat (I specifically wanted to try an incremental build, which appears to have worked just fine).

In deployment, it failed attempting to access an MSBuild executable (pdbcopy.exe) that didn’t exist at the expected path. That’s because I’m using VS 2017 and it’s assuming that file is located at its VS 2015 path. (Why they’re different between versions, I don’t know. Ask Microsoft.) So I went and fixed that with a symlink in the MSBuild folder. QED.

Then I kicked the incremental build again. Expected result would be that it’d scan my source files, build nothing because they’re already up-to-date (just like it did ten minutes ago), and move on to deployment, which would now work because the missing MSBuild file is now accessible on the expected path.

Instead, it is rebuilding the entire engine.

There is no sensible reason for this. I didn’t clean my project or remove any intermediate files. I didn’t touch any code. I ran a successful incremental build once, failed on copying the output of that build, made a minor system change completely external to and unrelated to the engine, and ran the exact same build script on the exact same data again… and got a completely different result.

I’d love to understand this process, and I’m honestly really trying. I’ve been making games professionally for 13 years and I’ve dealt with (and solved) a lot of really hairy stuff in that time. I’m not the type to demand easy answers handed to me on a silver platter, or to give up on solving a problem because the going gets a little rough. I do expect to have to work for it and I’m more than prepared to do so.

But this? This is just nonsense. Thousands of dollars worth of billable hours wasted waiting for totally unnecessary (yet apparently unavoidable) engine rebuilds does not endear me to this engine, and I can’t presently afford to burn dozens more billable hours digging into the massive codebase to figure out why this keeps happening and submit a real fix. I haven’t found anything documented – either in official docs, forums, or AnswerHub – that explains how or why UBT decides things needs to be rebuilt, so I’m flying pretty much blind trying to troubleshoot it. And every failed iteration wastes another 40-60 minutes of building time, which adds up insanely fast.

I guess there’s no real point to this post other than to rant, but man I am just super frustrated with this.

11

Not related to your question, but I’m also using a 7700k and a full engine rebuild takes only 20 minutes. Something must be configured wrong on your end for it to take twice as long.

When the compiling starts, does it say something like “x jobs, 8 in parallel” or “4 in parallel”? If it’s 4, try setting processor count multiplier to 2 in buildconfigration.xml. It sometimes fails to detect that hyperthreading is available.

12

That is promising! It does say “4 in parallel”… but it still says that even after I increased the processor count multiplier in the BuildConfiguration.xml. :confused: That said, Resource Monitor does report 100% CPU utilization across all 8 logical processors during the build, so uhhhh tentative yay? Maybe? I’ll see how long this build actually ends up taking.

Switching gears, and exploring this some more… on Friday I built the engine from scratch in Visual Studio, and then made an installed build without the -clean flag immediately after that. The installed build built incrementally, so it seems like both buildchains are sharing the same intermediate files (provided they’re built with the same solution configuration, as far as I can tell).

However, as noted in my previous reply, that incremental installed build failed launching an external tool after the compilation itself was already done, and once I fixed that external tool and re-ran the build it rebuilt the entire engine. So, despite being an incremental build without the -clean flag, it seems like it invalidated its intermediate files somewhere along the line. That seems suspect.

Anyway, I got in this morning to discover that second build did complete successfully. So I decided to test my hypothesis that the VS and installed build processes share intermediate files. If that’s true, that would mean that intermediate files would have been populated by the installed build that just completed, so running an incremental VS build now should report everything already up-to-date, and build nothing.

Actual result: VS is currently rebuilding the entire engine.

I’m a little mystified how I can build in VS and then get a quick incremental build with RunUAT.bat, but not the other way around. I guess I’ll dig into the installed build process and see if it’s explicitly nuking intermediate files somewhere along the way…?

13

Which build configuration did you use? I copied the one setting over to the global one in appdata and it seems to work.

14

Yeah, I initially modified the BuildConfiguration.xml that lives in Engine/Programs/UnrealBuildTool but it reverted itself as soon as I ran the build. Then I searched the docs and discovered that overrides have to be installed elsewhere. Now I have a copy in roaming appdata which sets the ProcessorCountMultiplier to 2 and it works as expected: all eight cores firing away.

15

I ran an experiment last night to try to narrow this down some more:

  1. Did a clean solution rebuild (using the Development Editor configuration) in Visual Studio. This built everything, as expected.
  2. Did an incremental build in Visual Studio immediately after. This built nothing new, as expected (all targets up-to-date).
  3. Made an incremental installed build (omitted the -clean flag). This rebuilt everything, which was surprising.
  4. Made a second incremental installed build immediately after. This built nothing new, as expected (all targets up-to-date).
  5. Did an incremental build in Visual Studio (on the same Development Editor configuration as before). This rebuilt everything, which was again surprising.

So it seems like if you make an installed build it invalidates your targets from a source build, and vice-versa. That makes switching back and forth between the two really painful. For example, say you’re working on engine customizations and therefore iterating on source builds. Once you’ve got one clean source build done, you get quick incremental builds as long as you stick to the same solution configuration. Once you’re ready to deploy those changes to the team, you make an installed build, which is a full rebuild (even if you omit -clean). But then as soon as you look at the solution again, your source build is also a full rebuild, even though no source files have changed. So you’re incurring a 2x full rebuild every time you need to deploy an installed build but also continue working on the engine beyond that.

This all seems really unnecessary.

One thing I did notice is that in each case where a full rebuild was triggered unexpectedly, I saw a line in the build log to the effect of (and I’m paraphrasing as I don’t have it in front of me right now)

That “UnrealBuildTool.exe is newer” seems suspect to me. I notice both it and the UnrealHeaderTool get rebuilt whenever I switch from source to installed or vice-versa. I don’t know (yet) if this is the only thing causing the entire source to be rebuilt but it sure seems to be a likely candidate.

16

I doubt they ever intended you to use installed builds and normal ones together like that, because they only make the installed builds using a build farm… chances are it simply doesn’t work.

17

Well it kinda sucks for us mere mortals who need to supply a (small) team with builds but don’t have a bunch of spare machines to dedicate to a build farm. :expressionless:

18

That will absolutely be it, and of course it’s also completely unnecessary if you’re not modifying UBT C# source. I’d try making the UBT executable read only to see if you can track down when/why it wants to rebuild it.

19

Now I’m wondering if it is necessarily the case that a full include scan implies a full rebuild. Perhaps not, though from memory it feels like that is generally the case.

This seems like it could be relevant, in particular the adjustment to RebuildString.

20

So what kind of files require full rebuilds and which are ok for build? It’s risky making engine mods some times.