We’re working on some changes to make compiling your C++ game fast. This post is about one of the changes we’re testing now for the upcoming 4.8 release. For those of you brave enough to use the latest master branch code, you may have noticed that UnrealBuildTool will startup very quickly – even when working with huge game projects and the full engine source code. Here is a quick overview of this new feature. Would love to your hear feedback!
TL;DR version
UnrealBuildTool now starts up fast and checks dependencies very quickly. After the first time building a target, compiling a change should begin almost instantly. The first time you trigger a build, we create a “Makefile” with everything needed to compile that target. It will reuse that makefile in subsequent runs. There is some new console output that shows you what’s going on.
!! IMPORTANT !! When you add new C++ source files, you now must update your Visual Studio project files! Either add the new file to the project yourself using the IDE, or just regenerate all project files. This is a C++ workflow change, as previously UBT would be able to detect added or removed source files automatically. Similarly, if you change a *Build.cs or *Target.cs file, or modify your global BuildConfiguration settings, you must refresh project files manually before compiling again! We’re still investigating whether we can make this automatic in a future version.
If you have any problems, you can revert to the old system by turning off bUseUBTMakefiles in your BuildConfiguration.
Long version below
So I’ve been on a crusade to make it fast for programmers to make small C++ changes quickly, even in big projects. There are many challenges, but one of the most obvious was that it could take UnrealBuildTool up to 15 seconds to figure out whether it even need to compile anything!
There wasn’t a lot of low hanging fruit with what UBT was doing to prepare a project for building, so we decided to cache everything off into a Makefile so it could be used over and over again as you work. The first time you compile a target, we create a Makefile for it. This is called the “gather” step. In subsequent builds for that target, we load the makefile, check for outdated source files, run UnrealHeaderTool (if needed), then kick off the compiler. This is the “assembling” step. The assembling step is really stripped down – it just GOES.
UBT still quickly scans thousands of source files every time you build, but we do it very efficiently by caching off a flat list of resolved file paths for any given translation unit. C++ dependency scanning has been rewritten and now takes almost no time at all. It no longer has to load source file contents each time you build, it only checks timestamps. I also borrowed an idea from the excellent Ninja build system: We no longer spider through source files to update our include graph unless a translation unit was already out of date. Think about it – unless a source file itself was modified, it can never become outdated by includes other than the ones we already knew about. So on your very first build of a target we scan includes, but then only again if a translation unit was outdated (either because you changed the source file, or you touched a header that was already known to be included by it.)
Even better, we can update our include graph for these modified files asynchronously on a thread, while UBT is busy compiling your changed code. So effectively this now takes no time at all. (As an aside, Ninja solved this by using the /showIncludes option every time you compile, but it was very noisy and requires platform-specific handling as well as special PCH handling.)
If you were building a target, then you switch to building a different target, UBT detects that and will force a rescan of the full include graph. This is important because it can’t tell whether you modified a shared source file while you were working with that other target, so it needs to rebuild it’s cache. Hopefully it should just work automatically.
Hot Reload is fully supported! This is really cool because Hot Reload can now complete in less than a second in the best case. Awesome! Because every time a hot reload is initiated, we need to make up new file names for the output DLLs (to avoid collisions with already-loaded DLLs), supporting hot reload requires us to “patch” loaded makefiles to update the output file names as well as a few other things. We store a different Makefile for hot reload, so that you can ping pong between hot reloading and regular builds without having to wait for a makefile to be rebuilt.
!! Important !! With this feature, UBT cannot always detect when a new Makefile is needed. If you add new source files or delete source files, you need to make sure your project file is updated first! You can either regenerate project files, or you can manually make the change to the project and save it. UBT will notice this and create a new Makefile for that target.
There are other changes that UBT cannot detect, and you need to regenerate project files manually. (You can also rebuild UnrealBuildTool.exe (faster), which will force a new Makefile to be generated.)
- Syncing down new code from Perforce or GitHub
- Adding a UCLASS to a file that didn’t have one before (new generated code)
- Changing global build settings (BuildConfiguration)
- Added new source plugins
- Changing your system environment variables
- Installing a new platform SDK
- Dropping in source for a new plugin
- Changing UnrealHeaderTool itself
Sometimes UBT can detect itself that you need a new Makefile, such as if the project files were saved since you built last, or if UnrealBuildTool itself was recompiled. It will print out information to the console when building that shows what’s going on, and why a new Makefile was needed. Hopefully we can make it even smarter in the future without compromising performance.
There are some new intermediate files for this feature. These all get deleted if you do a clean.
- Makefile.ubt: This is the actual makefile we generated. It has everything needed to build a target.
- HotReloadMakefile.ubt: Hot reload version of the makefile (if needed).
- LastBuildTargets.txt: This tells UBT what you were building last. If you switch targets, UBT needs to do more work.
- FlatCPPIncludes.bin: Contains a mapping of every .cpp file to all of its included files (directly or indirectly). Used for fast outdatedness checking.
Note: UBT will no longer automatically recompile UnrealHeaderTool during incremental builds. It will only do that the first time, when your Makefile is created. We thought this was a good enough trade-off, but be wary if you are working in UnrealHeaderTool!
Bugs!! It was a complicated project, so there are still probably some issues left. Many of the known issues are notated in the code with “@anonymous_user_5851a2121 ubtmake”. For example, the feature doesn’t work at all with RPC or with certain platforms. And there are some edge cases if UBT is terminated unexpectedly where your include cache can become invalid. We’re going to keep working on this feature in hopes of using it by default in UE 4.8.
–Mike
Thanks Mike, it was quite strange that starting the whole process sometimes took more than the actual compiling time! I love to see such improvements, are there even more planned?