Iwyu

Ben_Marsh · November 23, 2016, 9:13pm

I’ve just submitted a change to master branch on GitHub to convert the engine to an “include what you use” model. This is a big change!

Rather than including big headers like Engine.h and UnrealEd.h everywhere, each header now includes just what it needs. There will doubtless be a few teething problems switching over to this, but we’ve been seeing a 20-50% improvement on build times in its staging branch, so hopefully it’ll be worthwhile.

Here’s the commit description (4ba423868f9b6349d8f80a6e36a7e040cd003d50) with some more specifics:

Convert UE4 codebase to an "include what you use" model - where every header just includes the dependencies it needs, rather than every source file including large monolithic headers like Engine.h and UnrealEd.h.

Measured full rebuild times around 2x faster using XGE on Windows, and improvements of 25% or more for incremental builds and full rebuilds on most other platforms.

  * Every header now includes everything it needs to compile.
        * There's a CoreMinimal.h header that gets you a set of ubiquitous types from Core (eg. FString, FName, TArray, FVector, etc...). Most headers now include this first.
        * There's a CoreTypes.h header that sets up primitive UE4 types and build macros (int32, PLATFORM_WIN64, etc...). All headers in Core include this first, as does CoreMinimal.h.
  * Every .cpp file includes its matching .h file first.
        * This helps validate that each header is including everything it needs to compile.
  * No engine code includes a monolithic header such as Engine.h or UnrealEd.h any more.
        * You will get a warning if you try to include one of these from the engine. They still exist for compatibility with game projects and do not produce warnings when included there.
        * There have only been minor changes to our internal games down to accommodate these changes. The intent is for this to be as seamless as possible.
  * No engine code explicitly includes a precompiled header any more.
        * We still use PCHs, but they're force-included on the compiler command line by UnrealBuildTool instead. This lets us tune what they contain without breaking any existing include dependencies.
        * PCHs are generated by a tool to get a statistical amount of coverage for the source files using it, and I've seeded the new shared PCHs to contain any header included by &gt; 15% of source files.

Tool used to generate this transform is at Engine\Source\Programs\IncludeTool.

The tool I’ve used to convert the codebase over is pretty custom, but we’re going to be looking at fixing it up to run over our internal games in the next few weeks. Feel free to check it out if you like, but don’t expect it to work without a lot of tweaking just yet.

In the meantime, the intent was to leave most of the big monolithic headers (Engine.h etc…) alone so as to not break backwards compatibility, but you may run into the odd thing missing.

Mr_SquarePeg · November 26, 2016, 6:12am

This is awesome!

Moss · December 5, 2016, 1:53pm

Oh nice! This will make it easier to maintain our own game code!

anonymous_user_513d2d4e · January 26, 2017, 7:14am

It’s good. Thanks a lot.
I have a question. Why does “iwyu” perform better than using precompiled monolithic headers on full rebuild??

anonymous_user_a7d9c852 · March 23, 2017, 2:40pm

Sounds awesome. How do I activate this feature? I just went through this article
https://docs.unrealengine.com/latest/INT/Programming/UnrealBuildSystem/IWYUReferenceGuide/index.html
and tried to use the
PCHUsageMode.UseSharedPCH
But I can’t include this in my game.build.cs file. Anything I am missing?

Vertex_Soup · July 16, 2017, 3:33pm

Ben Marsh;627943:

I’ve just submitted a change to master branch on GitHub to convert the engine to an “include what you use” model. This is a big change!

Rather than including big headers like Engine.h and UnrealEd.h everywhere, each header now includes just what it needs. There will doubtless be a few teething problems switching over to this, but we’ve been seeing a 20-50% improvement on build times in its staging branch, so hopefully it’ll be worthwhile.

Here’s the commit description (4ba423868f9b6349d8f80a6e36a7e040cd003d50) with some more specifics:

The tool I’ve used to convert the codebase over is pretty custom, but we’re going to be looking at fixing it up to run over our internal games in the next few weeks. Feel free to check it out if you like, but don’t expect it to work without a lot of tweaking just yet.

In the meantime, the intent was to leave most of the big monolithic headers (Engine.h etc…) alone so as to not break backwards compatibility, but you may run into the odd thing missing.

Hi Ben, is there some example how to use this tool convert pre IWYU codebase to required convention? I would like to know this as I spent past day just fighting compile errors and various stuff in many plugins that are required to use this. I would like to enable and convert my project with it to save on typing and pulled hairs… Thank you.

EDIT:
I mean this one: Tool used to generate this transform is at Engine\Source\Programs\IncludeTool

Zeblote · July 16, 2017, 8:34pm

I’m also interested in running include tool for our game project. We’re making quite the mess here with all those fine includes… half of them probably unnecessary…

Ben_Marsh · July 17, 2017, 1:31pm

I think the main change is that this allows us to decouple the contents of precompiled headers from a successful compile. Previously, we manually included precompiled headers everywhere, so it wasn’t easy to remove less used things from them without having to add them to a bunch of places that included them. As part of the dependency analysis that IncludeTool does, we can generate statistically good coverage for the modules that consume it.

Ben_Marsh · July 17, 2017, 1:37pm

I’ve written up some brief documentation on how to use IncludeTool and committed it to the 4.17 branch. The program itself hasn’t changed in a while, so it should apply to older versions too. Use it at your own risk!

https://github.com/EpicGames/UnrealEngine/tree/4.17/Engine/Source/Programs/IncludeTool

Feureau · August 9, 2017, 3:46am

Hi, the link to the github page 404’d. Is the include tool being deprecated or something?

cyaoeu · August 9, 2017, 5:28pm

You need to be logged in and a member of the github Epic Games organization thing.

Zeblote · January 7, 2018, 12:55am

I just tried to use this finally, but can’t seem to get it to work. I installed the cross compile toolchain for 4.18, copied my project to a subfolder of UnrealEngine for it to find it, but the command from the readme doesn’t work:


C:\UnrealEngine>Engine\Binaries\DotNET\IncludeTool.exe -Mode=Optimize -Target=BrickadiaEditor -Platform=Linux -Configuration=Development -WorkingDir=C:\IncludeToolWorking -OutputDir=C:\IncludeToolOutput -SourceFiles=-/Engine/... -OptimizeFiles=-/Engine/... -OutputFiles=-/Engine/...
Caching contents of C:\UnrealEngine...
No files found after running source file filter

[USER=“1345”]Ben Marsh[/USER] am I missing something here? Does it have to be run from a specific folder?

Wolflight · July 17, 2019, 11:40pm

Just discovered this post been messing round with this trying to get it to work in 4.22, been getting a bunch of headaches, thinking it might be easier to downgrade my project to 4.17, without opening the editor and breaking assets, and getting it working that way, posted an answerhub question about my progress, which at least gets as far as trying to create the dependency model but then explodes trying to resolve the build hash defines, which I’m assuming were not there in 4.17? anyway heres the answerhub link, InputTool commandline args - Community & Industry Discussion - Unreal Engine Forums