Announcement

Collapse
No announcement yet.

Iwyu

Collapse
X
  • Filter
  • Time
  • Show
Clear All
new posts

    Iwyu

    I've just submitted a change to master branch on GitHub to convert the engine to an "include what you use" model. This is a big change!

    Rather than including big headers like Engine.h and UnrealEd.h everywhere, each header now includes just what it needs. There will doubtless be a few teething problems switching over to this, but we've been seeing a 20-50% improvement on build times in its staging branch, so hopefully it'll be worthwhile.

    Here's the commit description (4ba423868f9b6349d8f80a6e36a7e040cd003d50) with some more specifics:

    Convert UE4 codebase to an "include what you use" model - where every header just includes the dependencies it needs, rather than every source file including large monolithic headers like Engine.h and UnrealEd.h.

    Measured full rebuild times around 2x faster using XGE on Windows, and improvements of 25% or more for incremental builds and full rebuilds on most other platforms.

    * Every header now includes everything it needs to compile.
    * There's a CoreMinimal.h header that gets you a set of ubiquitous types from Core (eg. FString, FName, TArray, FVector, etc...). Most headers now include this first.
    * There's a CoreTypes.h header that sets up primitive UE4 types and build macros (int32, PLATFORM_WIN64, etc...). All headers in Core include this first, as does CoreMinimal.h.
    * Every .cpp file includes its matching .h file first.
    * This helps validate that each header is including everything it needs to compile.
    * No engine code includes a monolithic header such as Engine.h or UnrealEd.h any more.
    * You will get a warning if you try to include one of these from the engine. They still exist for compatibility with game projects and do not produce warnings when included there.
    * There have only been minor changes to our internal games down to accommodate these changes. The intent is for this to be as seamless as possible.
    * No engine code explicitly includes a precompiled header any more.
    * We still use PCHs, but they're force-included on the compiler command line by UnrealBuildTool instead. This lets us tune what they contain without breaking any existing include dependencies.
    * PCHs are generated by a tool to get a statistical amount of coverage for the source files using it, and I've seeded the new shared PCHs to contain any header included by > 15% of source files.

    Tool used to generate this transform is at Engine\Source\Programs\IncludeTool.
    The tool I've used to convert the codebase over is pretty custom, but we're going to be looking at fixing it up to run over our internal games in the next few weeks. Feel free to check it out if you like, but don't expect it to work without a lot of tweaking just yet.

    In the meantime, the intent was to leave most of the big monolithic headers (Engine.h etc..) alone so as to not break backwards compatibility, but you may run into the odd thing missing.

    #2
    This is awesome!

    Comment


      #3
      Oh nice! This will make it easier to maintain our own game code!
      Sr. Engine Programmer @ www.playspace.com - moritzwundke.com
      Remember: be polite and respect other peoples opinions - Join the Unofficial Unreal Discord Channel - Found a bug? Then use the Bug Report Form to get it fixed ^^

      Comment


        #4
        It's good. Thanks a lot.
        I have a question. Why does "iwyu" perform better than using precompiled monolithic headers on full rebuild??

        Comment


          #5
          Sounds awesome. How do I activate this feature? I just went through this article
          https://docs.unrealengine.com/latest...ide/index.html
          and tried to use the
          PCHUsageMode.UseSharedPCH
          But I can't include this in my game.build.cs file. Anything I am missing?

          Comment


            #6
            Originally posted by Ben Marsh View Post
            I've just submitted a change to master branch on GitHub to convert the engine to an "include what you use" model. This is a big change!

            Rather than including big headers like Engine.h and UnrealEd.h everywhere, each header now includes just what it needs. There will doubtless be a few teething problems switching over to this, but we've been seeing a 20-50% improvement on build times in its staging branch, so hopefully it'll be worthwhile.

            Here's the commit description (4ba423868f9b6349d8f80a6e36a7e040cd003d50) with some more specifics:



            The tool I've used to convert the codebase over is pretty custom, but we're going to be looking at fixing it up to run over our internal games in the next few weeks. Feel free to check it out if you like, but don't expect it to work without a lot of tweaking just yet.

            In the meantime, the intent was to leave most of the big monolithic headers (Engine.h etc..) alone so as to not break backwards compatibility, but you may run into the odd thing missing.
            Hi Ben, is there some example how to use this tool convert pre IWYU codebase to required convention? I would like to know this as I spent past day just fighting compile errors and various stuff in many plugins that are required to use this. I would like to enable and convert my project with it to save on typing and pulled hairs... Thank you.

            EDIT:
            I mean this one: Tool used to generate this transform is at Engine\Source\Programs\IncludeTool

            Comment


              #7
              Originally posted by Vertex Soup View Post
              Hi Ben, is there some example how to use this tool convert pre IWYU codebase to required convention? I would like to know this as I spent past day just fighting compile errors and various stuff in many plugins that are required to use this. I would like to enable and convert my project with it to save on typing and pulled hairs... Thank you.

              EDIT:
              I mean this one: Tool used to generate this transform is at Engine\Source\Programs\IncludeTool
              I'm also interested in running include tool for our game project. We're making quite the mess here with all those fine includes... half of them probably unnecessary...

              Comment


                #8
                Originally posted by changmin View Post
                It's good. Thanks a lot.
                I have a question. Why does "iwyu" perform better than using precompiled monolithic headers on full rebuild??
                I think the main change is that this allows us to decouple the contents of precompiled headers from a successful compile. Previously, we manually included precompiled headers everywhere, so it wasn't easy to remove less used things from them without having to add them to a bunch of places that included them. As part of the dependency analysis that IncludeTool does, we can generate statistically good coverage for the modules that consume it.

                Comment


                  #9
                  Originally posted by Zeblote View Post
                  I'm also interested in running include tool for our game project. We're making quite the mess here with all those fine includes... half of them probably unnecessary...
                  I've written up some brief documentation on how to use IncludeTool and committed it to the 4.17 branch. The program itself hasn't changed in a while, so it should apply to older versions too. Use it at your own risk!

                  https://github.com/EpicGames/UnrealE...ms/IncludeTool

                  Comment


                    #10
                    Originally posted by Ben Marsh View Post
                    I've written up some brief documentation on how to use IncludeTool and committed it to the 4.17 branch. The program itself hasn't changed in a while, so it should apply to older versions too. Use it at your own risk!

                    https://github.com/EpicGames/UnrealE...ms/IncludeTool
                    Hi, the link to the github page 404'd. Is the include tool being deprecated or something?

                    Comment


                      #11
                      Originally posted by Feureau View Post
                      Hi, the link to the github page 404'd. Is the include tool being deprecated or something?
                      You need to be logged in and a member of the github Epic Games organization thing.

                      Comment


                        #12
                        I just tried to use this finally, but can't seem to get it to work. I installed the cross compile toolchain for 4.18, copied my project to a subfolder of UnrealEngine for it to find it, but the command from the readme doesn't work:

                        Code:
                        C:\UnrealEngine>Engine\Binaries\DotNET\IncludeTool.exe -Mode=Optimize -Target=BrickadiaEditor -Platform=Linux -Configuration=Development -WorkingDir=C:\IncludeToolWorking -OutputDir=C:\IncludeToolOutput -SourceFiles=-/Engine/... -OptimizeFiles=-/Engine/... -OutputFiles=-/Engine/...
                        Caching contents of C:\UnrealEngine...
                        No files found after running source file filter
                        Ben Marsh am I missing something here? Does it have to be run from a specific folder?

                        Comment

                        Working...
                        X