I realize this is a very big job, and if it was considered a priority would have been tackled a long time ago, but I’m going to bring it up anyway. It causes me hassle every day and I’ve no doubt it does to most anyone writing C++ with UE4, whether or not they’re aware of the underlying issue.
So to be blunt, the engine includes are a total mess. Specifically, I’d say there are two primary problems - overreliance on precompiled headers, and lack of organization of public headers.
Precompiled headers
UBT’s enforcement of putting the PCH include before all others leads to the problem of engine headers reliant on includes within the PCH in order to compile. This is rife throughout the engine, to the point where there are many headers that don’t bother to include any dependencies whatsoever, but just assume the declarations they rely on have been included via the module’s PCH.
The result is bizarre compilation errors deep inside engine code when using an engine header in a module that doesn’t happen to have all the same includes within its PCH as the module in which the engine header was developed. This leads to trawling through the API reference/code to find in what header the offending type is declared, then working out whether that header should be included directly or indirectly via some other, etc. Not only does this happen during active development, but also regularly when a new engine version is released, due to modifications made to engine headers/PCHs.
Public headers
Given the size of the engine, clearly it’s not reasonable to specify low level individual includes for every single thing needed. That said, without a well organized and documented set of ‘batch include’ headers that just wrap includes to related types, the fallback in practice is including high-level headers that end up pulling in half the engine via nested includes. I can’t imagine this is good for compile times even with PCH use.
Furthermore, it means the PCH files themselves end up huge. A PCH that includes only ‘Engine.h’ comes out at around 1GB. With plugins with multiple modules becoming more and more widely used, disk usage is becoming a big problem because of this. Simply disabling PCH generation doesn’t help as the object files all end up much bigger instead. The fundamental problem is that most code has no use for 95% of what gets included via ‘Engine.h’, so most of the disk usage is wasted.
What could help
I don’t know if perhaps the PCH having to come first is based on some important reason relating to compilation times. Assuming it’s something that can’t really be changed, a preprocessor define or UBT flag that suppressed PCH use could at least aid in verifying self-contained headers. Engine modules could periodically be built with this enabled to catch occurrences of PCH-reliant headers. Obviously, removing all the existing such cases would be a substantial job.
As for the second issue, I think a refactor of public headers is needed. Ideally there would be a set of documented headers for external use that consisted purely of other include statements. For smaller modules there could just be a single module header, but generally it would need breaking down into individual headers for logically related sets of functionality. This would aid both in cutting down the amount of code included by our own headers, as well as in controlling what went into our module PCH files so as to save on disk use.