Horde fails multi-configuration build task when using ConditionalAddModuleDirectory() to compile optional files.

We are attempting to conditionally compile files (including UObjects) within a Plugin based on the target configuration (dev/shipping etc). Our setup is that we have a base “Debugging” plugin (essentially, an ImGui implementation) which contains a single “DeveloperTools” module, and our project has numerous other plugins/modules within which we want to compile optional files when this base plugin is also compiled for the target configuration.

The standard way to approach this it seems, would be to create “DeveloperTools” modules within those plugins. We haven’t yet attempted this, but the key issue is that this would require the “Runtime” part of the plugin to export symbols purely for the purpose of the debugging tool, which we want to avoid. There is no ability to conditionally export classes/symbols for different build configurations because of UHT.

So instead, we are making use of ConditionalAddModuleDirectory() to append folders when appropriate. The setup of one of our “debuggable” plugins is as follows:

// .UPlugin file - only declares a single “Runtime” module which is always compiled.

MyPlugin.uplugin { "Modules": [ { "Name": "MyPluginRuntime", "Type": "Runtime", "LoadingPhase": "Default" } }// MyPluginRuntime.Build.cs - Conditionally adds the additional paths if the “DebugPlugin” is built.

public MyPluginRuntime(ReadOnlyTargetRules Target) : base(Target) { if (IsPluginEnabled("DebugPlugin")) { string DebugModulePath = Path.Combine(ModuleDirectory, "..", Name + "Debug"); ConditionalAddModuleDirectory(new DirectoryReference(DebugModulePath)); } }// MyPluginDebug.Builds.cs - Adds additional folder paths.

public MyPluginDebug(ReadOnlyTargetRules Target) : base(Target) { PrivateIncludePaths.AddRange(new string[] { "MyPluginDebug/Private" }); }We can see that this is working - logging the folders/files that UBT acquires from UEBuildModuleCPP.cs.Compile, we can see the “Debug” folders are excluded when building shipping, and included for other builds (which is correct for our setup)

Unfortunately, our issue is that when we start a build on our build farm via Horde, the build fails to build non-shipping builds due to missing ###.gen.cpp files and/or missing ###.generated.h files (our debug modules contain UObjects/reflected types). However, this is only the case when we start a task that builds Development/Test binaries as well as Shipping binaries. (Note that we are deliberately including developer/debug tools for test configurations)

If we send the different build configurations to the farm separately (i.e, build shipping first, then build development/debug), the build succeeds. The issue almost seems to be that the machines are sharing compilation units for different build configurations when performing a batched build. Since UHT has (rightfully) not produced generated code for the shipping build, the development/debug builds fail. Presumably, this is a non-deterministic failure based on whichever configuration is built first.

We have also seen issues with incremental builds failing. We tried using pre-processor to essentially comment-out the types but since they are UCLASS, that’s not possible. Is there a canonical way to achieve this that doesn’t involve creating a second, fully-fledged debugging module and exporting otherwise internal types/symbols from our runtime module?

Hey there,

There’s a lot going on here, so I want to make sure I’m taking away the right details for the failure/success path.

Repro Questions:

  • When you build locally from clean for development, everything works?
  • When you build on the buildfarm from clean for development, does it fail?
    • “However, this is only the case when we start a task that builds Development/Test binaries as well as Shipping binaries. (Note that we are deliberately including developer/debug tools for test configurations)”
      • So this issue only occurs when we are building shipping as well? What’s the resulting build graph execution look like? Do you have any logs from Horde to

> to be that the machines are sharing compilation units for different build configurations when performing a batched build

  • There’s nothing explicitly sharing compilation units between job steps on the same branch, that wouldn’t also be the case in a local setup
    • UHT generated code not-withstanding, since it’s not config constrained (target and platform scoped); this could be a hint that UHT isn’t running in your specific circumstance, wherein it should due to the debug module requiring reflected types
  • My hunch here is that Horde is simply highlighting an order of operations issue that would also occur on a local user machine given the *build.cs setup you’ve got here.

Ideas around how we can isolate the issue:

  • I wonder if in your build graph you can force an explicit run of UHT for your failure scenario
    • “Since UHT has (rightfully) not produced generated code for the shipping build, the development/debug builds fail. Presumably, this is a non-deterministic failure based on whichever configuration is built first.”
    • My working hypothesis is that when you do run in the development build, there’s some previous build state (not obj files per say, but the generated cpp/h from uht) from Shipping, which are being used.
      • You could try a clean build on local machine; build shipping, then build development to test the hypothesis
      • If this fails, we can try adding a forceful UBT UHT invocation to the build graph via RunUBT -Mode=UnrealHeaderTool “-Target=<TARGET> <PLATFORM> <CONFIGURATION> -Project=\”<PROJECT_FILE>\“” (or directly to UBT “dotnet UnrealBuildTool.dll -Mode=UnrealHeaderTool “-Target=UnrealEditor Win64 Development””

Kind regards,

Julian

Hey there,

No worries - happy to help as best I can here.

>If we kick off a build of both Development AND Shipping in the same task

  • Specifically, do you have the buildgraph fragment that is issued here? Ordering will really matter here. What I’m thinking is that Shipping builds first - doesn’t emit the necessary UHT; Development runs after; intermediates (not compilation units; things like makefiles or UHT metadata) signal that we don’t need to run UHT for your circumstance; we end up with error.

I mean quite literally just issuing two calls, back to back on a clean. So:

  • dotnet UNrealBuildTool.dll -Target=“TARGETNAME Win64 Shipping” -Clean
  • dotnet UNrealBuildTool.dll -Target=“TARGETNAME Win64 Development” -Clean
  • dotnet TARGETNAME Win64 Shipping
  • dotnet TARGETNAME Win64 Development

At this point, I think we are trying to track down exactly what files are not being explicitly generated. My hunch is that the above will fail for local user as well. Again the build graph details matter a lot for the build farm case where you’re observing both dev and shipping on same ‘task’ resulting in dev build failure.

Kind regards,

Julian

Hey there,

Thanks for this. If you try and reproduce this locally that would be the last remaining detail. [mention removed]​ - do you have any thoughts here on what could be the best path forward for the UHT generation? I think the issue they are seeing here is that the UHT generation isn’t occurring due to some intermediate state.

Kind regards,

Julian

Hey there,

Just spoke with the UHT SME (Tim) on this, and I do think there is some credence to the hypothesis that UHT is not running for the debug configuration - that would generate the aforementioned files. Can you please verify this on your local machine (not horde):

  • Clean your intermediates
  • Build your shipping configuration
    • Copy over UBT log for later
  • Build your debug configuration
    • Copy over UBT log for later

Attach both logs. We want to see whether UHT is running both times (because it would need to, as indicative of your setup). From there, we have a more concise repo that removes variables (Horde - and again I don’t think this is a horde issue; horde makes no claims or intentions around sharing compile intermediates - it’s merely the state of the agent machine in an incremental workspace). ExecuteHeaderToolIfNecessaryInternalAsync becomes more interesting, particularly AreGeneratedCodeFilesOutOfDate.

Julian

Hey Julian, thanks for the detailed response. I’ll try to get the extra information you need.

When you build locally from clean for development, everything works?

  • That’s correct. All configurations will build without issues locally (by that I mean, just changing target in VS and hitting build)

When you build on the buildfarm from clean for development, does it fail?

  • If we build Development on it’s own, it succeeds. If we build Shipping on it’s own, it succeeds. If we kick off a build of both Development AND Shipping in the same task, the Development build fails.

When you say “try a clean build on local machine; build shipping, then build development to test the hypothesis” - do you mean to run the same BuildGraph task on the local machine that we would run on Horde, i.e. build shipping and dev in the same task. Or simply clean the workspace, build shipping, then build development?

---

We decided to experiment by creating those fully-fledged ‘DeveloperTool’ modules for our debug objects and this did actually compile without any issues. The caveat remains though, in that we have to export a lot of symbols/types we wouldn’t otherwise export to achieve this.

So just for clarity now I realise I didn’t include this in the original post, the errors we see are essentially these:

`\OptionalCodeFolder\MyOptionalObject.h(#): fatal error C1083. Cannot open include file: ‘MyOptionalObject.generated.h’: No such file or directory.

\MyProject\Intermediatte\Build\Win64\x64\MyGameClient\Development\MyPlugin\Module.MyPlugin.2.cpp(#): fatal error C1083: Cannot open include file: ‘../../MyProject/Intermediate/Build/Win64/MyGameClient/Inc/MyPlugin/UHT/MyOptionalObject.gen.cpp’: No such file or directory`I’ve been told this is the Agent/Node we use

`


``RunUAT.bat BuildGraph -Script=Game/Build/BuildGame.xml -Target="Publish Targets" -SingleNode="Compile Targets Win64" -Set:GamePlatforms=Win64;Linux;XSX -Set:GameConfigs=Development;Test;Shipping -Set:ServerPlatforms=Win64;Linux;LinuxArm64 -Set:ServerConfigs=Development;Test;Shipping -Set:UseIncrementalCompileAgents=true`Let me know if that's not all the info you need. While scouring through the log, I notice it frequently switches back and forth between the different configurations.

** For MyGameServer-Win64-Shipping **

// do some work

** For MyGameServer-Win64-Test **

// do other work

etc..