Random LNK2011 error involving SharedPCH

Hello,

We are currently encountering random LNK2011 errors on our build machines when building our game targets (mostly Test builds, but sometimes Shipping builds) :

LNK2011: precompiled object not linked in; image may not run

For context :

  • UE 5.7.3
  • BuildGraph
  • UBA (no UBA cache service)
  • Incremental builds (no Intermediates cleanup)

We tracked down that the file involved is : SharedPCH.Engine.Project.ValApi.ValExpApi.Cpp20.h.obj

What we observe :

  • The first build of a given configuration always succeeds
  • The issue sometimes appears on subsequent builds of the same configuration
  • When it happens, the SharedPCH… obj file is overwritten and interesting fact, the timestamp of this “new” obj is always older than the one generated from the first build
  • This “new” obj file is linked in resulting in the LNK2011 error.

This issue only occurs on our build machines, we can’t reproduce it locally and there’s no Engine change in between the successful and failing builds.

Our build pipeline never manipulates Intermediates

We tried cleaning Intermediates and UBA local cache of all build machines without success

We also tried to update UBA binaries to the binaries of 5.8 preview but the issue remains.

We have yet no idea about the exact time the sharedpch obj is overwritten and where this new obj comes from (as it is older to the first sharedpch obj built for the given config on the build machine)

Did you already encounter a similar issue ?

Can UBA be reintroducing stale artifacts in any way ?

Any insight would be greatly appreciated

Thanks

[Attachment Removed]

Hi,

I have never heard of issues like this before. Are you saying that the pch obj file is written even when the pch build action is not running? something else overwriting it? and this overwritten obj file, does it look normal?

Has this always happened for you? or is it a new thing?

If you don’t use the Uba cache service then afaik, there is no way old obj files could be introduced. Also strange with the time stamp of the obj file. How much older is it?

Is it always that ValApi.ValExpApi file?

[Attachment Removed]

Hey Henrik,

Yes exactly, the pch is overwritten when the pch build is not running.

We still have no idea when exactly the overwrite happen. The “new” obj looks normal indeed but fails to link so it probably come from another build config…

This started happening recently, we switched to BuildGraph and UBA (we were using sn-dbs before) nearly at the same time so I don’t know if it’s because of one or the other. But we never had a repro locally so I’m looking more in depth in our BuildGraph scripts

The pch can be older by 30mn to more than a day which is strange because it’s always older than the first build on a given configuration when starting from a clean state (no Intermediates)

It’s always SharedPCH.Engine.Project.ValApi.ValExpApi.Cpp20.h.obj though

[Attachment Removed]

Update : We figured it out

The pch obj was added as an artifact of the compilation step and if the next step was executed by another worker, it took the obj compiled by the first worker, overriding the obj it could already have compiled.

We will fix our BuildGraph script :slight_smile:

[Attachment Removed]

ah :slight_smile: Thanks for updating me

[Attachment Removed]

We are seeing a similarly random issue that also only happening on our builders:

Module.HoudiniEngineRuntime.5.cpp.obj : error LNK2011: precompiled object not linked in; image may not runUE5.7.4 (also saw it on 5.6)

BuildGraph

UBA

Incremental builds

Horde

It’s happening in our game compile, which is only dependent on set version and build tools in our build graph, which I don’t think have any overlap in the artifacts.

[mention removed]​ If you have any more info on how you debugged your issue, that would be appreciated.

[Attachment Removed]

Hello,

I work with Corentin, and I can explain how we identified and fixed the issue.

We investigated the contents of the Build Graph artifacts. Since we use the Unreal Engine plugin for TeamCity, these artifacts are stored on the NAS.

While inspecting a compilation node artifact, we noticed that some files from the Intermediate directory were included:

[Image Removed]

This means that later, when another worker retrieves this artifact, for example to package the game, it also imports those Intermediate files.

During a subsequent, separate compilation step, this can cause the worker to fail with the error LNK2011

To fix this, we changed the node so that it does not produce the artifact only from the full node declaration. Instead, we explicitly filter out files from the Intermediate directory:

<Node Name="$(CompileNodeName)" Produces="#$(Platform)$(TargetConfiguration)Binaries" Requires="$(CompileRequires)">
 
    <Compile Target="$(ProjectName)" Project="$(UProject)" Platform="$(Platform)"
 
        Configuration="$(TargetConfiguration)"
 
        Arguments="$(ExtraProjectCompileArguments)"
 
        Tag="#$(Platform)$(TargetConfiguration)Artifacts"/>
 
    <Tag Files="#$(Platform)$(TargetConfiguration)Artifacts" Except=".../Intermediate/..." With="#$(Platform)$(TargetConfiguration)Binaries" />
 
</Node>

So later worker do not receive Intermediate files.

[Attachment Removed]

Great. Is there anything needed to go back to Epic? I don’t know TeamCity. Is this a plugin Epic provides somehow?

[Attachment Removed]

Since implementing Warren and Corentin’s fix in our buildgraph we’ve not seen this issue again.

Could be useful if the compile task had the same filter and except options that the tag task has to save us the extra step.

[Attachment Removed]

Can you expand on that last sentence. You want to hook something into the UBT c# code to decide if something should run or not?

[Attachment Removed]

Currently in build graph to stop accidentally transferring the intermediate files we have to do this:

<Compile Target="$(ProjectName)" Project="$(UProject)" Platform="$(Platform)" Configuration="$(TargetConfiguration)" Arguments="$(ExtraProjectCompileArguments)" Tag="#$(Platform)$(TargetConfiguration)Artifacts" />
 
<Tag Files="#$(Platform)$(TargetConfiguration)Artifacts" Except=".../Intermediate/..." With="#$(Platform)$(TargetConfiguration)Binaries" />

and then ensure we only use “#$(Platform)$(TargetConfiguration)Binaries” as the requires field in the rest of the nodes

It would be useful to have the “Except” functionality from the Tag task included in the Compile task to avoid the extra step and stop and files being inadvertently passed around if we use the node name in the Requires field instead of the tag. Like this:

<Compile Target="$(ProjectName)" Project="$(UProject)" Platform="$(Platform)" Configuration="$(TargetConfiguration)" Arguments="$(ExtraProjectCompileArguments)" Tag="#$(Platform)$(TargetConfiguration)Binaries" Except=".../Intermediate/..." />

This is in no way urgent though, the first option works fine.

[Attachment Removed]

Ok, I have no idea what these xml things are, it is outside my knowledge :slight_smile:

If there are things you need in UBT/UBA to enable something else, then maybe I can help, otherwise happy that you figured it out :slight_smile:

[Attachment Removed]