UnrealBuildAccelerator issues with link and cache

Development Pipeline & Plugins
, ,
1

Hello,

I’m trying out UBA on 5.5.

I’ve setup a test server which also runs UbaCache.exe, and two workers.

So far, distribution and cache seem to work and I’m pleased I did not have big issues. Thanks for the existing documentation pages, even if they are not complete obviously, they are already really helpful.

I have two issues that I don’t know if they are from something on our end or not.

  1. Link error LNK1168

When UBT needs to rebuild some modules, for example just by touching a source.cpp file, it ends up with systematic link errors such as this one:

LINK : fatal error LNK1168: cannot open C:\dne\monorepo-preview\Game\TEST\Binaries\Win64\UnrealEditor-TEST.dll for writing

The files exist but do not appear to be locked.

Cleaning and then doing a full build from start does work and builds a working editor.

2.Cache root path issues

During build, caching works except for our in-house plugins and games. I suspect this is because we patched the engine to store them outside of UE’s root directory.

This happens with a lot of logs about PATH/FILE WITHOUT ROOT:

UbaCacheClient - PATH WITHOUT ROOT: C:\dne\monorepo-preview\DNE\Plugins\GameSystem\DNEDialogue\Source\DNEGenDialogueEditor\Public\DN (inside C:\dne\monorepo-preview\Game\TEST\Intermediate\Build\Win64\x64\UnrealEditor\Development\TESTEditor\TESTEditorModule.cpp.d at offset 147677)

UbaCacheClient - PATH WITHOUT ROOT: C:\dne\monorepo-preview\DNE\Plugins\Core\DNECondition\Intermediate\Build\Win64\UnrealEditor\Inc\ (inside C:\dne\monorepo-preview\Game\TEST\Intermediate\Build\Win64\x64\UnrealEditor\Development\TESTEditor\TESTEditor.Shared.rsp at offset 17)

UbaCacheClient - FILE WITHOUT ROOT: C:\dne\monorepo-preview\DNE\Plugins\GameSystem\DNEDialogue\Source\DNEGenDialogueEditor\Public\DNEAssetTypeActionDialogue.h (TESTEditorModule.cpp (Compile [x64]))

UbaCacheClient - PATH WITHOUT ROOT: C:\dne\monorepo-preview\DNE\Plugins\Core\DNECondition\Intermediate\Build\Win64\x64\UnrealEditor\ (inside C:\dne\monorepo-preview\Game\TEST\Intermediate\Build\Win64\x64\UnrealEditor\Development\TESTEditor\UnrealEditor-TESTEditor.dll.rsp at offset 1603)

and also happens inside "CmdKey"s or .lib files.

When looking at the related files I can see it points to full paths, is there something I can configure anywhere to avoid UBA getting confused?

Or would you say our patch introduces full paths, but the system expects only relative ones?

I understand this has to do with sharing cache artifacts between workspaces in different locations, but I saw that VFS was to solve this in 5.6.

2

Hey there Antoine,

I’d think that this is likely related to code existing outside of the root. VFS could probably address this, but I’m not confident. Do you happen to have some of your patches to the engine handy for me to review?

A couple of related, but minor questions:

  • Can you provide your BuildCofniguration.xml and command line?
  • Can you provide the initiating machines UBT log?
  • Does this work without caching?

Kind regards,

Julian

3

Hey Antoine,

And one final question, does it work when not using UBA? Quite related documentation around debugging UBA - try -NoUBA or -NoUBALocal

Whilst this is likely related to your patch - we do have an unwritten policy where we want UBA to reflect the local workflow; so if it works in local we should see what’s going on in the UBA case.

Regards,

Julian

4

Hey Antoine,

Thanks for the details here.

I’d really recommend rebuilding UBA under debug config, and collecting the related logs here so that I can send this off to one of the subject matter experts in this area.

Julian

5

Hey Antoine,

Unfortunately I don’t see the patch attached (just the logs - and thanks for that).

>You can see a linker error that can’t open the DLL. However the file exists.

  • Good to know; [mention removed]​ do we have any thoughts on this? So it seems like there are some patches (aforementioned - still need to obtain the code), but it sounds like we have divergences that allow us to work outside of the project root (this is similar to AdditionalDirectories), so I expect the issue is coming from this.

It’s a stretch, but if you fix the spaces, does link start working? I’m curious why I’m not seeing the file not found error with respect to the headers in the logs.

Kind regards,

Julian

6

>I fixed the include directives already, they were not related to these link issues.

Good stuff - thanks for clarifying.

And apologies Antoine - I missed that.

I’ll see if I can reproduce this locally.

Kind regards,

Julian

7

Hey there Antoine,

Just a heads up that I’m still working to reproduce this locally. I’ll let you know how it goes, as I’ve decided to step back and try to get it working through the AdditionalPluginDirectories route first, as opposed to trying to reimplement local divergences.

Kind regards,

Julian

8

Hey there Antoine,

Thank you very much for your patience - I’ve been pretty backed up with other work the last couple days due to internal conferences.

So I’ve tried at great length to reproduce this with the AdditionalPluginDirectories, and I have yet to reproduce this as such (I’ve migrated over 100+ plugins to various external folders - mapped via uproject AdditionalPluginDirectories). After reviewing the breadth of the divergence, I can only assume that this issue is likely emanating from such changes. UBT and UBA do have some coupling together. That being said, I would still expect a raw UBACli.exe command to work here. Would it be possible for you to run your command above:

UbaCli.exe -workdir=C:\dne\monorepo-preview\UE\Engine\Source -cache="192.168.3.127" local "C:\Program Files (x86)\Microsoft Visual Studio\2022\BuildTools\VC\Tools\MSVC\14.38.33130\bin\Hostx64\x64\link.exe" @C:\dne\monorepo-preview\DNE\Plugins\Core\DNEAnimationCore\Intermediate\Build\Win64\x64\UnrealEditor\Development\DNEAnimationCoreRuntime\UnrealEditor-DNEAnimationCoreRuntime.dll.rsp

… with debug UBA binaries (as described here). This should at least help with verbosity. This could be somewhat related to the files being pulled in from out of sync root - but again I’d expect detours to help with that (unless we are missing some type of virtualization/map update [mention removed]​ ).

9

Hey Antoine,

A couple of follow up suggestions to see if we can further narrow things down *from the divergence*.

Can you try:

  • Getting the new binaries for UBA from ue5-main (they should be backwards compatible)
  • Try running with uba (no cache) against BlankProgram
  • In the failure case please provide the session logs
  • Are you utilizing any kind of symlink?

Julian

10

Hey Antoine,

Sounds good - I’ll keep it parked in “open”.

Kind regards,

Julian

11

Hi Antoine,

Julian forwarded these things to me and I must say it looks very strange. What version of windows are you on? I can see that the code in uba potentially is wrong but I don’t know how to reproduce this. Do you have special command line options when building BlankProgram?

For more details… this is what is weird in your log file. It calls CreateFileW on BlankProgram.exe with “read”.. and succeeds.. and then it closes it and opens it again with “read”. This is what cause the “ALREADYEXISTS” weird error.. which I will look into how that can happen.

D CreateFileW ..\Binaries\Win64\BlankProgram.exe RPC_MESSAGE CreateFile T NtCreateFile 300004 (C:\dne\monorepo-preview\UE\Engine\Binaries\Win64\BlankProgram.exe) -> Success D GetFileInformationByHandle (file) 300004 (#) -> Success (size: 16588800) D NtClose 300004 (#) -> Success D CreateFileW ..\Binaries\Win64\BlankProgram.exe D NtCreateFile ALREADYEXISTS (C:\dne\monorepo-preview\UE\Engine\Binaries\Win64\BlankProgram.exe) -> Error

But also, the question is why it opens it for read multiple times in your log but not in mine… in my build (same sdk and everything) it opens the file for write straight away.. here’s the same place in the log on my machine

D CreateFileW ..\Binaries\Win64\BlankProgram.exe T NtClose 512 (UNKNOWN) -> Success D NtCreateFile (MEMORY) WRITE 300003 (E:\dev\fn\Engine\Binaries\Win64\BlankProgram.exe) () -> Success D CreateFileMappingW (MEMORYFILE) File 300003 Protect 4 Size 2097152 (E:\dev\fn\Engine\Binaries\Win64\BlankProgram.exe) -> 300004 D MapViewOfFileEx 300004 Size 2097152 (E:\dev\fn\Engine\Binaries\Win64\BlankProgram.exe) -> 0x20b55df0000 T NtClose 300004 (E:\dev\fn\Engine\Binaries\Win64\BlankProgram.exe) -> SuccessI will see if I can repro the bug on my machine with a unit test and come back to you here.

12

I’ve investigated further and only explanation I can see is that this is a CreateFileW with GENERIC_WRITE and OPEN_EXISTING and there is an unimplemented path in uba for that in combination with “output files”.. “.exe” is an tracked as an output file from the linker.

I still don’t understand why you get this while I have never seen this with the 100s of devs using this daily.. but I will add code to cover this case. Only thing that sucks is that when you link remotely your path will need to download the old .exe file just to then overwrite it :slight_smile:

13

wait, do you have incremental linking or something like that explicitly enabled? that definitely could explain this.

14

aha, there we have it.. I get repro when I enable incremental linking.. at Epic no-one has incremental linking enabled

15

ok, I have submitted a fix for this on our devline. It will most likely not be backported anywhere and I might have to fight to get this into 5.6 since we have locked the release down.

Since binaries are backward compatible you can take code from latest Github in ~2 hours and build the binaries locally.. that should work fine.

16

this was the fix but it could be a bit hard to take it in isolation.. better to take it all

https://github.com/EpicGames/UnrealEngine/commit/a8fbe8545f567b7ce84b0f77ae55ad3b3abb4596

17

That is a good question. It was turned because of

  1. Files are getting larger.. and Fortnite is really struggling with pdb/exe/dll sizes in general
  2. Because of pch and large unity files the incremental linking often get more or less entirely invalidated and then the link time actually is longer than linking from scratch.

so in some situations it is nice to use incremental linking.. but the disadvantages outweighed the advantages in our case.

18

Thanks Henrik, Julian,

Do you have any further recommandations/ideas about the PATH WITHOUT ROOT issue?

I’ve looked at the code in Uba tools and it looks like maybe it can be an easy fix, but I don’t know the general architecture of Uba inside out. Maybe Uba is just missing another root path in its internal list, that would account for our DNE plugins which are stored in an unconventional place, because of our patch (previously shared)?

My goal is to determine if I can modify our patch somewhat easily, or if we need to completely drop it and use AdditionalPluginDirectories instead (larger endeavor).

19

[mention removed]​ I’m not sure if you have any feedback regarding the above?

As a more general statement, I’d tend to urge folks to move away from larger divergence in tooling as it can lead to other unpredictable behaviour in dependent systems, but I can appreciate that it’s a larger effort to sunset said divergence in favour of AdditionalPluginDirectories.

20

oh sorry, I missed that one…

So the PATH WITHOUT ROOT means that there are references out to files that are not registered as roots to the build system.. this means that the cache entries will contain local paths which are likely not compatible with other machines unless they have identical paths.. This is how the cache system works.. it turns absolute paths into relative paths from closest root.

Sometimes, some modules refer out to external things.. in that case you must declare that in the .build.cs file. It is currently limited to only one root but you set it with “ExtraRootPath”

So looking at SuperLuminal.build.cs as an example.. it adds the “SuperLuminal” root and then register the local path to it… this means that we can go in the other direction on another machine when downloading from cache.

See if you can figure out what this is. To me it looks like DNEDialogue has some external dependencies that needs this ExtraRootPath.

Good luck :slight_smile:

/h