distributed compilation distcc ice cream etc

[uberthoth]

What is the current status of distcc with Unreal Engine?

There used to be a build configuration bAllowDistccLocalFallback

But this term no longer exists in the documentation according to the search mechanism.

There a few news posts.

Everything else seems to be forum posts from 2015 or earlier

There is a big discussion here about the topic in general:

UBT as a distributed compilation tool

It would also apply to packaging a project, which works via UBT as well. Some of us re-package quite often to test things, and spend lots of time waiting for compilation (even when just compiling some game code). Considering it would be working over LAN, the transfer, at least, wouldn’t take much time. I’d be very interested in seeing Epic’s opinion on this, regardless.

I tried to adapt what amigo has said here:

Distcc

Should probably also mention: I’m setting the options for UnrealBuildTool in the file BuildConfiguration.xml in Engine/Saved/UnrealBuildTool/BuildConfiguration.xml, and my ~/.dmucs/hosts-info looks like this: 10.9.8.58 9 10 10.9.8.26 9 10 10.9.8.22 9 10 I’m using linux if that wasn’t obviuos already, using the 3dluvr patch for UnrealEngine. Any help would be greately appreciated in advance, working without my build accelerators is very painful

But I feel I’m missing more pieces of the puzzle.

I agree completely with the staff here:

@RCL

If someone can get distcc to work, that is probably the single best thing that can speed up the compilation.

Also, ice cream should certain get a mention, they have some reasons why they think you should use ice cream over distcc.

Regardless, I’ve seen both perform admirably over the years at spreading out compilation tasks, you can easily achieve an overall time reduction factor of 10x or more with a small farm.

As an alternative, there is this FastBuild which I have not tried, and will, but at least one other has tried with no success here

Or Incredibuild, which really I find disdainful that they exclude linux from the free version, if anything it should be the other way around with Windows in the paid for with free trial only subscription and Linux in the free version, shame on you Incredibuild, I’ll skip for now.

And just to document some of the older topics that did not seem to go anywhere, but are on this topic:

Linux and distcc

I live in a house with multiple computers. What my fellow room-mates don’t know is that their computers are each host to a linux virtual machine running ubuntu server. I have SSH access to each of these machines, some of them are somewhat powerful. I have distcc installed on all of them, I’m trying to do a distributed compile. The usual… $ distcc-pump make UE4Clinet CC=“distcc clang++” doesn’t seem to be working. It compiles, but distcc-pump doesn’t distribute any work. Any ideas on why this …

Can I distribute shader compiling?

Is it possible to distribute the shader compiling? That’s the part that seems to slow me down the most when working in Unreal. I’ve got a spare system next to my computer that I’m going to be using for source control so if I can distribute some of the processing over to it, that would be helpful.

[uberthoth]

Of useful note, I seem to have gotten Fastbuilds working on linux:

fbuildworker -mode=dedicated                                                                        
FBuildWorker v1.11
Listening on port 31264
0 Connections
1 Connections
0 Connections
1 Connections
0 Connections

I have something like this for all hosts env vars:

export FASTBUILD_WORKERS="10.0.0.66;10.0.0.160;10.0.0.11;10.0.0.18"
export FASTBUILD_BROKERAGE_PATH="/unreal/unrealPJX/FASTBUILD_BROKERAGE"
export FASTBUILD_CACHE_PATH="/unreal/unrealPJX/FASTBUILD_CACHE"
export FASTBUILD_TEMP_PATH="/unreal/unrealPJX/FASTBUILD_TEMP"

where /unreal is an NFS mounted in to all hosts, I am not certain they should be sharing the temp and cache directories like that amongst the hosts or maybe they should be local to each host and not shared. But it is working at the moment. Also, I had to create a directory:

/opt/UnrealEngine/Engine/Extras/ThirdPartyNotUE/FASTBuild/Linux

and place the two files you get from here in there and I had to copy them over to have both the lower case version and the mixed case versions:

\ls -alh
total 7.2M
drwxr-xr-x 2 thoth thoth 4.0K Oct 20 12:24 .
drwxr-xr-x 5 thoth thoth 4.0K Oct 20 12:14 ..
-rwxr-xr-x 1 thoth thoth 1.8M Oct 20 12:24 fbuild
-rwxr-xr-x 1 thoth thoth 1.8M Oct 20 12:14 FBuild
-rwxr-xr-x 1 thoth thoth 1.9M Oct 20 12:24 fbuildworker
-rwxr-xr-x 1 thoth thoth 1.9M Oct 20 12:14 FBuildWorker

FBuildWorker, may be unnecessary as I’m manually starting on each host with something like fbuildworker -mode=dedicated

And, finally, I had to alter my BuildConfiguration.xml in /home/thoth/.config/Unreal Engine/UnrealBuildTool/BuildConfiguration.xml:

<?xml version="1.0" encoding="utf-8" ?>
<Configuration xmlns="https://www.unrealengine.com/BuildConfiguration">
  <BuildConfiguration>
    <bAllowFASTBuild>1</bAllowFASTBuild>
  </BuildConfiguration>
  <FASTBuild>
    <bEnableDistribution>true</bEnableDistribution>
    <FBuildExecutablePath>/home/thoth/.bin/fbuild</FBuildExecutablePath>
    <CacheMode>ReadWrite</CacheMode>
  </FASTBuild>
</Configuration>

And on a few hosts they showed in the brokerage directory as 127.0.1.1, I had to alter their /etc/hosts file to change their line from:

127.0.1.1  host1.example.com   host1

to:

10.0.0.6  host1.example.com   host1

[ScrewdriverHyena (Screwdriver)]

SN-DBS is the way

[uberthoth]

Tell me more! Is there any documentation?

[uberthoth]

update:

my current env vars:

#export FASTBUILD_WORKERS=
export FASTBUILD_BROKERAGE_PATH="/unreal/FASTBUILD_BROKERAGE"
export FASTBUILD_CACHE_MODE="rw"
export FASTBUILD_CACHE_PATH="/unreal/FASTBUILD_CACHE/cache"
export FASTBUILD_CACHE_PATH_MOUNT_POINT="/unreal/FASTBUILD_CACHE"
mkdir -p /tmp/FASTBUILD_TEMP
chmod 777 /tmp/FASTBUILD_TEMP
export FASTBUILD_TEMP_PATH="/tmp/FASTBUILD_TEMP"

1. Notice I am commenting out #export FASTBUILD_WORKERS as the brokerage directory is verified to be working.
2. /tmp/FASTBUILD_TEMP is local to each worker and no longer shared.
3. Unreal is no longer an NFS mount
4. FASTBUILD_CACHE and FASTBUILD_BROKERAGE are individual NFS mounts
5. extremely liberal perms have been given to the tmp directory

EDIT: I am still receiving some errors about fileID like:

Failed to store fileId 1016 for manifest 0x1ff03b798451c16c

I’ve opened an issue with fastbuild, but I get the impression this maybe an issue with unreal and perhaps how it is building the .bff file.