I have some questions regarding how to speed up an automated Unreal build pipeline.
Here’s what our build pipeline looks like. We launch instances from AWS EC2, and run a script on them to handle our builds. This script will pull the project from git, build it, deploy it, then terminate the instance — builds are independent, and data is not persisted between builds. Additionally, the instances have Unreal already installed (pre-installed on the machine image). We have our own flavor of Unreal, and build it from source. Finally, we build for Windows and Android, but for this post, I will focus on the Android build.
One of the things that we are doing to speed up build times is caching project files from previous builds. For example, after finishing a build, we can upload various files/folders to a cloud. On a subsequent build, we can download these cached folders before we start building, to speed up build times. This is the only way data can be shared between builds. Currently, we are caching the project’s Saved, Intermediate, and DerivedDataCache folders.
The unreal build steps are:
Build script.txt (889 Bytes)
Here is where I’m running into some trouble. I noticed that Unreal Engine files get touched as well, so I tried caching the following folders:
unreal/Engine/Intermediate/Build/Android
unreal/Engine/Intermediate/Build/Win64
However, this resulted in what seems like a full recompile of the editor. In the “Building myProjectEditor” step, an excessive number of files are generated, which takes a long time. Here is the output log when we run the following commands:
- .\unreal\Engine\Binaries\Win64\UnrealVersionSelector-Win64-Shipping.exe /switchversionsilent ‘./unrealproject/myProject/myProject.uproject’ ‘.\unreal’
- ./unreal/Engine/Build/BatchFiles/RunUAT.bat BuildCookRun -project=”…/unrealproject/myProject/myProject.uproject” -build -skipcook -compileeditor
Editor Full Recompile Log.txt (598.2 KB)
Additionally, when trying to build the game for Android, the build fails. Here is an output log after running the this step:
- ./unreal/Engine/Build/BatchFiles/RunUAT.bat BuildCookRun -project=“…/unrealproject/myProject/myProject.uproject” -clientconfig=Shipping -distribution -cook -FastCook -SkipCookingEditorContent -CookPartialgc -iterate -build -platform=“Android” -stage -pak -package -stagingdirectory=“…/builds/Android” -cookflavor=ASTC
Failing Android Build Step.txt (230.1 KB)
This doesn’t happen when we don’t cache the Unreal Engine Intermediate folder. Here is the output log for a run that does not cache these folders:
- .\unreal\Engine\Binaries\Win64\UnrealVersionSelector-Win64-Shipping.exe /switchversionsilent ‘./unrealproject/myProject/myProject.uproject’ ‘.\unreal’
- ./unreal/Engine/Build/BatchFiles/RunUAT.bat BuildCookRun -project=”…/unrealproject/myProject/myProject.uproject” -build -skipcook -compileeditor
Compile Editor Reference.txt (56.3 KB)
My questions are:
- What exactly is “myProjectEditor”, and why does it fully rebuild when caching Unreal Engine Intermediate folders.
- How is building “myProjectEditor” different from building “myProject”
- Why is “myProject” built both in the compile editor step, and in the packaging (last) step?
- Are there any Unreal Engine folders we can cache to speed up build times?