We’ve built a pipeline to avoid every developer/non-dev on the team having to compile the engine from scratch, and I’d like a second opinion before we lean on it further, as we did not find any online resource of people or studios doing that.
Here’s the shape of it:
The git repository contains both the engine (root-level Engine/ folder) and the game (MyGame/), all in one git history, so that we the project is a native project.
Engine\Build\Build.version has CompatibleChangelist = 0 / IsLicenseeVersion = 1 / IsPromotedBuild = 1. Changelist is updated at each engine change (see below)
A Jenkins job builds the Editor on every push to develop, using the buildgraph task “Tag Output Files” from the file BuildEditorAndTools.xml (To build everything locally in the workspace, but don’t copy or publish anything). We increment the Changelist property of Build.version each time and then runs a Python publish script.
The publish pipeline:
The publish script scans files ending with .dll / .exe / .lib / .modules / .target in the folders “Engine/Binaries/Win64”, “Engine/Platforms”, “Engine/Plugins”, “Engine/Intermediate/Build/BuildRules”, “Engine/Intermediate/Build/Win64/**/”, and also the files “Engine/Intermediate/Build/BuildRules/*.json", "Engine/Intermediate/Build/Win64/**/*.h”. We found out that the files from “Engine/Intermediate” are required when using an installed build of the engine (More on that below)
Files are content-hashed (BLAKE3, with a local mtime+size cache to skip re-hashing unchanged files between runs) and diffed against the last published state.
Only changed/added files get zipped and uploaded to S3 as a “delta” tied to the current git commit SHA. A full “checkpoint” archive gets published every N versions so a client that’s far behind can jump forward instead of replaying hundreds of small deltas. Old deltas/checkpoints beyond a retention window get pruned automatically.
The sync script (run via pre-commit hooks on post-checkout/post-merge/post-rewrite, or manually) is executed only on non-developers:
walks the local git ancestry from HEAD, finds the nearest commit that has a published version, and downloads/extracts only what’s missing (checkpoint jump + delta replay) directly into the working tree.
creates a local-only InstalledBuild.txt in Engine/Build so the engine behaves like a precompiled/installed build and they can just compile the game code to start the editor, the engine DLLs are the ones downloaded by the sync script
Programmers have the choice to not run the sync script, and not have this InstalledBuild.txt and UBT just behaves normally and they compile engine + game code.
Is mixing a source-built engine with an InstalledBuild.txt marker (on the client side only, not affecting the actual repo/build) a supported or sane combination, or could it cause side effects that we could run into?
We would appreciate hearing from anyone who’s tried something similar, especially anywhere this kind of setup silently broke down at scale (large teams, long-lived branches, frequent engine updates, etc.).
Hello! Thanks for sharing the write-up of your approach.
Overall, the approach makes sense, but you may be able to lean on some existing UnrealGameSync implementation details to avoid future issues.
In case you weren’t aware, one of the features of UGS is essentially this: providing precompiled binaries (PCBs) of an editor built for a project to content creators who don’t need to work with the code base or rebuild locally to see their changes.
However, the UGS approach requires Perforce, so you won’t be able to set this up directly with a Git project.
One key difference I’ve found between these approaches is that the UGS approach does not use `InstalledBuild.txt`, and PCB workspaces remain as ordinary source builds.
Instead, UGS rewrites `Engine/Build/Build.version` to set `CompatibleChangelist` to the last code change and `IsPromotedBuild` to 0. UGS then skips the engine compile steps altogether.
Another win is that you may be able to use the existing “Stage for UGS” node in the “BuildEditorAndTools.xml” build graph to prepare your files.
This node does actively filter out the “Intermediate” files, but if you switch to the PCB route over the installed build route, you won’t need these files.
Sorry for the long delay, it’s quite time consuming to test those things, and I had to redo all my tests (enabling different flags in Build.version, syncing various files) to give more context on why we think we should try to do things this way.
Using the UGS approach when we sync on non-dev machines works fine on a linear history like it is often the case with Perforce. We can regenerate the binaries each time any source file is updated in the engine or in the game, and people would just pull the updated binaries and could just open the editor.
In our workflow with git we rely heavily on branches from develop, which are reviewed and tested by jenkins, before being merged back to develop. When a non-dev uses a branch which has game code modifications, and opens the uproject, the engine will detect that compilation is needed, and since it never compiled anything, UBT will recompile the whole engine.
By syncing the engine as an InstalledBuild engine, we can just recompile the game with something like
D:\TestsUE\UEGameWithSources\Engine\Binaries\ThirdParty\DotNet\10.0\win-x64\dotnet.exe "D:\TestsUE\UEGameWithSources\Engine\Binaries\DotNET\UnrealBuildTool\UnrealBuildTool.dll" MyGameEditor Win64 Development -Project=D:\TestsUE\UEGameWithSources\MyGame\MyGame.uprojectand then open the project as usual.
We’re still a bit hesitant with the IsPromotedBuild flag which we’re not quite sure is meaningful or not in our case. Installed builds from the launcher are set to 1, but we’re using a custom engine so maybe it’s not relevant for us, and it’s hard to tell exactly how it can impact us.
You’re welcome! No worries about the delay; this is definitely quicker to write about than test and iterate on, and not something we have any direct experience navigating.
If you’re happy with the installed approach, then I’d stick with what’s working.
The IsPromotedBuild flag is useful to keep set to 1 on CI builds, as it gives you a stable build ID rather than a GUID, which would differ on a rebuild of the same code.
It won’t do anything on the client side, as an installed engine clears the flag for anything built locally.