Using UGS with prebuilt from source Unreal engine

Development Pipeline & Plugins
, ,
1

We are currently using UGS in our studio to help artists and content creators to work without having to build the editor. The setup we use is based of the documentation from Perforce U (link here), i.e:

  1. We have a from source version of the engine kept in a perforce stream called engine-mainline.
    1. This from source version is currently coming from git.
  2. Our game is stored in a stream game-mainline, which has a mapping to pull in the engine-mainline source.

The folder layout is:

root

|__Engine

|__… (Other folders such as Samples, LocalBuilds etc.)

|__Game

|__ <game source and assets here>

When we build the UGS target, we use the default build graphs from `Engine/Build/Graph/Examples` and the Node `Submit To Perforce For UGS`.

This step appears to require the engine source code to be present in order to build.

We intentionally rarely make changes to the engine source code to make it easier for us to upgrade our game to support new engine verions and therefore shouldn’t have a need to build the engine very often. Furthermore our CI is currently required to sync the engine source code even if no changes are made to the engine on a clean build.

This had us think whether it would be possible to reconfigure our streams and UGS such that:

  1. We build the engine source code into binaries alone using a dedicated CI pipeline.
  2. We then upload the binaries to our perforce binaries to an artifact store or perforce stream.
    1. Note that this would only happen if there are changes to the engine source code.
  3. When a new change is submitted to the game source code, the CI downloads the engine binaries from the artefact store, use these to build the UGS editor target, and then publish the prebuilt engine binaries, with the UGS editor target binaries, and upload this to the stream in perforce where UGS pulls it pre built binaries from for the new version of the game build.

Note: this is different to the default usecase of PCB for UGS, as rather than building the engine binaries when building the editor target, we only use the prebuilt engine binaries, saving us both in terms of build times, and perforce sync times on our build agents.

We have scanned the documentation, but haven’t been able to find anything indicating that this can be done, however, it seems like something that should be possible to do.

So to the question, can this be done? and if so, what would we have to consider when setting this up from a build graph and UGS perspective?

Hopefully the UGS tool doesn’t necessitate the engine source to be there to build and package for the UGS editor target, but haven’t had the chance to dig deeper into the code to see if there are assumptions built into UGS that would prevent the approach above.

Lastly, has anyone else attempted this in the past?

2

I did the “cached Engine binaries” pipeline with BuildGraph and TeamCity for a project that was based on 5.3, then updated to 5.4. This was before Horde’s GA release in 5.5. It was effectively an implementation of Horde’s BuildGraph semantics, as it was in 5.4, massaged to work with TeamCity’s systems.

In theory, you can do something similar in Horde with a custom task that unpacks the appropriate binaries. Speaking from experience, I do not recommend going down this route for the sake of buildtime improvements:

  • The best way to produce small Editor binaries is to only include plugins that are used by the project; transfer and unpack time for something as big as the Editor is significant. This means the project definition and sources should be available during an Editor build.
  • The validity of cooked assets depend on having confidence that the tools used for cooking are at its intended version. This very quickly devolves into wanting to have the Engine source files for validation during cook and package.
  • UBT is very good with avoiding unnecessary compilation when using persistent workspaces. Our “no-op” editor builds typically take less than 40 seconds, around half of which is coordination and snapshotting overhead.

As for some concrete suggestions for increasing build speed:

  • Take a good study of BuildGraph, and write your own pipelines according to the project’s needs. Pay special attention to how dependencies are transferred between Agents, as this transfer time can dominate overall build time if done carelessly.
  • Consider excluding debug info files from being passed between agents.
  • Use UBA and Zenserver for caching intermediate artifacts.
3

Hi,

Another option would be to use Installed Builds for the PCBs of the Engine. You would need to move the project outside the Engine root. The main problem is that the tools are not built to handle the project PCBs separately from the Engine’s. That would also make the distribution and alignment of both PCB packages.

You do need a part of the engine source, ie the public headers, in order to build the project. This is another thing to consider. You wouldn’t save much by getting rid of the Private Engine code.

I would recommend that you stick to the current flow as building what you describe would require lots of efforts and the benefits don’t seem to be that much.

Regards,