Is there a way for Buildgraph nodes to conditionally consume a product of a required node?

Development Pipeline & Plugins
, ,
1

Hey

While investigating how Nodes trigger dependencies, and how produced artifacts trickle down, we have made a pipeline for building and testing but was wondering if there are options we’re missing for skipping fetching build products on dependent nodes defined by ‘Produces’

This is an example we have:

  • Location 1 has our builder machines and our artifact store
  • Location 2 has our tester machines
  • We want a single machine to fetch the build from Location1->Location2 and send it to a local network drive. Then Location 2 testers can pick it up internally and run tests
  • We also want on our testing templates to have the option for asking for a build, so conditionally testers will either pick up an existing build, or ask Location 1 to produce it, wait for it to be relay and then start testing

In the example buildgraph, what will happen is that `BuildRelayer - Location 2` will fetch the build from Location1, but then each of the testers will also fetch it from Location1 through the requirements before executing

The ways we’re solving this right now:

  • Use chainedjobs to break templates into just building and just testing, and have options to trigger the chainedjob

This works well, but to do it we have to have conditional nodes on the builder template buildgraph, that only happen if the ‘Testing’ option is true, and then tie the chainedjobs to that, since chainedjobs don’t have a conditional

  • Use the After keyword for testers, and update a general “RequiredNodes” with the publish/relay jobs

This is what we tended to do when we want order of execution, but don’t want to fetch produced files. This works well but needs to keep track of a full list of required nodes (so now “RunAllTests” becomes Test1;Test2;Publish Windows Client(conditionally) which starts to muddle the definition a bit

  • Remove ‘Produces’ and use Create/Retrieve artifact

This is our current solution and it works great, so builders send explicit artifacts with CreateArtifacts, Relays use RetrieveArtifact to get them and copy them, and then Testers can just require the relay without getting any file implicitly.

The downside of this is that we can’t use Produces with the above, as it brings us back to the original issue, so all dependent nodes use the explicit RetrieveArtifact task.

  • Have the producing node also produce some #BuildReport file that we can depend on instead

This feels very hacky but it works well, so Testers conditionally require #BuildReport instead of the “Build Windows Client”, they respect the order of operations, and cause a publish to trigger

So my questions are:

  • Is there some way to require a node but only conditionally receive its `Produces` artifacts?
  • How will the OptionalRequires linking work in regards to this? We assume it will be like Requires rather than After
  • Is there a way to have multiple artifact stores across locations that replicate artifacts, so that Location2 agents can always look there when using RetrieveArtifact? If so we’d be able to remove the relay part completely
  • Are we maybe doing something wrong in general?

Thank you!

NodeDepedenicesExample.xml(2.62 KB)

2

Steps to Reproduce

<?xml version='1.0' ?>
<BuildGraph xmlns="http://www.epicgames.com/BuildGraph" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://www.epicgames.com/BuildGraph ../Schema.xsd" >
    <!-- The idea here is that our artifact store is on Location 1 where our builedrs are. We also run tests on a remote location
        So we want a single machine to pick up the build from Location 1, send it ta network drive and others to pick it up from there 
        
        We also want the option to either request a new build, or not (expecting that a build is already available)
    -->
 
    <Option Name="RequestBuild" Restrict="true|false" DefaultValue="true" Description="Request a new build to be published before running tests"/>
    <Property Name="TestRequirements" Value="" />
    <Property Name="TestRequirements" Value="Relay Windows Client Build To Location 2" If="$(RequestBuild)"/>
 
    <!-- This agent simulates a build machine that fetched all needed requirements and published a Windows Client Build -->
    <Agent Name="Builder - Location1" Type="UAT" >
        <Node Name="Publish Windows Client" Produces="#WindowsClientBuild">
            <WriteTextFile File="$(RootDir)/ProoducedBuild/WindowsClientBuild.txt" Text="This is a build believe it or not!"/>
            <Tag Files="$(RootDir)/ProoducedBuild/WindowsClientBuild.txt" With="#WindowsClientBuild"/>
        </Node>
    </Agent>
 
    <Property Name="Location2LocalNetworkDrive" Value="\\Location2\Builds\" />
 
    <Agent Name="BuildRelayer - Location 2" Type="Relay" >
        <!-- We expect to fetch the build from Location1, and then send it to a local network drive-->
        <Node Name="Relay Windows Client Build To Location 2" Requires="#WindowsClientBuild">
            <Copy From="$(RootDir)/ProoducedBuild/WindowsClientBuild.txt" To="$(Location2LocalNetworkDrive)/WindowsClientBuild"/>
        </Node>
    </Agent>
 
    <!-- We now want these two testers to run after the Build Relay and copy the build from their local network drive.
         We don't want produced artifacts to reach them
    -->
    <Agent Name="GauntletTester1 - Location 2" Type="Tester" >
        <Node Name="Run Test 1" Requires="$(TestRequirements)">
            <Copy From="$(Location2LocalNetworkDrive)/WindowsClientBuild" To="$(RootDir)/TestEnvironment"/>
 
            <Log Message="Starting Test 1" />
        </Node>
    </Agent>
 
    <Agent Name="GauntletTester2 - Location 2" Type="Tester" >
        <Node Name="Run Test 2" Requires="$(TestRequirements)">
            <Copy From="$(Location2LocalNetworkDrive)/WindowsClientBuild" To="$(RootDir)/TestEnvironment"/>
 
            <Log Message="Starting Test 2" />
        </Node>
    </Agent>
 
    <Aggregate Name="RunAllTests" Requires="Run Test 1; Run Test 2" />
</BuildGraph>
3

Hello, BuildGraph has three distinct dependency mechanisms, each with different artifact behaviour:

  • Requires
    • all produced artifacts fetched
    • hard fail if missing at lease one artifact
  • OptionalRequires
    • artifacts fetched if they exist
    • warning only and continues if missing
  • After
    • no artifact transfer

1. Is there a way to require a node but only conditionally receive its Produces artifacts?

No native mechanism exists for this. The behaviour is hard-wired:

  • Requires always fetches all Produces artifacts from the required node. If the tag file list doesn’t exist, the build throws an exception
  • After never fetches artifacts, it’s ordering-only

There is no Requires variant that says “depend on this node but skip its outputs.” Your current workarounds appear to be correct approaches given this limitation.

2. How will OptionalRequires work regarding artifact fetching?

OptionalRequires does fetch artifacts, similar to Requires, but with try/catch handling. If the optional dependency’s artifacts can’t be found, it logs a warning and continues rather than failing. So OptionalRequires is like Requires in that it fetches artifacts - it just doesn’t fail if they’re missing. It would not solve your problem of skipping artifact fetching from a node that did produce outputs, because if the artifacts exist, they’ll still be fetched.

3. Is there a way to have multiple artifact stores across locations that replicate?

Not in BuildGraph’s temp storage system. Artifact transfer between nodes on different agents goes through a single shared storage location. Your relay pattern is the right approach for cross-location scenarios, or using explicit CreateArtifact/RetrieveArtifact with a storage system you control.

4. Are you doing something wrong in general?

No, The Requires/Produces system couples dependency ordering with data transfer, and there’s no way to decouple them within a single Requires declaration.

Your option 3 (removing Produces, using explicit CreateArtifact/RetrieveArtifact) may be the best solution because it gives you full control over what gets fetched and from where, and separation between ordering (After) and data transfer (explicit artifact ops). The sentinel tag pattern (option 4) is a reasonable approach if you want to keep using Produces for some nodes:

 <Node Name="Publish Windows Client" Produces="#WindowsClientBuild;#BuildReport">
   <!-- ... build steps ... -->
   <WriteTextFile File="$(RootDir)/BuildReport.txt" Text="Build complete"/>
   <Tag Files="$(RootDir)/BuildReport.txt" With="#BuildReport"/>
 </Node>
 
 <!-- Testers require only the tiny report file, not the full build -->
 <Node Name="Run Test 1" Requires="#BuildReport">
   <Copy From="$(Location2LocalNetworkDrive)/WindowsClientBuild" To="$(RootDir)/TestEnvironment"/>
 </Node>

Matthew