Work From Home: Cloud DDC (experimental)

[Amanda.Schade]

One of the major challenges of working over VPN is having to rebuild derived data after syncing new content (compiling shaders, compressing textures, and so on). When working on-site, we encourage the use of a shared derived-data cache for storing this content - a network share that the editor can read from and write to in order to cache converted assets. https://docs.unrealengine.com/en-US/…che/index.html

Access to the derived-data cache is typically highly latency sensitive due to being on the critical path for many editor operations, and accessing a network share over VPN can cause large hitches and very slow startup time.

We experimented with a few ways to improve the situation when running the editor from home. First, we tried uploading individual files to a cloud storage provider (this performed worse than regenerating the data locally due to the increased latency and time spent establishing connections). Next, we tried archiving the shared DDC folder and putting that in a cloud hosted drive (this performed badly due to the sheer size of the DDC and constant churn).

The solution we ultimately found success has been the following:

1. We added a DDC backend that records any queries being made against the DDC to a plain text file.
2. We added an automated test that loads into a few commonly used maps in the editor and runs the play-in-editor command. The DDC recording backend mentioned above generates a text file to a network share.
3. We added an AutomationTool command (UploadDDCToAWS), which scans all the text files, concatenates the derived data files into 100mb bundles, and uploads them to Amazon S3. The script is run every 4 hours on our on-site build farm.
4. We added a new DDC backend which can download and uncompress these bundles at editor startup time, removing the latency required to do so on demand. Even downloading 10s of GBs over a home network connection provides reasonable performance due to caching and the re-use of existing bundles between iterations of the UploadDDCToAWS.

1 - SETTING UP THE ACCESS RECORDER

This is done by running the engine with a custom DDC backend configured through the game’s DefaultEngine.ini file which looks something like this:

[EnumerateForS3DDC]
MinimumDaysToKeepFile=7
Root=(Type=KeyLength, Length=120, Inner=AsyncPut)
AsyncPut=(Type=AsyncPut, Inner=Hierarchy)
Hierarchy=(Type=Hierarchical, Inner=Shared)
Shared=(Type=FileSystem, ReadOnly=false, Clean=false, Flush=false, DeleteUnused=true, UnusedFileAge=5, FoldersToClean=10, MaxFileChecksPerSec=1, Path=\\path	o\your\regular\shared\ddc, EnvPathOverride=UE-SharedDataCachePath, WriteAccessLog="%GAMEDIR%Saved/Logs/DDCAccessLog.txt")

Note: The Shared node in the graph contains a Path parameter pointing to your normally configured network DDC, and a WriteAccessLog parameter specifying where to output the access log to.

Support for the WriteAccessLog attribute was added in CL 12166433.

To enable this DDC backend when running the editor, pass the -DDC=EnumerateForS3DDC parameter on the command line.

2 - AUTOMATED TEST

Implementation of this process is game-specific, and we do not currently have an example available. You could have someone periodically generate this data manually if necessary, as long as they run the editor with the recording backend enabled.

3 - UPLOADING DATA TO S3

The UploadDDCToAWS command is implemented in /UE4/Main/Engine/Source/Programs/AutomationTool/Scripts/UploadDDCToAWS.cs (see CL 12626541), and typically run using the RunUAT batch file. The command takes the following arguments:

• -Bucket=… Specifies the name of the S3 bucket to upload to
• -CredentialsFile=… Specifies the path to a configuration file containing S3 credentials. See here for the format of this file.
• -CredentialsKey=… Specifies the section name within the credentials file to take credentials from.
• -CacheDir=… Path to the shared network DDC. Paths from the recorded DDC access logs will be resolved against this base directory.
• -FilterDir=… Path to the directory containing recorded access logs.
• -Days=… Number of days to retain files in FilterDir. Any files older than this will be removed.
• -Manifest=… Specifies the path within the current workspace to store the URLs to the root manifest. This file will be checked into P4. See notes below about how bundle URLs, manifest URLs, and the root manifest interact.
• -KeyPrefix=… Object prefix to use for everything uploaded to a bucket.
• -Reset If set, existing bundles will not be reused.

4 - DOWNLOADING DATA AT RUNTIME

Enabling the download of the data from S3 requires a new DDC graph to be configured, or modification of the default one.

To modify the default configuration to use the S3 backend rather than a network share, add a section as follows to your game’s DefaultEngine.ini file:

Shared=(Type=S3, Manifest="%GAMEDIR%Build/S3DDC.json", BaseUrl="https://foo.s3.us-east-1.amazonaws.com/", Region="us-east-1", AccessKey="abc123", SecretKey="def465")

Here, the Manifest argument specifies the path to the root manifest submitted to source control by the UploadDDCToAWS command, BaseUrl/Region specify the S3 bucket to download from, and AccessKey/SecretKey specify the credentials to download with.

The S3 derived data backend is implemented in //UE4/Main/Engine/Source/Developer/DerivedDataCache/Private/S3DerivedDataBackend.cpp and S3DerivedDataBackend.h.

The following changelists are required:

12149604 12149624 12155157 12156082 12158937 12195879 12459547 12468805

An option to disable the S3 DDC can be shown in the editor preferences panel by adding the following setting to DefaultEditor.ini:

[EditorSettings]
bShowEnableS3DDC=true

IMPLEMENTATION DETAILS

• Text files generated in (1) are removed from the network share by the UploadDDCToAWS command (3) after 7 days.
• The S3 backend (4) is configured with a secret key and access key, but these credentials have to be submitted to source control. To improve security, bundles are given a random, unguessable URL, and are indexed by a manifest - also uploaded with an unguessable URL. Paths to the last few days of manifests are stored in a configuration file in Perforce, and are deleted after that. The bucket is configured to deny LIST requests, so even having the access key, secret key and manifest URL will only provide access to the latest bundle data for a few days.
• Bundles are retained in the active manifest as long as at least 40% of their data is still referenced. Once a bundle is discarded, the data within it can be added to new bundles.

ALL CHANGELISTS REQUIRED TO IMPLEMENT CLOUD DDC

• https://github.com/EpicGames/UnrealE…4ce4dffb9135bd
• https://github.com/EpicGames/UnrealE…e2df15b9fc9dd4
• https://github.com/EpicGames/UnrealE…b46c7140c2d06e
• https://github.com/EpicGames/UnrealE…84689947691d30
• https://github.com/EpicGames/UnrealE…bbde7ea84e11aa
• https://github.com/EpicGames/UnrealE…a3f58a85bc24ad
• https://github.com/EpicGames/UnrealE…9a7d55cf49457f
• https://github.com/EpicGames/UnrealE…ea9134f54ea893
• https://github.com/EpicGames/UnrealE…5c305c8f8985d2
• https://github.com/EpicGames/UnrealE…b61ea6fdc47623

---

See more work from home tips!

[Xydrel]

Hi there,

I am attempting to follow the instructions here however we are encountering issues with the RunUAT.bat UploadDDCToAWS call and the format of the credentials file. The doc states to look “here” for what the structure is, however “here” is not actually a link.

If we could get some more details on that, it would be greatly appreciated! Thanks!

[mastercoms]

Looks like: Using the AWS Credentials File and Credential Profiles - AWS SDK for PHP

[Ranisz]

I’m trying to configure an UE 5.0.2 agent that uploads DDC to S3 and it looks like AWS libraries are missing, is there any prerequisite I need to add to the agent setup?

==============================================================================
ERROR: System.IO.FileNotFoundException: Could not load file or assembly 'AWSSDK.Core, Version=3.3.0.0, Culture=neutral, PublicKeyToken=885c28607f98e604'. The system cannot find the file specified.
       File name: 'AWSSDK.Core, Version=3.3.0.0, Culture=neutral, PublicKeyToken=885c28607f98e604'
          at AutomationTool.UploadDDCToAWS.ExecuteBuild()
          at AutomationTool.BuildCommand.Execute() in D:\build\++UE5\Sync\Engine\Source\Programs\AutomationTool\AutomationUtils\BuildCommand.cs:line 344
          at AutomationTool.Automation.Execute(List`1 CommandsToExecute, Dictionary`2 Commands) in D:\build\++UE5\Sync\Engine\Source\Programs\AutomationTool\AutomationUtils\Automation.cs:line 254
          at AutomationTool.Automation.Process(ParsedCommandLine AutomationToolCommandLine, StartupTraceListener StartupListener, HashSet`1 ScriptModuleAssemblies) in D:\build\++UE5\Sync\Engine\Source\Programs\AutomationTool\AutomationUtils\Automation.cs:line 153
       
       
       (see C:\Users\domin\AppData\Roaming\Unreal Engine\AutomationTool\Logs\C+Program+Files+Epic+Games+UE_5.0\Log.txt for full exception trace)

FileNotFoundException: Could not load file or assembly 'AWSSDK.Core, Version=3.3.0.0, Culture=neutral, PublicKeyToken=885c28607f98e604'. The system cannot find the file specified.
   at AutomationTool.UploadDDCToAWS.ExecuteBuild()
   at AutomationTool.BuildCommand.Execute() in D:\build\++UE5\Sync\Engine\Source\Programs\AutomationTool\AutomationUtils\BuildCommand.cs:line 344
   at AutomationTool.Automation.Execute(List`1 CommandsToExecute, Dictionary`2 Commands) in D:\build\++UE5\Sync\Engine\Source\Programs\AutomationTool\AutomationUtils\Automation.cs:line 254
   at AutomationTool.Automation.Process(ParsedCommandLine AutomationToolCommandLine, StartupTraceListener StartupListener, HashSet`1 ScriptModuleAssemblies) in D:\build\++UE5\Sync\Engine\Source\Programs\AutomationTool\AutomationUtils\Automation.cs:line 153
==============================================================================

[aleXandr2010]

Could you please provide S3DDC.json file example, or some prototype - I couldn’t find any resource how it should b configured.

[ASC-KrisMorness]

I just went through quite the journey getting this to work, specifically with step 3. I’ll try to help out with some hints here, but still some rough patches to solve.

• I could not get RunUAT.bat UploadDDCToAWS to recognize AWSSDK.Core so was getting the same exceptions as Ranisz. My workaround was to literally run it through the debugger with AutomationTool as my active project. Also this became necessary because I was editing cs files to add additional logging while troubleshooting so I needed to build it anyways. It just worked that way. I still need to figure out why the bat file can’t find it.
• The credentials file is indeed a simple text file following the link mastercoms posted. You can call it whatever you want like: aws-credentials.txt. Looks like this:

    [default]
    aws_access_key_id = YOUR_AWS_ACCESS_KEY_ID
    aws_secret_access_key = YOUR_AWS_SECRET_ACCESS_KEY
    
    [MyProject]
    aws_access_key_id = ANOTHER_AWS_ACCESS_KEY_ID
    aws_secret_access_key = ANOTHER_AWS_SECRET_ACCESS_KEY

• You can add Log.TraceInformation(string.Format("AWS File Upload Request:\n BucketName: {0}\n Key: {1}\n FilePath: {2}", BucketName, Key, File.FullName)); right before PutObjectRequest request = new PutObjectRequest(); in UploadDDCToAWS.cs in UploadFileInner – as it gives you specific diagnostic info for each upload attempt.
• An example command: UploadDDCToAWS -Manifest="d:/path/to/manifest" -FilterDir="d:/path/to/project/Saved/DDCForAWS" -Bucket="mys3/sharedDrive/s3ddc" -CredentialsFile="path/to/aws.credentials.txt" -CredentialsKey="MyProject" -CacheDir="path/to/shared/ddc" -Submit
• If you see any upload exception containing The specified bucket is not valid. then you need to lose your prefix on your -Bucket input. No s3://, no https://, etc. That took some trial and error to solve.
• I found it useful to download the AWS CLI. On Windows, it seems like an install, but all it does is extend your command line.
• And here’s a cheatsheet to AWS CLI commands. I discovered commands are quite picky. If you do a command like aws s3 ls s3://mycompany/sharedDrive/s3ddc/bulk it’ll just report PRE bulk/ which is useless. You need to append / to get the actual contents of the directory. In my case 39 buckets of roughly 100MB files.
  
  

We have a custom engine version based off 5.0.1, but looks like the system hasn’t changed at all since this was initially introduced near the beginning of covid.

[Arukhan]

I had this same error, and once I manually added the following dependency to the automation tool csproj file (Engine\Source\Programs\AutomationTool\AutomationTool.csproj), and rebuilt it the error disappeared:

    <Reference Include="AWSSDK.S3">
      <HintPath>..\..\..\Binaries\ThirdParty\AWSSDK\AWSSDK.S3.dll</HintPath>
      <Private>true</Private>
    </Reference>

[Ranisz]

Except there are no DLLs in this folder. Is this one of the installer options to bring them in?

[heivoll]

I’m having the same issues regarding the AWSSDK.Core. You mention running AutomationTool through the debugger. Any chance you could provide an example of the full command you run to do this?

[yuanzhongliang (苑中梁)]

if your Engine is download in Epic Launch ranther than build by source code, you can just rebuild AutomationTool, but be careful add related c#project in same solution in VS2022, and use NuGet add awss3, finally rebuild the whole solution.