General guideline on using AWS FSx Netapp Ontap on build machine

Development Pipeline & Plugins
, ,
1

Hi All

Is there a recommended way to configure FSx Netapp Ontap for UE builds? Shall one mount it via NFS or iSCSI lun ?

We are using flexclone volume (also flex volume) via NFS and getting strange, slow behavior plus RAM issues in build log though we get 50-70% RAM usage:

09:08:22 UbaSessionServer - Disable remote execution (remote sessions will finish current processes) 09:08:22 ------ Building 4042 action(s) started ------ 09:08:28 UbaSessionServer - Allocation failed in MapViewOfFile (The paging file is too small for this operation to complete.).. process will sleep and try again 09:08:28 [1/4042] Compile [x64] Module.EditorSubsystem.cpp 09:08:28 [2/4042] Link [x64] UnrealEditor-EditorSubsystem.lib 09:08:28 UbaSessionServer - Allocation failed in VirtualAlloc (The paging file is too small for this operation to complete.).. process will sleep and try again

[108/4643] Copy [x86-64] ChaosClothingSimulationMesh.ispc.generated.h UbaSessionServer - NtCreateFile for TraverseDir on \172.31.49.86\vol1_clone1 failed with error code c0000033 UbaSessionServer - NtCreateFile for TraverseDir on \172.31.49.86 failed with error code c0000033 UbaSessionServer - NtCreateFile for TraverseDir on \ failed with error code c0000033

We have also noticed the following:

In Event Viewer, event source: Mup

{Delayed Write Failed} Windows was unable to save all the data for the file ;E:000000000005df76\172.31.49.86\vol1_clone1\j_work_zspace.vs\zSpace\v17\Solution.VC.db-shm; the data has been lost. This error may be caused by network connectivity issues. Please try to save this file elsewhere.

As iSCSI is block-based (not file-based as NFS) we will get an empty disk so this does not make a sense, right ?

NOTE: Visual studio is located on local C: drive and custom UE5 engine, the project and DDC are located on flexclone (also tried with flex vol).

Best Regards

2

Hey Narek,

Thanks for the question here. I can’t speak to the AWS FSx Netapp Ontap configuration, as this not what we use internally. We have investigated it, and had encountered some performance challenges at scale with slow/cold starts.

I will say that our build pipeline is quite memory intensive due to it’s breadth, so I’d suggest ensuring that you have:

  1. Sufficient RAM (for code we use 8xlarge)
  2. Sufficient disk space for virtual memory + paging. As you’ve noted this may not be strictly applicable given the non-file based NFS - but I don’t have much experience or context for this.

Some things I’d try is to turn off UBA entirely in your build execution via -NoUba + MaxParallelActions-16 (I’d be interested to get your more complete build log to see exactly how many executors it’s trying locally). This should further relieve memory pressure - the wider you go the more memory you’ll consume (especially in the large PCH context).

Julian

3

Thanks for the answer.

Using 16 threads made difference, the process was able to start and escalade , however it failed.

We tried building the Engine via the following commands as we usually do:

call “%CD%/Setup.bat” --force

call “%CD%/GenerateProjectFiles.bat”

call “C:\Program Files (x86)\Microsoft Visual Studio\2019\Community\MSBuild\Current\Bin\MSBuild.exe” “%CD%/UE5.sln” -target:“Engine\UE5” -property:Platform=Win64;Configuration=“Development Editor” -verbosity:diagnostic

call “C:\Program Files (x86)\Microsoft Visual Studio\2019\Community\MSBuild\Current\Bin\MSBuild.exe” “%CD%/UE5.sln” -target:“Programs\UnrealLightmass” -property:Platform=Win64;Configuration=“Development Editor” -verbosity:diagnostic

Got the errors(the following is log from 2nd try):

------ Building 9 action(s) started ------

[1/9] Link [x64] UnrealEditor-AddContentDialog.dll

..\Intermediate\Build\Win64\x64\UnrealEditor\Development\WidgetCarousel\UnrealEditor-WidgetCarousel.lib : fatal error LNK1136: invalid or corrupt file

[2/9] Link [x64] UnrealEditor-ControlRigDeveloper.dll

..\Intermediate\Build\Win64\x64\UnrealEditor\Development\VisualGraphUtils\UnrealEditor-VisualGraphUtils.lib : fatal error LNK1136: invalid or corrupt file

[3/9] Link [x64] UnrealEditor-CommonUI.dll

..\Intermediate\Build\Win64\x64\UnrealEditor\Development\WidgetCarousel\UnrealEditor-WidgetCarousel.lib : fatal error LNK1136: invalid or corrupt file

[4/9] Link [x64] UnrealEditor-OnlineSubsystemUtils.dll

..\Intermediate\Build\Win64\x64\UnrealEditor\Development\Voice\UnrealEditor-Voice.lib : fatal error LNK1136: invalid or corrupt file

[5/9] Link [x64] UnrealEditor-Convai.dll

..\Intermediate\Build\Win64\x64\UnrealEditor\Development\Voice\UnrealEditor-Voice.lib : fatal error LNK1136: invalid or corrupt file

[6/9] Link [x64] UnrealEditor-RigVMDeveloper.dll

..\Intermediate\Build\Win64\x64\UnrealEditor\Development\VisualGraphUtils\UnrealEditor-VisualGraphUtils.lib : fatal error LNK1136: invalid or corrupt file

Do you use something similar to FSx Netapp for the same purpose that can be shared?

Best regards

4

Hey there Narek,

A couple of further suggestions, as I do think this is mostly related to memory - especially due to the constriction further helping. Have you been able to expand memory at all? What about further constraining to 8 cores? is this still running in a UBA context as well?

I’d suggest a couple other ideas to help:

`

true

false `\>Do you use something similar to FSx Netapp for the same purpose that can be shared?
  • Unfortunately no - we don’t use something similar to my knowledge.

Kind regards,

Julian