UAT has become too slow to build - nuget issue?

Development Pipeline & Plugins
, ,
1

Hi Epic,

​

It used to be that calling RunUAT.bat didn’t have a noticeable overhead. I think that was the case up to the early UE5 releases. Over time, UAT seems to have grown, and now it has many more dependencies. Building UAT’s “script modules” can take 25 seconds, and I think I’ve even seen it take ~45 seconds in some cases. Example:

​

RunUAT.bat -ScriptsForProject=“…uproject” Turnkey …

Running AutomationTool…

Using bundled DotNet SDK version: 8.0.300 win-x64

Starting AutomationTool…

Parsing command line: …

Initializing script modules…

Building 46 projects…

Build 46 projects time: 25.79 s

​

25 seconds for a no-op build of some C# projects is much worse than a no-op build of the editor itself, which is a legendary C++ beast :). Obviously, this is a problem, since UAT adds overheard everywhere in our automation.

Looking at the build output inside Visual Studio, this seems to be related to the nuget restore. It can take ~500ms per project:

Restored …\Engine\Source\Programs\Shared\EpicGames.Oodle\EpicGames.Oodle.csproj (in 3 ms).

Restored …\Engine\Source\Programs\Shared\EpicGames.UBA\EpicGames.UBA.csproj (in 457 ms).

Restored …\Engine\Source\Programs\Shared\EpicGames.OIDC\EpicGames.OIDC.csproj (in 483 ms).

Restored …\Engine\Source\Programs\Shared\EpicGames.MsBuild\EpicGames.MsBuild.csproj (in 513 ms).

Restored …\Engine\Source\Programs\Shared\EpicGames.UHT\EpicGames.UHT.csproj (in 542 ms).

Restored …\Engine\Source\Programs\Shared\EpicGames.Serialization\EpicGames.Serialization.csproj (in 562 ms).

Restored …\Engine\Source\Programs\Shared\EpicGames.IoHash\EpicGames.IoHash.csproj (in 580 ms).

Restored …\Engine\Source\Programs\Shared\EpicGames.Core\EpicGames.Core.csproj (in 597 ms).

Restored …\Engine\Source\Programs\Shared\EpicGames.Build\EpicGames.Build.csproj (in 615 ms).

Restored …\Engine\Source\Programs\Shared\EpicGames.Horde\EpicGames.Horde.csproj (in 677 ms).

Restored …\Engine\Source\Programs\UnrealBuildTool\UnrealBuildTool.csproj (in 733 ms).

…

This means: 46 projects x 0.5 secs = 23 secs, which roughly matches the 25 secs from the earlier RunUAT log fragment from UGS.

It’s interesting that consecutive runs of RunUAT.bat don’t always exhibit this problem. Same with VS. I’m not sure what triggers it.

I assume it’s not just us experiencing this issue?

Can something be done about it? Would licensing issues prevent you from checking-in the nuget dependencies?

Could I try to take control via the RunUAT flags, perhaps? It seems that UAT.exe compiles its dependencies at runtime, but perhaps it will respect me passing -nocompileuat to RunUAT.bat? If so, maybe we all need to switch to calling BuildUAT.bat at the start of our pipelines, and then have RunUAT.bat default to -nocompileuat everywhere.

What do you think?

Thanks.

2

This is something we’re aware of and are investigating solutions for as we are also encountering this. I don’t have much more to provide at this time but yes you are not the only one this affects.

I believe building via Visual Studio is appearing to thrash the UAT scripts that are built with separately managed dependency checking when run via RunUAT.bat so it causes rebuilds when switching back and forth.

3

I’ve made a Jira for tracking this and optimistically will be targeting a resolution for 5.8, will be here when made public https://issues.unrealengine.com/issue/UE-350233

4

Ah, interesting. I thought that our CI is affected too (where we don’t build with VS), but maybe I’m wrong. I’ll pay more attention when I get a chance. Meanwhile, I’m happy to wait for a fix from you.

Thanks Joe!