dotnet.exe process stays running after cook is finished on build machine

Development Pipeline & Plugins
, ,
1

Just like you’d expect, we have build machines that sync latest from Perforce and use RunUAT.bat to BuildCookRun the game.

You’d also expect that after that build is finished (RunUAT.bat has returned), there’d be no further processes open from that build’s workspace directory. Unfortunately for us, there’s a dozen or so dotnet.exe processes hanging out after the build is done, with MSBuild somewhere in the commandline. You can imagine the problems this can cause - clean checkouts fail, for example, because it can’t delete a dotnet.exe that’s running. I’ve put a partial list (character limit) of files that can’t be deleted at the end of this message.

Has anyone else seen or fought this behavior, or something like it? It “probably wouldn’t be that hard” to add a build step to kill these things, but if there’s a better answer, I’d love to hear it.

Thanks!

-S

All of these are in the E:\build_path_here\Engine\Binaries\ThirdParty\DotNet\8.0.300\win-x64\ subdirectory, cannot be deleted because dotnet.exe is still running:

dotnet.exe

sdk\8.0.300\MSBuild.dll

sdk\8.0.300\Microsoft.Build.Framework.dll

sdk\8.0.300\Microsoft.Build.Tasks.Core.dll

sdk\8.0.300\Microsoft.Build.Utilities.Core.dll

sdk\8.0.300\Microsoft.Build.dll

sdk\8.0.300\Microsoft.DotNet.Configurer.dll

sdk\8.0.300\Microsoft.DotNet.InternalAbstractions.dll

sdk\8.0.300\Microsoft.NET.StringTools.dll

sdk\8.0.300\Newtonsoft.Json.dll

sdk\8.0.300\NuGet.Build.Tasks.dll

sdk\8.0.300\NuGet.Commands.dll

sdk\8.0.300\NuGet.Common.dll

sdk\8.0.300\NuGet.Configuration.dll

sdk\8.0.300\NuGet.Frameworks.dll

sdk\8.0.300\NuGet.Packaging.dll

sdk\8.0.300\NuGet.ProjectModel.dll

sdk\8.0.300\NuGet.Versioning.dll

sdk\8.0.300\System.CommandLine.dll

sdk\8.0.300\dotnet.dll

shared\Microsoft.NETCore.App\8.0.5\Microsoft.Win32.Primitives.dll

shared\Microsoft.NETCore.App\8.0.5\Microsoft.Win32.Registry.dll

shared\Microsoft.NETCore.App\8.0.5\System.Collections.Concurrent.dll

shared\Microsoft.NETCore.App\8.0.5\System.Collections.Immutable.dll

…a handful of others in that subdirectory

2

Hello!

The processes you are seeing are likely related to the MS build server. It should normally shutdown after 15 minutes of inactivity.

The following explains how to configure it and force shut it down: https://learn.microsoft.com/en\-us/visualstudio/msbuild/msbuild\-server?view\=vs\-2022\#shut\-down\-or\-disable\-msbuild\-server

Can provide more details on the workflow that requires deleting the workspace after the build?

Regards,

Martin

3

Aha, that’s very likely it! Thanks!

> details on the workflow that requires deleting

We don’t do this every time.

Occasionally, however, a build machine will need to delete a workspace for drive space preservation reasons, or we want a clean build or something, and so these extra-long-lived processes become a problem, or at least an inconvenience.

Also doesn’t it just “feel wrong” to have vestiges left over running after a build is finished?

Thanks again!

-S

4

Thanks for the details. I was wondering if this was related to some of our processes that we could improve.

The 15 minutes timeout for the MS Build service is actually a way to improve the performance when compiling. The processes are being kept alive so they can be reused instead of paying the startup cost for every compilation unit. It is fair to shut the server down if you don’t mind to pay the cost of restarting the processes on the next job.