Since Friday we are noticing that some agents are taking really long times in compiling (3 hours in one case for which I have included log from horde of the compute task) .
In the log we see:
UbaAgent> : UbaSessionClient - Got remote execution disabled response from host (will finish 16 active processes)
What can we do to understand the long compile times.
It there a setting to have the agent to abort compilation and have the requesting machine to complete locally when an agent is taking a long time?
can you share the .uba trace file? That usually gives good hints of what is the bottleneck.
It exists on the machine that initiated the build.. usually in Engine/Programs/UnrealBuildTool/ *.uba
if your initiator is a horde machine then it ends up somewhere else, don’t remember where.
In case you didn’t know you can open the .uba file with the uba visualizer (found in Engine/Binaries/Win64/UnrealBuildAccelerator/x64/UbaVisualizer.exe)
The file will contain file names (not paths).. in case there is anything secret there you don’t want me to see
Good question, I am not versed in these support systems. I guess you can always email me directly, I bet you can figure out my email
What you can test is setting uba session AllowKeepFilesInMemory to false. It defaults to true on windows and allows output files that has been built/received from helpers to be kept in memory until linking. These files are unmapped but still holds a file mapping handle to them so they do take up space in the page file.
you can test setting it in Engine/Binaries/Win64/UnrealBuildAccelerator/x64/UbaHost.toml
Unfortunately I have not tested these options very much likely so I want to reserve myself for potential issues. Also, there is no good way of setting this on the agent right now so I believe you have to hack this in from code just to verify if it helps your scenario.
Other than that, try to get that .uba file to me and I can help you dissect it
we are further investigating and tuning for our production environment.
After reducing the amount of agents (virtual machines) and giving the remaining ones more memory (56GB) we no longer have agents that are stuck for over an hour.
We do however occasionaly have over 10 minutes and programmers need to interrupt the compilation and restart.
Our first project, where UBA is active, seems to need agents with at least 56GB and that is the most we can give for the moment.
I have seen doing test that it is possible for a requesting machine to completely use physical memory and the PC becomes unresponsive, this also seems to keep the agents from finishing. I had to kill Visual Studio after I was using all the agents and the task time on all of them (25 agents) was over 25 minutes.
Apart from seeing how to reduce memory needs on the project (multiple actions with Module.projectname.n.cpp is where we are having issues), what methods exists to avoid local machines or agents from using too much memory?
In tests, reducing the cores with CpuCount on the agent.json did not seem to affect the memory.
Locally on my machine with 32GB even setting MaxParallelActions to 1 I am still ending with over 99% of physical memory in use.
