UbaAgent running on Linux/Wine immediately throws AWS autoscale rebalancing on compile

Hi!

I’m running a container with Wine on AWS with Horde Agents built out as extra UBA capacity. The container connects to Horde and begins a compute task, but then immediately fails on a AWS autoscale rebalancing. I’m running on instanced compute, so no auto scale should be happening? I’ve giving the agents 1 physical 2 logical cpu, and 3GB ram. I could play with values some more but I’m trying to figure out the root cause for this error being thrown immediately? Any info would be great. Thanks in advance!

uba-wine64> : UbaAgent v5.8.0-Uba_v1.0.0-46566495 (Cpu: 2, MaxCon: 8, Dir: "C:\Uba\Uba", StoreCapacity: 20Gb, Zone: us-east-1b, AWS: i-0c319a45e86060bf3 on-demand autoscale, Linux/Wine-11.0,  https://horde.internal.eve.games/lease/6a31bd3d531885f8547d3a80)
uba-wine64> :   Will poll for external events in file Z:\opt\horde\work\.horde-termination-signal
SEND 0 ExecuteOutput      [        96] =  20 20 54 49 50 3A 20 45 6E 61 62 6C 65 20 42 42..
uba-wine64> :   TIP: Enable BBR congestion algorithm and fq (Fair Queue) queuing discipline (Current: cubic)
SEND 0 ExecuteOutput      [         2] =  0D 0A
uba-wine64> : 
SEND 0 ExecuteOutput      [        98] =  55 62 61 53 74 6F 72 61 67 65 20 2D 20 44 61 74..
uba-wine64> : UbaStorage - Database saved in <1ms (0 file entries, 1 cas entries. deleted 0 dropped cas files)
SEND 0 ExecuteOutput      [        78] =  55 62 61 43 6C 69 65 6E 74 20 28 62 31 62 61 66..
uba-wine64> : UbaClient (b1bafbab-11cf-453f-99dc-22ad8ade342a) - Listening on 0.0.0.0:7001
SEND 0 ExecuteOutput      [        96] =  55 62 61 43 6C 69 65 6E 74 20 28 62 31 62 61 66..
uba-wine64> : UbaClient (b1bafbab-11cf-453f-99dc-22ad8ade342a) - Connected to server... (0x0000055A5C0101D0)
SEND 0 ExecuteOutput      [       117] =  55 62 61 53 74 6F 72 61 67 65 43 6C 69 65 6E 74..
uba-wine64> : UbaStorageClient - Database loaded from C:\Uba\Uba\cas\casdb (v32) in <1ms (contained 1 entries estimated to 1.2mb)
SEND 0 ExecuteOutput      [        38] =  43 6C 69 65 6E 74 20 73 65 73 73 69 6F 6E 20 32..
uba-wine64> : Client session 260616_211647 started
SEND 0 ExecuteOutput      [        53] =  55 62 61 53 65 73 73 69 6F 6E 43 6C 69 65 6E 74..
uba-wine64> : UbaSessionClient - Connected to server EVC-HORDE-16
SEND 0 ExecuteOutput      [       120] =  55 62 61 53 65 73 73 69 6F 6E 43 6C 69 65 6E 74..
uba-wine64> : UbaSessionClient - AWS autoscale rebalancing. Will stop scheduling processes and send failing processes back for retry
SEND 0 ExecuteOutput      [       105] =  55 62 61 43 6C 69 65 6E 74 20 28 62 31 62 61 66..
uba-wine64> : UbaClient (b1bafbab-11cf-453f-99dc-22ad8ade342a) - Disconnected from server... (0x0000055A5C0101D0) (1)
SEND 0 ExecuteOutput      [        35] =  43 6C 69 65 6E 74 20 73 65 73 73 69 6F 6E 20 32..
uba-wine64> : Client session 260616_211647 done



[Attachment Removed]

Hello! In order to trigger the “AWS autoscale rebalancing” code path, two conditions must be met in Engine\Source\Programs\UnrealBuildAccelerator\Common\Public\UbaAWS.h.

Firstly, when Cloud::QueryInformation() is called at start-up, UBA queries the auto-scaling lifecycle instance metadata service endpoint (“/latest/meta-data/autoscaling/target-lifecycle-state”).

If this returns a 200 status code, m_isAutoscaling is set to true, and the string " autoscale" is appended to the start-up banner.

https://github.com/EpicGames/UnrealEngine/blob/5\.7\.4\-release/Engine/Source/Programs/UnrealBuildAccelerator/Common/Public/UbaAWS.h\#L137

Secondly, Cloud::IsTerminating() is called on every poll, and when m_isAutoscaling is set to true, it queries the same auto-scaling lifecycle instance metadata service endpoint.

We check for a 200 status code, and return true when the response does not contain the string “InService”.

https://github.com/EpicGames/UnrealEngine/blob/5\.7\.4\-release/Engine/Source/Programs/UnrealBuildAccelerator/Common/Public/UbaAWS.h\#L221

You mentioned these are running on instanced compute; could you please try querying the same service endpoints on the instance to confirm which values UBA would be seeing?

TOKEN=`curl -X PUT "http://169.254.169.254/latest/api/token" -H "X-aws-ec2-metadata-token-ttl-seconds: 21600"` \
&& curl -H "X-aws-ec2-metadata-token: $TOKEN" -v http://169.254.169.254/latest/meta-data/autoscaling/target-lifecycle-state

The command above is taken directly from the linked AWS documentation page, which provides additional information.

[Attachment Removed]