Unable to gather analytics on UBA usage in compiles

Development Pipeline & Plugins
, ,
1

Hey!

So I recently setup some extra UBA capacity for our Horde server and we wanted to gather some overall usage analytics for scheduling capacity. The UBA agents are connected to the same Horde server as our full agents.

Now I see in the metrics.telemetry.json setup we have:

    {
      "name": "IsUBAExecutor",
      "value": "[Content removed] \"Unreal Build Accelerator\")"
    },

But when I scroll through the dump in MongoDB, I see zero entries with this Payload. It definitely seems like there’s a notion that these agents should be emitting a payload on finishing compute leases, but I’m not seeing it so far.

In addition, I see that Agents themselves gather usage telemetry, and that would be a great secondary source of information to build metrics out of. Would the agent telemetry trigg

[Attachment Removed]

2

Hello!

I think the following post could be helpful in resolving your problem: [Content removed]

Regards,

Martin

[Attachment Removed]

3

Hey there,

Just reviewing some of our internal horde setup, we do have values for:

{
	"name": "IsUBAExecutor",
	"value": "[Content removed] \"Unreal Build Accelerator\")"
},

In addition, I see that Agents themselves gather usage telemetry, and that would be a great secondary source of information to build metrics out of. Would the agent telemetry trigger a metric endpoint if it was setup?Yes indeed, but they use different underlying analytics systems.

One thing you could try is logging out in a preflight what endpoints are indeed being sent to:

// Engine\Source\Programs\UnrealBuildTool\System\Telemetry.cs; 
// public void RecordEvent(TelemetryEvent eventData)
foreach (KeyValuePair<Tuple<Uri, string>, TelemetryEndpoint> endpoint in _endpoints)
{
	endpoint.Value.RecordEvent(eventData);
	// Log out here the endpoint details
}

I’d be curious to see whether we are picking up the config properly in the UBT context.

Kind regards,

Julian

[Attachment Removed]

4

Hey there,

Apologies for the delay in response here.

I am confused with the above:

"-TelemetryProvider=“StudioTelemetry.Provider.Horde”

-> to my UBT calls and got a successful connection immediately so as an interim I’m leaving that call in.

This seems like it worked? I’m not sure how to reconcile that with the subsequent:

-> So unfortunately it does not look like -TelemetryProvider is working when calling UBT. It doesn’t seem to make a connection back to the analytics API.

To recap the main configuration points:

INI:

Anywhere in the Engine ini hierarchy (Engine/Config/DefaultEngine.ini or a restricted/NotForLicensees Engine ini — not a project’s DefaultEngine.ini, UBT reads the hierarchy with projectDir = null):

 [StudioTelemetry.Provider.Horde]
 APIServerET="https://your-horde-server.com/"
 APIKeyET=HordeAnalytics.Prod
 APIEndpointET="api/v1/telemetry/engine"

Command line:

UnrealBuildTool.exe … -TelemetryProvider=StudioTelemetry.Provider.Horde

Configuration.xml:

BuildConfiguration.xml:

<Telemetry><Providers>StudioTelemetry.Provider.Horde</Providers></Telemetry>You should be seeing UBT emit it’s loaded telemetry endpoints in the log:

logger.LogDebug(“Loaded telemetry endpoint Name={Name} Key={Key} Url={Url}”, provider, apiKeyET, baseAddress);

Do you see this at all?

Kind regards,

Julian

[Attachment Removed]

5

Hey there,

Yes that’s an alternative for now, but it makes me sad we didn’t make this extensible otherwise. I looked into this some more on my end to see how we can make this a bit more sustainable over the long haul.

I think this will be OK for now & will make it so you can keep your own project files as the authority:

@ https://github.com/EpicGames/UnrealEngine/blob/5\.7/Engine/Source/Programs/UnrealBuildTool/UnrealBuildTool.cs\#L548

// Parses the uproject, and tries tof old in endpoints
Utils.TryParseProjectFileArgument(Arguments, Logger, out FileReference? TelemetryProjectFile, false);
TelemetryService.Get().AddEndpointsFromConfig(TelemetryProjectFile?.Directory, Logger);

… used in concert with -TelemetryProvider= will allow you to select from whatever superset you create with the configs.

I’ll see if I can get this patched in 5.8.

Kind regards,

Julian

[Attachment Removed]

6

Well that revealed:

[StudioTelemetry.Provider.HordeAnalytics]
Name=HordeAnalytics
ProviderModule=AnalyticsET
UsageType=EditorAndClient
APIKeyET=HordeAnalytics.Dev
APIServerET="<correct horde url>"
APIEndpointET="api/v1/telemetry/engine"
 
[StudioTelemetry.Provider.Horde]
Name=HordeAnalyticsProd
ProviderModule=AnalyticsHorde
UsageType=EditorAndClient
APIKeyET=HordeAnalytics.Prod
APIServerET="<correct horde url>"
APIEndpointET="api/v1/telemetry/engine"

I think I need to merge these two into something like:

[StudioTelemetry.Provider.Horde]
   Name=HordeAnalytics
   ProviderModule=AnalyticsET
   UsageType=EditorAndClient
   APIKeyET=HordeAnalytics.Dev
   APIServerET="<correct url>"
   APIEndpointET="api/v1/telemetry/engine"

[Attachment Removed]

7

[mention removed]​ I ran a preflight with the above and had no difference. I get cook analytics from the editor just fine, but nothing around Compiles with UBT.

I looked back at the metrics, and confirm that they were working before we merged 5.7 into our project, and stopped reporting afterwards. Were there any changes or configuration changes needed post 5.7? I know there were a couple features added to analytics in that release.

[Attachment Removed]

8

Hey Julian,

Yeah your filter looks identical to ours and for sure it was submitting metrics up until our 5.7 merge. Setting up a preflight now with the extra logging, will post here later with results.

[Attachment Removed]

9

Hey,

So ran a couple builds locally. My logging confirmed that UBT was not detecting the endpoint in DefaultEngine.ini. i added

-TelemetryProvider="StudioTelemetry.Provider.Horde"to my UBT calls and got a successful connection immediately so as an interim I’m leaving that call in.

I’m still a little concerned that the functionality broke for our project with the 5.7 merge. Was there some sort of default that changed how the TelemetryProvider is set on UBT?

[Attachment Removed]

10

Hi,

So unfortunately it does not look like -TelemetryProvider is working when calling UBT. It doesn’t seem to make a connection back to the analytics API.

When running compiles from a build graph script and using UBT calls directly, what would be the best way to pass in the configuration?

[Attachment Removed]

11

Hi is there more guidance I can get here?

[Attachment Removed]

12

Just to step back:Anywhere in the Engine ini hierarchy (Engine/Config/DefaultEngine.ini or a restricted/NotForLicensees Engine ini — not a project’s DefaultEngine.ini, UBT reads the hierarchy with projectDir = null):

So we do have this config in our GameName/Config/DefaultEngine.ini and that’s not valid? We’ve made all our game/build configurations from that file. We don’t even have a DefaultEngine.ini in the Engine/Config folder.

[Attachment Removed]

13

And to continue on that line of thinking, would I just be able to add a Engine/Config/DefaultEngine.ini file there with just the telemetry values as a possible fix?

[Attachment Removed]