About missing data for Horde Telemetry

Development Pipeline & Plugins
, ,
1

Hello,

I’m looking to setup the various telemetry views & metrics. I’ve gotten the Live, Load, Cook, Cache tabs working but I’m still getting nothing for the Memory, Content, Compile, Build, Package tabs.

Regarding memory, how is that data generated? Do we need to run APTs via Gauntlet, will it generate per platform and per configuration metrics?

Regarding content, same question? Is it fed from a Zen Server with telemetry enabled?

Regarding compile, those metrics seem to only be supported by UBA? Is there a way to get that data for fastbuild?

Regarding build, I’ve adjusted the metrics to point to the correct job names and step names, but nothing is coming through, any idea why that could be?

Regarding package, the metric is looking for Core.IoStoreChunk but I have no idea where that is coming from. The IoStoreChunk seems to point to PakFileUtilities but what process reports that to Horde?

This question was created in reference to: [Horde [Content removed]

Thank you.

[Attachment Removed]

2

Hello! One way we ingest metrics at Epic is through the CSVToTelemetry tool in our build pipeline.

https://github.com/EpicGames/UnrealEngine/tree/ue5\-main/Engine/Source/Programs/CSVTools/CSVToTelemetry

For example, nightly builds in Horde may use UnrealPak to dump the data from the paks to a CSV, which is then uploaded with CSVToTelemetry.

This is how some of the data in the Content and Package views are populated, and why this is not generated out of the box.

In 5.7, StudioTelemetry calls have also been added in IoStoreUtilities.cpp, though the commit message notes “UnrealPak will send telemetry ONLY when the -Telemetry is supplied on the commandline”.

https://github.com/EpicGames/UnrealEngine/commit/35db879dba686da39ee41eb445b005e9bb032d09

https://github.com/EpicGames/UnrealEngine/commit/09408a4266885dfa0c27a9595239338fd6e85a47

For the Memory view, FEditorTelemetry::RecordEvent_MemoryLLM & FRuntimeTelemetry::RecordEvent_MemoryLLM both call StudioTelemetry with the “Core.Memory.LLM” event name.

However, both require ENABLE_LOW_LEVEL_MEM_TRACKER to be set. Please see this for more information on the low-level memory tracker:

Per the metric definition in `Horde\HordeServer\Defaults\default-metrics.telemetry.json`, the hardware platform will form part of the grouping logic.

https://dev.epicgames.com/documentation/unreal\-engine/using\-the\-low\-level\-memory\-tracker\-in\-unreal\-engine

The Compile metrics defined in the defaults file (“ubt-uba-*”) are indeed UBT/UBA specific, but you can create additional metrics and add them to this view.

We don’t appear to have any examples for FastBuild, but you can look to the StudioTelemetry plugin, CSVToTelemetry, or UnrealBuildTool’s Telemetry.cs integration for guidance on extending metrics support.

https://github.com/EpicGames/UnrealEngine/blob/ue5\-main/Engine/Source/Programs/UnrealBuildTool/System/Telemetry.cs

For the Build views not showing any data, I recommend connecting to the MongoDB to check that the various elements of the filter align with the job data that you’re receiving.

Here, you can also check the Metrics and Telemetry collections to verify that you’re receiving data. If you continue to experience issues, please share your globals.json, server.json and BuildConfiguration.xml.

[Attachment Removed]

3

Thank you for the detailed answer!

[Attachment Removed]