Hi! I’m working on getting analytics visible in our Horde setup. Right now our Horde reads in its globals.json file from Perforce, so the relevant section I’ve added based off the tutorial is:
"plugins" : {
"analytics": {
"stores": [
{
"id": "engine",
"include": [
{
"path": "default-metrics.telemetry.json"
},
{
"path": "default-analytics.dashboard.json"
}
]
}
]
}
},
Horde seems to have accepted these paths, as the 2 files are in the same folder as the globals.json file in Perforce (and an incorrect name gave me a config error in Horde). In the server logs I see telemetry events, and I see the events in our MongoDB but no views are found at /analytics. Is there somewhere on the machine I can see where any potential err
Hey there!
Apologies for the delay in response here.
So any issues should be logged out to the server (can find locations & more details here). So when you refer to telemetry, are you specifically speaking to telemetry (or do we mean metrics). Based on your configuration above, it should be working - especially if you’re seeing metrics in your mongodb.
If alternatively you have a local setup (not grabbing the configuration from perforce), do you see views? There was an esoteric issue that I vaguely recall in the past where configs pulled from p4, on linux, had some issues with path resolution (I’ll try and dig this up). I just want to rule out the possibility this is the issue.
Could you upload a screenshot of your analytics view (Tools -> Analytics)? And if possible, open up the developer console (On Chrome, ctr+shift + j) - I’d like to see if there is anything going wrong on the get request. This could be ACL related.
Kind regards,
Julian
Hey there Sean,
Yeah this is odd for sure. So all this is doing is poking the “/api/v1/telemetry/views” api (related controller code). If you go to your HORDE_URL/swagger/index.html - try issuing a request to the /api/v1/telemetry/views. Once you get a response, this could give us some hints.
Let me know what the response is when you hit that endpoint from the swagger interface - we are doing an auth check there… So I wonder if there is maybe an acl configuration missingL
"acl": {
"entries": [
{
"claim": {
"type": "http://epicgames.com/ue/horde/user",
"value": "julian.eps" //as an example
},
"actions": [
"QueryMetrics"
]
}
]
Kind regards,
Julian
Hey Sean,
No worries at all - getting your feedback here on this feature is helpful for us in trying to improve this and move it out of experimental status.
Ahhh dang - yes! Stale defaults threw me for a loop at one point as well. I am going to create a ticket to be a bit more loud when we encounter a scenario where views or metrics fail to resolve - its a frustrating error to track down.
We don’t have any documentation for this outside of https://github.com/EpicGames/UnrealEngine/blob/5\.7/Engine/Source/Programs/Horde/Docs/Tutorials/Analytics.md, https://github.com/EpicGames/UnrealEngine/blob/5\.7/Engine/Source/Programs/Horde/Docs/Config/Schema/Telemetry.md\#telemetryviewconfig as examples.
So one thing you need to really make sure of is that when you update your BaseEngine.ini:
[StudioTelemetry.Provider.Horde.Dev]
Name=HordeAnalyticsDev
ProviderModule=AnalyticsET
UsageType=EditorAndClient
APIKeyET=HordeAnalytics.Dev
APIServerET=INSERT_YOUR_HORDE_URL_HERE
APIEndpointET="api/v1/telemetry/engine"
[StudioTelemetry.Provider.Horde]
Name=HordeAnalyticsProd
ProviderModule=AnalyticsHorde
UsageType=EditorAndClient
APIKeyET=HordeAnalytics.Prod
APIServerET=INSERT_YOUR_HORDE_URL_HERE
APIEndpointET="api/v1/telemetry/engine"
From this configuration above, your horde server would be publishing various telemetry.
Now regarding what we have, admittedly it can be a bit spotty.
I’ll give a very concrete example here that may help demystify things a tad, and give you a kick-off point.
(This all is best illustrated by trying yourself with your BaseEngine.ini pointing to your horde instance).
- Have a look at Core.UBT.Completed - https://github.com/EpicGames/UnrealEngine/blob/5\.6/Engine/Source/Programs/UnrealBuildTool/System/Telemetry.cs\#L85 - this event is emitted whenever you build via UBT
- Now follow up on the metrics collection side here: https://github.com/EpicGames/UnrealEngine/blob/5\.6/Engine/Source/Programs/Horde/HordeServer/Defaults/default\-metrics.telemetry.json\#L825 - this defines how we turn a telemetry into a metric
- So for this example, we simply average over a 30 minute window
- Now if we jump over to the view side https://github.com/EpicGames/UnrealEngine/blob/5\.6/Engine/Source/Programs/Horde/HordeServer/Defaults/default\-views.telemetry.json\#L583 - - doh! no such consumption occurs for this metric as it’s not referenced by it’s id () anywhere. Let’s add one!
- In the compile region, add this view:
{
"name": "UBT Duration",
"display": "Value",
"metrics": [
{
"id": "ubt-completed-duration",
"alias": "FullDuration"
}
]
},
When we now go back to our analytics view, we should see it visualized for the grouping semantics it was emitted from.
I hope this clears up how it works end to end.
Kind regards,
Julian
My pleasure Sean! Should you hit any issues just feel free to poke this thread 
Happy to help.
Kind regards,
Julian
Hey there Sean,
Yes you need to enable this explicitly from the server.json (it just must be specified within the analytics plugin json prop):
"plugins":
{
"Analytics":{
"Sinks":
{
"Mongo":
{
"Enabled": true,
"RetainDays": 1
},
}
}
… which is similar to the above.
Julian
Hey there Sean,
Apologies for the delay on my end, as it was a long weekend for me over here.
State.JobStepRef is a legitimate event that we emit. How this works is that we multiplex our telemetry events out to different endpoints, with the analytics sink being one of them. This is likely a result of the filter:
"filter": "$\[?([Content removed]
… specifically, the jobName is likely to be throwing a wrench here. I’d give that a look first.
Julian
HI Julian,
Thanks for the response. I’m referring to telemetry and views for sure, as in I want to be able to see overall cook times, job times, etc in views as the goal here.
I don’t have a local setup but this is what the current situation is on both our test and production horde servers. I’m going to try and see if I can configure it to pull the json files locally somehow. The console does say no telemetry views configured, so I’m wondering if there’s more that needs to happen in these default files, or if there’s an issue with using default files downloaded P4 as well.
Let me know if I can pass along any more detail and thanks in advance!
[Image Removed]
Hi Julian,
I got identical results using a local globals.json file versus a perforce one. I also tried using literal paths in the analytic plugin section and got the same result:
"plugins" : {
"analytics": {
"stores": [
{
"id": "engine",
"include": [
{
"path": "D:\Horde\Config\default-metrics.telemetry.json"
},
{
"path": "D:\Horde\Config\default-analytics.dashboard.json"
}
]
}
]
}
},
I know you posted log locations, but is there any fuller logging or thoughts on how to figure out where in the telemetry view process this is failing?
Hi Julian,
Made some progress here.
I added the QueryMetrics to the ACL as suggested because the Swagger was giving a 401 error the first times I tried. Once I did that though, I got a 200 with an empty body, suggesting that either the views aren’t getting loaded or aren’t present somehow?
I also found in our logs this line:
[04:22:46 dbg] Initial snapshot for update: { Revision: "0b9bd04932c53aa0d354aa253ed5d72305481e7c", ServerVersion: "5.5.4-250512", Dependencies: [KeyValuePair2 { Key: perforce://default//engine/horde-configuration/.Horde/.Config/BleedingEdge/globals.json, Value: “174766” }, KeyValuePair2 { Key: perforce://default//engine/horde-configuration/.Horde/.Config/BleedingEdge/catalyst/project.json, Value: "171728" }, KeyValuePair2 { Key: perforce://default//engine/horde-configuration/.Horde/.Config/BleedingEdge/catalyst/project.png, Value: “171728” }, KeyValuePair2 { Key: perforce://default//engine/horde-configuration/.Horde/.Config/BleedingEdge/catalyst/main/stream.json, Value: "175872" }, KeyValuePair2 { Key: perforce://default//engine/horde-configuration/.Horde/.Config/BleedingEdge/engine/project.json, Value: “171728” }, KeyValuePair2 { Key: perforce://default//engine/horde-configuration/.Horde/.Config/BleedingEdge/engine/project.png, Value: "171728" }, KeyValuePair2 { Key: perforce://default//engine/horde-configuration/.Horde/.Config/BleedingEdge/engine/horde-configuration/stream.json, Value: “171728” }, KeyValuePair2 { Key: perforce://default//engine/horde-configuration/.Horde/.Config/BleedingEdge/default-metrics.telemetry.json, Value: "171728" }, KeyValuePair2 { Key: perforce://default//engine/horde-configuration/.Horde/.Config/BleedingEdge/default-analytics.dashboard.json, Value: “171728” }] }`(BleedingEdge is our test config)
That to me confirms that we’re getting the json files succesfully.
One point I want to check (and I’m going to try and find them now) is that is there a chance that the default files are out of date? I don’t know how old these ones are, and its possible they’ve been passed around in this folder since we updated to any one of the 5.x releases.
Let me know your thoughts and thanks again for working through this with me!
[Image Removed]
And that was it. Installed a new local version of horde and copied the bundled defaults over to the test server and it loaded just fine. Looking at the two files, there was definitely a lot of drift from whenever our old default file was grabbed and the latest version bundled into the Horde install.
Thanks again for your help! I think I’m set for now
[Image Removed]
Hey Julian,
Sorry to keep this thread going, but now that I have metrics and some data populating I’m a little confused. Based off of the defaults I thought there was already some gathering of metrics from Horde jobs for Compiles/Cooks/etc to build off of, but it looks like its only some Editor events. Is there documentation on how to configure horde jobs to drop in data for the telemetry plugin?
The metric -> views flow makes a ton of sense as well as there being a second ProviderModule needed for Horde events specifically. I’ll work on integrating these changes but my next in office day is 10/6. If I find a new set of areas to go over I’ll make a new thread then! Thanks again for the help this week
Hi Julian!
So I did the changes to DefaultEngine.ini and I was able to almost immediately see metrics populate from some of the pre-defined compile metrics. However I was looking through our MongoDB and I can’t seem to manually find those entries in the Telemetry table (though I can find all the earlier telemetry from the editor metrics). Is there another table created that has all the horde specific telemetry present?
I think this one might be on us. I just dove into the telemetry table we have and its last entry was from about 8 months ago (incidentally or not, lines up with an engine update for our project).
Currently that field looks like:
"Telemetry": [
{
"Type": "Mongo",
"RetainDays": 1
}
],
Which does not match the documentation, so getting that updated to see if that solves my issue.
Based off the documentation it looks like this needs to be added to the server.json for a telemetry sink setup:
"plugins": {
"analytics" : {
"sinks": {
"mongo": {
"retainDays": 1,
"enabled": true
}
},
"enabled" : true
}
},
But I’m not seeing the results appear on my test instance and I’d like to confirm this is indeed the issue before making changes on my prod instance. The pre-packaged server.json still uses the old setup, is this some drift that happened during updates between code and documentation?
Good news, I got the sink working it just took a little while for the logs to start appearing. Looks like the default metrics have a lot of filters that block them from showing up in the tables, so that’s my next step is re-writing/re-making metrics for our jobs.
However I did see a 'Payload.EventName == ‘State.JobStepRef’ in the default-metrics that surrounds average run-time for horde jobs and steps, which would definitely be information we want to start tracking, but I don’t see those events in the MongoDB Analytics table (I however do see a JobStepRef table there). I’m going to try and fix up the filters anyways, but I wanted to see if it was a possible bug that those events aren’t being sent to the Analytics table or if they’re not implemented yet in general?
1 Like