We’ve been having trouble with the Horde issue system creating strange groupings of problems, and tracked it down to almost all of our logs falling through to the HashedIssueHandler. This is caused by most logs being emitted by the engine not meeting the format expected by other IssueHandler implementations.
As an example, UE_ASSET_LOG provides asset data but the property is `Asset` instead of `asset` which is expected by the ContentIssueHandler. Even fixing that one case, a huge number of cook warnings and errors have context on the relevant asset but use a regular log macro and that information is lost.
Is this a problem you see internally at Epic as well? Are there any existing plans to adjust how this works, or if not, do you have advice on how we should best tackle it?
Hello! Thank you for all the info; it’s really valuable to get insight into your experiences and operations.
Fixing these at the site would be ideal, but due to the lift required to orchestrate this across the engine and the reluctance to cause significant code churn (which might itself introduce bugs or other issues), we’ve been doing this incrementally or using handlers.
I had a sync with our Build Health team; they had some info on this and some questions that might be useful to their work in this area.
On span creation, we do see issues that end up spanning multiple releases, and we also have lingering warnings that haven’t been addressed.
We’d be interested in knowing whether you found any generic warnings that result in this overlap, so we can look into addressing it.
Some other questions from the team:
1. How do you track your issues in general?
2. When an issue comes in, is it more valuable to triage the issue itself or the Horde job step that failed?
3. How do you deal with intermittent and duplicate issues? For example, Horde can close an issue when a test passes, but if it fails again soon after, the issue was not really fixed.
4. What has your experience been with automation and game testing in Unreal?
For logs that are already structured by the time they reach UAT (which from the editor, for example, appears to be all of them) ILogEventMatcher won’t touch them, as it only processes plaintext logs. We’ve added a layer in there to potentially adjust JsonLogEvents as well so that we can fix this in game stream code instead of always in Horde.
I’m doing some analysis now on other sources of fall through to HashedIssueHandler, but a large number of them seem to be just that the log is arbitrary plaintext and provides no additional context. As an example, I found a bunch of texture warnings emitted from TextureDerivedData.cpp. Any of the logs had awareness of the nature of the problem and the asset in question, but they all just format the information differently. It seems impractical to try to create coupled parsers to each of these on the consuming end. As tedious as it might be, it seems like the true solve is for these logs to just provide structured data at the callsite
A related but somewhat different problem with issues we noticed: the mechanics of span creation and assignment seems to lead to us sometimes creating giant blob issues that consume everything…
The situation we have is that sometimes for content that is farther out from release, warnings will persist for some time. So we might be on content 10 right now, but have some jobs building or validating content 11 and 12.
Content 11 has had some ebb and flow of problems, and on the whole created reasonably sized issues as a result. However, there’s some open set of issues when jobs for 12 is introduced. The first run on 12 also hits all of them, and because there’s no existing span for that step, all of them get globbed to a single new span. Because of fingerprint key overlap, it’ll get attached to whichever existing issue from 11 it found first. Now what was a reasonable issue previously has suddenly inherited a bunch of unrelated problems, that were also likely already tracked in another issue.
In theory this can be tuned with grouping rules, either by tweaking fingerprint types or using annotations to put up walls between categories of templates. However, those are useful features to be available to help turn new legitimate issues, and it’s a bit awkward that we need to be extra careful to not also accidentally create a mega blob.
Of course in some ways the solution is to just not have warnings that linger. I’m curious if your games follow that policy rigorously and therefore don’t see this happen. A potential systemic solution is to change the semantics of new span creation. Rather than requiring span creation before issue matching, each event can be turned into a temporary span and search for an issue. Only events that fail to find an existing home then form one or more new spans. Let me know what you think!