Potential for crash reporting server integration between Epic and licensees?

Development Programming & Scripting
, ,
1

Hi Epic. I’ve been triaging a bunch of crash/ensure reports in recent months. I often wish I could simply forward some of them to you.

​

We are using a service called Sentry. I believe Epic are no strangers to this, as I’ve seen some evidence of Sentry integrations in the engine, randomly. Anyway, that’s a detail, for now.

​

A significant percentage of our crashes and ensures are from core engine or editor code. While it’s possible that our various engine modifications have played a role in some of them, I’d suggest there’s still a fair number of them that Epic would be interested to know about.

​

A lot of them are object lifetime issues, where a pointer validity check can be enough. My personal favorite rare crash is when someone pressed ctrl-Z inside the BP editor at the wrong moment, and has offered no context, so I can only scratch my head. One thing is always the same though: our content creators are presumably losing work every time, and they have to waste time reloading everything :(.

​

If licensees could somehow safely forward such crashes/ensures to Epic, you would build a much richer DB than our individual ones. What appeared to be a one-off crash to us, could reveal interesting patterns and trends on your side, ultimately leading to a fix, or prevention at least. You get the idea, hopefully. Everyone will benefit, in this utopia :slight_smile:

​

The practicalities of such a crash forwarding system could be a nightmare, I think. We wouldn’t want to send the dmp or the log (too easy to leak secrets), so you’d be left with just the callstack. At least the callstack is useful on its own, and doesn’t require symbols. As for how we could send the callstacks to you, you might have to build a webhook-based service, or something like that. Sentry supports outgoing webhooks at least (Webhooks), but there’s no Sentry-to-Sentry integration from what I can tell. For other providers, who knows…

​

Despite these implementation issues, I found the idea interesting, and thought to share it here and get your thoughts. No worries either way. This doesn’t have to be anything other than a conversation starter on your end.

Thanks,

Kostas V.

[Attachment Removed]

2

Steps to Reproduce
N/A

[Attachment Removed]

3

Hi Kostas,

you are correct we are aware of Sentry (and some other third party providers of crash report ingestion). IIRC Sentry has two methods of supporting UE, one where they use our CrashReportClient and their custom endpoint for UE crashes. And one where they have a plugin and send all in their own format.

If you use the former the crash information is already stored on the machines. You could run CrashReportClient as a standalone process and just point it to a directory and it can send a report to Epic. If you are worried about proprietary information you will need to do some preprocessing on the files in the directory and strip out the information you don’t need. But in theory could have a pipeline where you:

  1. Collect(?) crashes from user machines perhaps to a central location
  2. Have some script that preprocess the log file and crash context.
  3. Run CrashReportClient on each directory to send them to Epic

However, this will be of limited use for us, since we can’t do much without symbolicated callstacks, which brings us to your suggestion of a Sentry-to-Epic transport. I can bring it up with our Online Ops team (who owns the crash reporting infrastructure, basically our own Sentry) if there is an appetite to exposed such a thing.

[Attachment Removed]

4

Hi again,

after internal discussion Developer Relations and QA though it best to no go forward with rpc approach. However, your developer relations contact is happy to receive a “top 100 bugs” report or similar, which they can then evaluate and match against our own reporting.

[Attachment Removed]

5

Great, thanks!

​

I didn’t know that the CrashReportClient is configured to connect to your crash reporting backend by default! :open_mouth: I had pressed the Send button a couple of times, as I was setting it up for Sentry. Oops, I hope I was using our non-secret project :). For years, I thought that the CRC is simply functional enough for each licensee to hook their own backend to it, and otherwise it would do nothing.

​

Anyway, we quickly hooked Sentry to the CRC, and were happily using it for a while. Recently, we have switched to their own crash reporter app. That added screenshot functionality for us (and now video too, apparently, as of two days ago?), which can be useful.

​

Thanks for responding. Happy to let this thread auto-close, of course. Not sure how you’ll message me, if your Ops team will ever implement something for me to use, but maybe you have visibility of my email address, via my username.

[Attachment Removed]