Crash during Aftermath marker resolve (use after free breadcrumb pointer)

Development Rendering
, ,
1

Hello folks,

We have integrated a few changes related to Aftermath, notably CL50650532, and it seems like we might be missing a CL to prevent Aftermath breadcrumb resolve from crashing. For reference, CL50650532 only send the Breadcrumb pointers to Aftermath and then use the Aftermath Editor to resolve them to a text at the moment of the crash.

However, depending on the workload of the frame, it seems like the lifetime of the breadcrumbs is not long enough and when resolving we are accessing deleted breadcrumbs pointer sent by Aftermath.

I’ve integrated CL48294654 but that didn’t help.

My current guess is that some breadcrumbs are deleted because they were part of a previous command list, but Aftermath still request them for resolve since they are near the crash location.

Any ideas?

[Attachment Removed]

2

Steps to Reproduce
Provoke a GPU crash, e.g. gpudebugcrash pagefault

Notice it sometimes crash trying to resolve marker

[Attachment Removed]

3

Hi Olivier,

I still need to hear back from the devs, but for what it’s worth, the D3D12 breadcrumb lifetime problem hasn’t been fully resolved yet. FRHIBreadcrumbNode pointers outliving the crash callback were known and unsolved even with the submitted CLs you found, as FMarker formats the breadcrumb name into a local buffer and lets Aftermath copy the string. CL50650532 removes that macro and makes pointers the only access mode, but it does not add any lifetime extension to go with it. Vulkan did end up implementing a better mechanism, though in CL 43106894. You might need to implement something similar, but I’ll make sure to get back to you in case we already have some changes lying around.

[Attachment Removed]

4

Ok, that sounds good to me. I got word back that we don’t have any changes to the D3D12 RHI to manage better the lifetime of the Aftermath breadcrumb handles as of now, so stick with the revert or see if you can adapt the Vulkan changes to your needs.

[Attachment Removed]

5

OK good to know. I’ve reverted to the old way of doing things locally (have Aftermath copy the string on Breadcrumb start/end). I’ll look into the Vulkan CL.

Thanks

[Attachment Removed]