Members of our team are hitting a very sporadic crash. It’s been hard to find a consistent repro, so I’ve been unable to debug it. But it’d be great to know if this is a crash you’re seeing reports of.
When this does occur, it seems to occur only when deserializing Niagara Systems, specifically the contained NiagaraEmitterCompiledDataStruct. Deep in the process, it ultimately fails the first checkf in FStructuredArchive::SetScope. Specifically, CurrentScope[Slot.Depth].Id does not equal Slot.ElementId, and the value in Slot.ElementId does not appear to match any entry in CurrentScope[Slot.Depth].
[Image Removed]
Any info on whether other customers are seeing this, or if it’s a known issue, is very helpful. Thanks in advance!
From the Niagara side of things I’ve not heard about crashes like this.
I just had a poke around our crash reporter / jiras and we certainly have crashes in this area, very low repro, but it doesn’t always appear to be from Niagara at first glance.
Do you know if it’s always the same asset? Or is there something in common with the assets, i.e. a particular pattern of what data interfaces are used perhaps?
It’s a variety unfortunately, I see a some inside FAsyncLoadingThread and FAsyncLoadingThread2.
Without digging in it’s hard to understand if this is from another root issue, and given I have at best a mini dmp I doubt I can gather a lot of useful context.
This crash is unfortunately prevalent enough to impact our team daily, but not prevalent enough to have a consistent repro for. Has the team internally had any luck tracking this one down?
It’s not one we’ve been encountering in any meaningful way from Niagara side (or at least nobody has raised it with us), I can pass this ticket onto the core team they will hopefully have better insight than me.
We’ve been seeing this across our team. For some folks its fairly frequent, other folks never run into it, and their workstations are basically the same hardware. We’ve tried running with “-stompmalloc” with the workflows that lead to the crash, but it hasn’t caught any problems. It does keep the crash from reproducing, which might indicate a race condition.
apologies for the silence from the Core team so far!
I had assigned this question to a team member who just went out on sabbatical and I didn’t realize it at the time, so it was unfortunately stuck in limbo for a few weeks.
I’m trying to get you some eyes on this next week.
Sorry for the long delay on this. To be able to dig further into this issue though, it would be good to get associated logs associated with this callstack. This isn’t an issue that is on our radar on Core side either or that I’ve encountered before. This would mean that the record is getting serialized on a different archive than it was created from. As far as this being a race condition, I do not believe it to be the case. Even if `FAsyncLoadingThread2` is involved, everything still runs on the game thread at the moment and there still isn’t anything being async in this case at this time. What is more probable doing is some kind recursion of some kind that isn’t being properly handled. What I can see from the callstack is that the niagara system is being force preloaded as a dependency of some blueprint being loaded and compiled. if you have other similar issue where the record scope is mismatching having those as well could help out narrow down the issue. This might be more of a blueprint compilation problem where recursion in its dependencies is causing problem than a Niagara issue.
If you have other callstacks where this check was triggered as well as logs, this would help narrow down the issue because at the moment, there is little to go on to be able you identify where this issues stems from.
Hard to get good data on it. We have a few instances of the crash, but I’ve only managed to trap it in a debugger once.
I’m looking through the most recent 10 instances we have of a crash in FStructuredArchiveSlot, and only half of them involve Niagara, so that is likely a red herring. The bug is not consistent, in that someone can hit the crash, but successfully load in their next session. One’s especially odd is one team member was hitting it fairly frequently while performing a super common action (starting PIE). A couple others have hit it, but less frequently. If it’s a data race/threading issue, then that wouldn’t be surprising.
Do all of the instances you see involve FAsyncLoadingThread2? I believe FAsyncLoadingThread2 was made default as of 5.5.1, so perhaps it’s a race or bug in async loading?