The crash we are getting involves the assignment of the MarkersPassedThisTick pointer as found on the FAnimTickRecord in UAnimMontage. It looks like the address is not valid for the copy. Internally the memcopy fails because of:
AnimMontage.cpp::1278
MarkerTickContext.MarkersPassedThisTick = *Instance.Montage.MarkersPassedThisTick;
Callstack (with some info removed) looks like:
VCRUNTIME140.dll!memcpy_*****() Unknown
\[Inline Frame] \-\-\-\-\-.exe!FGenericPlatformMemory::Memcpy(void \*) Line 596 C\+\+
\[Inline Frame] \-\-\-\-\-.exe!FMemory::Memcpy(void \*) Line 162 C\+\+
\[Inline Frame] \-\-\-\-\-.exe!ConstructItems(void \*) Line 116 C\+\+
\[Inline Frame] \-\-\-\-\-.exe!TArray\<FPassedMarker,TSizedDefaultAllocator\<32\>\>::CopyToEmpty(const FPassedMarker \*) Line 3179 C\+\+
> -----.exe!TArray<FPassedMarker,TSizedDefaultAllocator<32>>::operator=(const TArray<FPassedMarker,TSizedDefaultAllocator<32>> & Other) Line 507 C++
\-\-\-\-\-.exe!UAnimMontage::TickAssetPlayer(FAnimTickRecord \& Instance, FAnimNotifyQueue \& NotifyQueue, FAnimAssetTickContext \& Context) Line 1278 C\+\+
\-\-\-\-\-.exe!UE::Anim::FAnimSync::TickAssetPlayerInstances(FAnimInstanceProxy \& InProxy, float InDeltaSeconds) Line 193 C\+\+
The code is running on a worker thread. The montage is part of a triggered ability we assume due to the description of when it happened - and the trigger that did it was fairly consistent in nature (e.g. think picking up something), but the mini crash dump we had was not clear on the asset, nor repoable in any sense.
Looking at the code, it’s not clear how this would be corrupt, but we’re currently wrapping the dereference in a nullptr check to see if that helps. Obviously looking for any guidance around the likelyhood or potential cause of how this crash may happen?
Best regards,
John O’Kane
Steps to Reproduce
This is a rare issue we’ve encountered four or five times, resulting in a crash. Repo steps are not clear, but it certainly involves playing a montage during gameplay. It seems for us it might be in some way related to an particular asset since the type of action involved seemed consistent.
Hi, sorry for the delay getting back to you on this. I’ve been discussing it with the dev team, and it’s possible that we may have run into a similar bug to this previously. I’m waiting to hear back as to whether there’s a CL that you could try. I’ll follow up as soon as I have more info.
Hi, so the change I was discussing with the dev team is different but turns out not to be related to the issue that you’re running into (it was related to how the CMC handles notifies). The CL for the change that I think you’re referencing is 38234235. That was an ASAN failure we were hitting in Fortnite server builds. The callstack for the failure was different, but that doesn’t mean it couldn’t be related, since it’s memory stomping. Do you know if the montage is blending out when the crash happens? That was the specific case that was failing without that change. I don’t see any harm in backporting that CL if you wanted to.
Apart from that, it’d be good to get any extra information that you have. You said you have a crash dump - does the entire Instance ptr (from Instance.Montage.MarkersPassedThisTick) look invalid in the dump? Are you able to get any other log data from the crash?
Due to the limited nature of the minidump I couldn’t read the Instance or Montage field in the callstack and local variables: All <Unable to read memory>. The register in the disassembly that is pointing to the address of Instance seems ok?
MarkerTickContext.MarkersPassedThisTick = *Instance.Montage.MarkersPassedThisTick;
lea rcx,[r14+78h]
r14: 0x000000EF80B7E580
But I guess it should be, it’s the FAnimTickRecord - safe until next frame? And I might guess that the pointer for MarkersPassedThisTick in the record is ok (as in sensible) - it’s just the stuff at the location of the pointer that is gone/changed. My visual studio assembly foo isn’t so hot, so if you know a quick way to cast an address to visualise the full struct given an memory location, please let me know. I’m not sure how much the USTRUCT BODY macros etc can modify the entries and offsetting 78h into the record to read the address there didn’t reveal anything (just ??? in the memory viewer). This minidump is only 800kb.
What I can say is the source pointer and count in the mem copy is not good, see image attached: Src: 0xDDDDDDDDDDDDDDDD Count: 0xFFFFFFFE6666665C. That indicates the dereferenced MarksPassedThisTick TArray is not pointing at a sensible value and the failure is attempting to read beyond 0xFFFFFFFFFFFFFFFF.
[Image Removed]
“Unhandled exception at 0x00007FF9D6FA156D (VCRUNTIME140.dll) in UEMinidump.dmp: 0xC0000005: Access violation reading location 0xFFFFFFFFFFFFFFFF.”
The destination pointer seems ok I think: 0x00000248ADC00000 - but again, not an address I can peer into for this minidump.
I’m afraid I still don’t even know the montage yet. I’m guessing it’s player character related, or something that may cause a player to stop/remove/change some montage on their character - equip/unequip/put down. Initially I thought it might be related to ActorSequenceComponents because they were also common to two of the first three scenarios we had. But they didn’t have montages related to them at all. Then possibly something to do with network relevancy or some anim pause/culling scenarios that might cause things not to get regular or expedient updates. But this is just hypothesising.
Next steps for me are to update to 5.5. And then request that any future crashes are held so I can remote connect in the debugger. Are there any recommendations for ways to id the montage involved or state of it - e.g. adding more info into the FAnimTickRecord?
Regards,
John
It does look more and more like this is some kind of memory stomping. So you could try passing -stompmalloc or -stomp2malloc on the cmd line to see if those throw out anything useful. You can also try an ASAN build to see if that catches anything. That’s a bit more involved - you have to run UBT with the -EnableASan argument*.*
In terms of tracking down the asset, does the this pointer look valid? UAnimMontage::TickAssetPlayer should be being called on the montage asset for which you’re updating the montage instance. It’s also worth mentioning that UAnimMontage::TickAssetPlayer is only called on montages that are being blended via sync marker blending, so it has to be a montage asset that has UAnimMontage::SyncGroup set. If you go higher up the callstack, you may also be able to see the sync group.
Hello Euan, Sorry for delay - busy with the aforementioned 5.5 move and other work.
I can’t see the asset name in the given minidumps. The latest minidump is telling me that the this pointer of the FAnimSync in the frame is null (That is to say Sync.TickAssetPlayerInstances() is calling it a Sync variable that is null); but it’s getting so far into the function that this doesn’t make sense. I think for both prior reasons I will need to try catch it with a better crash dump or in a debugger.
So far I do not have any reports of the issue in 5.5, but it may be just a matter of time. I’ll try update soon when I know more.
No problem, sure let me know once you have more info. Did you have a chance to try with any of the memory stomping tools enabled?
Thanks for looking into it Euan.
I note 5.5 has some changes around ConditionalFlushCompletedMontages() and some bEarlyOutMontageWhenUninitialized logic. Would these relate to the CL fix you have mentioned? A move to 5.5 is soon on the cards for us, so if that’s the best solution here I’d take it.
Not yet. We were waiting for some more repos after the move to 5.5. As of yet, it hasn’t reoccured. Of course it might not be a bad idea to do this anyway just to check and see if it fires. Will keep you posted.