Rare crash in FWidgetAnimationState::OnStopped

Hello,

We’ve had a one-off crash report via Sentry which looks like it’s related to a stale delegate being called. The FWidgetAnimationState::OnStopped delegate is getting created when the widget anim state is being destroyed and then queued to be executed later. Something is happening between the queue and the execute which means the OnStopped function is now stale.

I haven’t looked too far into how this could be possible but an idea could be if ActiveAnimations reallocates in UUserWidget::GetOrAddAnimationState. This would move any pending delegates which means the resulting call is now pointing at garbage. I’m sure there are other places where this could be possible but I thought it was worth mentioning in the case it speeds up debugging.

I don’t think that this is something on our side breaking things but let me know if there’s any more data I can grab to help. We only have one instance of the crash so there’s not much to go off for us either.

[Attachment Removed]

I’m unable to see the callstack even though I did upload one.

EXCEPTION_ACCESS_VIOLATION_READ / 0x30: Fatal Error: EXCEPTION_ACCESS_VIOLATION_READ / 0x30
  WardogsClient-Win64-Test0x7ff65da624f7 SharedPointerInternals::FSharedReferencer<T>::{ctor} (SharedPointerInternals.h:570)
  WardogsClient-Win64-Test0x7ff65da624f7 TSharedPtr<T>::{ctor} (SharedPointer.h:1205)
  WardogsClient-Win64-Test0x7ff65da624f7 TWeakPtr<T>::Pin (SharedPointer.h:1514)
  WardogsClient-Win64-Test0x7ff65da624f7 FWidgetAnimationState::OnStopped (WidgetAnimationState.cpp:311)
  WardogsClient-Win64-Test0x7ff65da557cd Invoke (Invoke.h:66)
  WardogsClient-Win64-Test0x7ff65da557cd UE::Core::Private::Tuple::TTupleBase<T>::ApplyAfter (Tuple.h:326)
  WardogsClient-Win64-Test0x7ff65da557cd TBaseSPMethodDelegateInstance<T>::Execute (DelegateInstancesImpl.h:308)
  WardogsClient-Win64-Test0x7ff65d60c323 TDelegate<T>::Execute (DelegateSignatureImpl.inl:614)
  WardogsClient-Win64-Test0x7ff65d60c323 FMovieSceneEntitySystemRunner::GameThread_PostEvaluationPhase (MovieSceneEntitySystemRunner.cpp:1366)
  WardogsClient-Win64-Test0x7ff65d608846 FMovieSceneEntitySystemRunner::FlushNext (MovieSceneEntitySystemRunner.cpp:466)
  WardogsClient-Win64-Test0x7ff65d608b0a FMovieSceneEntitySystemRunner::FlushOutstanding (MovieSceneEntitySystemRunner.cpp:550)
  WardogsClient-Win64-Test0x7ff65da71cfa UUMGSequenceTickManager::ForceFlush (UMGSequenceTickManager.cpp:302)
  WardogsClient-Win64-Test0x7ff65da71cfa UUMGSequenceTickManager::TickWidgetAnimations (UMGSequenceTickManager.cpp:194)
  WardogsClient-Win64-Test0x7ff65d681815 Invoke (Invoke.h:66)
  WardogsClient-Win64-Test0x7ff65d681815 UE::Core::Private::Tuple::TTupleBase<T>::ApplyAfter (Tuple.h:326)
  WardogsClient-Win64-Test0x7ff65d681815 TBaseUObjectMethodDelegateInstance<T>::ExecuteIfSafe (DelegateInstancesImpl.h:713)
  WardogsClient-Win64-Test0x7ff65b5a64ab TMulticastDelegateBase<T>::Broadcast (MulticastDelegateBase.h:301)
  WardogsClient-Win64-Test0x7ff65b5a64ab TMulticastDelegate<T>::Broadcast (DelegateSignatureImpl.inl:1076)
  WardogsClient-Win64-Test0x7ff65b5c6881 FSlateApplication::TickAndDrawWidgets (SlateApplication.cpp:1728)
  WardogsClient-Win64-Test0x7ff65b5c65a2 FSlateApplication::Tick (SlateApplication.cpp:1640)
  WardogsClient-Win64-Test0x7ff65ef02364 FEngineLoop::Tick (LaunchEngineLoop.cpp:5965)
  WardogsClient-Win64-Test0x7ff65ef0bbdc EngineTick (Launch.cpp:60)
  WardogsClient-Win64-Test0x7ff65ef0bbdc GuardedMain (Launch.cpp:192)
  WardogsClient-Win64-Test0x7ff65ef0bcb9 GuardedMainWrapper (LaunchWindows.cpp:123)
  WardogsClient-Win64-Test0x7ff65ef0de79 LaunchWindowsStartup (LaunchWindows.cpp:277)
  WardogsClient-Win64-Test0x7ff65ef18dd3 WinMain (LaunchWindows.cpp:334)
  WardogsClient-Win64-Test0x7ff664b5c771 invoke_main (exe_common.inl:102)
  WardogsClient-Win64-Test0x7ff664b5c771 __scrt_common_main_seh (exe_common.inl:288)
  KERNEL32.DLL        0x7ffbd975e8d6 BaseThreadInitThunk
  ntdll               0x7ffbda86c3fb RtlUserThreadStart

[Attachment Removed]

I’ve just noticed that we have a much more common, very similar crash in Sentry.

EXCEPTION_ACCESS_VIOLATION_WRITE / 0x108: Fatal Error: EXCEPTION_ACCESS_VIOLATION_WRITE / 0x108
  WardogsClient-Win64-Test0x7ff63df0ecb6 FWidgetAnimationState::OnBegunPlay (WidgetAnimationState.cpp:295)
  WardogsClient-Win64-Test0x7ff63df0391d Invoke (Invoke.h:66)
  WardogsClient-Win64-Test0x7ff63df0391d UE::Core::Private::Tuple::TTupleBase<T>::ApplyAfter (Tuple.h:326)
  WardogsClient-Win64-Test0x7ff63df0391d TBaseSPMethodDelegateInstance<T>::Execute (DelegateInstancesImpl.h:308)
  WardogsClient-Win64-Test0x7ff63daba0d3 TDelegate<T>::Execute (DelegateSignatureImpl.inl:614)
  WardogsClient-Win64-Test0x7ff63daba0d3 FMovieSceneEntitySystemRunner::GameThread_PostEvaluationPhase (MovieSceneEntitySystemRunner.cpp:1366)
  WardogsClient-Win64-Test0x7ff63dab65f6 FMovieSceneEntitySystemRunner::FlushNext (MovieSceneEntitySystemRunner.cpp:466)
  WardogsClient-Win64-Test0x7ff63dab68ba FMovieSceneEntitySystemRunner::FlushOutstanding (MovieSceneEntitySystemRunner.cpp:550)
  WardogsClient-Win64-Test0x7ff63df1fe4a UUMGSequenceTickManager::ForceFlush (UMGSequenceTickManager.cpp:302)
  WardogsClient-Win64-Test0x7ff63df1fe4a UUMGSequenceTickManager::TickWidgetAnimations (UMGSequenceTickManager.cpp:194)
  WardogsClient-Win64-Test0x7ff63db2f595 Invoke (Invoke.h:66)
  WardogsClient-Win64-Test0x7ff63db2f595 UE::Core::Private::Tuple::TTupleBase<T>::ApplyAfter (Tuple.h:326)
  WardogsClient-Win64-Test0x7ff63db2f595 TBaseUObjectMethodDelegateInstance<T>::ExecuteIfSafe (DelegateInstancesImpl.h:713)
  WardogsClient-Win64-Test0x7ff63ba567cb TMulticastDelegateBase<T>::Broadcast (MulticastDelegateBase.h:301)
  WardogsClient-Win64-Test0x7ff63ba567cb TMulticastDelegate<T>::Broadcast (DelegateSignatureImpl.inl:1076)
  WardogsClient-Win64-Test0x7ff63ba76ba1 FSlateApplication::TickAndDrawWidgets (SlateApplication.cpp:1728)
  WardogsClient-Win64-Test0x7ff63ba768c2 FSlateApplication::Tick (SlateApplication.cpp:1640)
  WardogsClient-Win64-Test0x7ff63f3d15f4 FEngineLoop::Tick (LaunchEngineLoop.cpp:5965)
  WardogsClient-Win64-Test0x7ff63f3db13c EngineTick (Launch.cpp:60)
  WardogsClient-Win64-Test0x7ff63f3db13c GuardedMain (Launch.cpp:192)
  WardogsClient-Win64-Test0x7ff63f3db219 GuardedMainWrapper (LaunchWindows.cpp:123)
  WardogsClient-Win64-Test0x7ff63f3dd3d9 LaunchWindowsStartup (LaunchWindows.cpp:277)
  WardogsClient-Win64-Test0x7ff63f3e8333 WinMain (LaunchWindows.cpp:334)
  WardogsClient-Win64-Test0x7ff64507ae51 invoke_main (exe_common.inl:102)
  WardogsClient-Win64-Test0x7ff64507ae51 __scrt_common_main_seh (exe_common.inl:288)
  KERNEL32.DLL        0x7fff4669e956 BaseThreadInitThunk
  ntdll               0x7fff476c427b RtlUserThreadStart

[Attachment Removed]

Hey there,

We haven’t seen this one. I’ve logged an issue that you can follow here: https://issues.unrealengine.com/issue/UE-385279

Dustin

[Attachment Removed]

Apologies for the delay. Epic went on summer break after Unreal Fest. Yes, this has been fixed, and you’ll find backportable changes in the new Unreal 6 branch we’ve created. This one is two files.

https://github.com/EpicGames/UnrealEngine/commit/d757aa45f704f624f61ef505e97a1e236e3b9bb4

Dustin

[Attachment Removed]

Good day everyone.

Kieran is currently on vacation, so I had a look at the crashes, as we still got Sentry reports for it.

Ultimately, the fix [mention removed]​ linked probably fixed the issue partially, but I think it also needs a small additional change.

As already reported by Kieran, the callstack hits `FMovieSceneEntitySystemRunner::GameThread_PostEvaluationPhase (MovieSceneEntitySystemRunner.cpp:1366)`

That’s this code here:

for (const FSimpleDelegate& OnFlushed : TmpOnFlushedDelegates)
{
	OnFlushed.Execute();
}

The `TmpOnFlushedDelegates` is filled by swapping `OnFlushedDelegates` into it, at the top of the function.

Now, `OnFlushedDelegates` is filled in `::GameThread_UpdateSequenceInstances` like this:

// Always forward the OnFlushed delegate to be called at the end of the frame, even if the instance is no longer valid
if (Update.OnFlushed.IsBound())
{
	OnFlushedDelegates.Add(MoveTemp(Update.OnFlushed));
 
	// If we have any on-flushed delegates then we have to do a full Post-Eval phase
	AccumulatedUpdateFlags |= ESequenceInstanceUpdateFlags::NeedsPostEvaluation;
}

There are two things to point out.

  1. `if (Update.OnFlushed.IsBound())` makes sure that the delegate `IsSafeToExecute`, which means at this point it’s all valid.
  2. The comment above the if saying `even if the instance is no longer valid`

To me this means that we are expecting that the instance could become invalid from here until we actually call the `Execute()`.

In other words, `OnFlushed.Execute();` should become `OnFlushed.ExecuteIfBound();`.

That should™️fully resolve the issue. I think, it can happen that between `::GameThread_UpdateSequenceInstances` and `::GameThread_PostEvaluationPhase`, which are called within `::FlushNext`, the instance can indeed die, and there is nothing that checks anymore if the delegate `IsSafeToExecute`.

I haven’t submitted that change internally yet, but probably will to at least try and mitigate the crash.

Cheers.

[Attachment Removed]

As was mentioned, you could try OnFlushed.ExecuteIfBound(), or you could try OnFlushed.ExecuteIfSafe() in MoveSceneEntitySystemRunner.cpp (around 1371)and report back.

Dustin

[Attachment Removed]

Nope, I apologize; that was bad grammar and inference on my part. I mean that you found the possibly good path with ExecuteIfBound but wanted to suggest ExecuteIfSafe as well. We have come to similar conclusions.

Dustin

[Attachment Removed]

Great, thanks Dustin.

[Attachment Removed]

Hi Dustin,

I noticed that the link above now shows this issue as fixed. Is there any way I can see the fix so that I can cherry pick the changes? I assume it’s not available yet… what is version 42.20?

[Attachment Removed]

Hi Dustin,

No worries and thanks for the link. I’ve implemented the changes locally but we’re still seeing the second callstack that I sent through. Not sure about the first yet as we’ve only just started testing but hopefully that one has been solved.

I’ll update the thread if I do see the first issue pop up again.

[Attachment Removed]

Hey Dustin,

my bad, I didn’t read through this thread. I blindly assumed that the only stuff discussed was the UE-385279 and the fix commit.

However, that at least means we came to the same conclusion, which is worth something, right? :sweat_smile:

Thanks for letting me know. Appreciated!

Cedric

EDIT: Actually, this wasn’t mentioned anywhere (despite by me) unless I’m blind? So, I probably just misread your response as “This solution was already mentioned.” I will keep ExecuteIfSafe() in mind, thanks!

[Attachment Removed]