Gameplay Effect Crash In FAggregator Destructor

Development Programming & Scripting
, ,
1

We have a crash in our game stemming from duration based gameplay effects. The effect in question has a brief duration, ~5s, with a stack size of 1 and refreshes on reapplication. The effect may be applied frequently, potentially per frame, potentially multiple times in one frame. We are seeing a crash in gameplay attribute spec captures, but the effect in question does not capture any gameplay attributes. The only attributes we can observe are from UAbilitySystemComponent::GetOutgoingDurationCapture and UAbilitySystemComponent::GetIncomingDurationCapture, which as far as we can tell are defaults to effects with duration. This only appears to occur in packaged builds so far.

We also get a similar callstack in FActiveGameplayEffectsContainer::ApplyGameplayEffectSpec:4211, when calling UnregisterLinkedAggregatorCallbacks.

The crash also occasionally triggers from garbage collection. There is another thread with a similar callstack from garbage collection here: [Content removed] - this appears to be unresolved but gives us ideas to watch for. As far as we know we aren’t explicitly destroying pawn actors or ability systems, nor are we deleting attribute sets at runtime.

[Image Removed]

[Attachment Removed]

2

Steps to Reproduce
No reliable repro, just seems to happen. Some details of our setup:

  • Can occur in either single player or networked environment
  • There are AI controlled NPCs and player controlled pawns. The ability system is attached to the pawn actor, and every pawn has one.
  • The crash has triggered on both NPCs and players.
    [Attachment Removed]
3

Not seeing the callstack for some reason so pasting here:

[Inline Frame] FMallocBinned2::CanaryTest(const FMallocBinned2::FFreeBlock *) Line 308 C++

FMallocBinned2::Free(void * Ptr) Line 765 C++

FMemory::Free(void * Original) Line 688 C++

[Inline Frame] TSizedHeapAllocator<32,FMemory>::ForAnyElementType::{dtor}() Line 690 C++

[Inline Frame] TArray<FActiveGameplayEffectHandle,TSizedDefaultAllocator<32>>::{dtor}() Line 1000 C++

FAggregator::~FAggregator() Line 377 C++

SharedPointerInternals::TReferenceControllerWithDeleter<FAggregator,SharedPointerInternals::DefaultDeleter<FAggregator>,1>::DestroyObject() Line 386 C++

[Inline Frame] SharedPointerInternals::TReferenceControllerBase<1>::ReleaseSharedReference() Line 227 C++

SharedPointerInternals::TReferenceControllerBase<1>::ReleaseSharedReferenceNoInline(SharedPointerInternals::TReferenceControllerBase<1> * ReferenceController) Line 258 C++

[Inline Frame] SharedPointerInternals::FSharedReferencer<1>::{dtor}() Line 613 C++

DestructItems<FGameplayEffectAttributeCaptureSpec,int>(FGameplayEffectAttributeCaptureSpec * Element, int Count) Line 93 C++

[External Code]

TArray<FActiveGameplayEffect,TSizedDefaultAllocator<32>>::RemoveAtSwapImpl(int Index) Line 2144 C++

[Inline Frame] TArray<FActiveGameplayEffect,TSizedDefaultAllocator<32>>::RemoveAtSwap(int) Line 2188 C++

> FActiveGameplayEffectsContainer::DecrementLock() Line 6403 C++

[Inline Frame] FScopedActiveGameplayEffectLock::{dtor}() Line 6428 C++

FActiveGameplayEffectsContainer::CheckDuration(FActiveGameplayEffectHandle Handle) Line 5397 C++

[Inline Frame] Invoke(void(UMovieSceneSequencePlayer::*)(FFrameTime)) Line 66 C++

[Inline Frame] UE::Core::Private::Tuple::TTupleBase<TIntegerSequence<unsigned int,0>,FFrameTime>::ApplyAfter(void(UMovieSceneSequencePlayer::*)(FFrameTime) &) Line 326 C++

TBaseUObjectMethodDelegateInstance<0,UMovieSceneSequencePlayer,void __cdecl(void),FNotThreadSafeNotCheckedDelegateUserPolicy,FFrameTime>::Execute() Line 697 C++

[Inline Frame] TDelegate<void __cdecl(void),FNotThreadSafeNotCheckedDelegateUserPolicy>::Execute() Line 614 C++

FTimerUnifiedDelegate::Execute() Line 356 C++

FTimerManager::Tick(float DeltaTime) Line 1087 C++

UWorld::Tick(ELevelTick TickType, float DeltaSeconds) Line 1790 C++

UGameEngine::Tick(float DeltaSeconds, bool bIdleMode) Line 1883 C++

UILLGameEngine::Tick(float DeltaSeconds, bool bIdleMode) Line 215 C++

FEngineLoop::Tick() Line 5834 C++

[Inline Frame] EngineTick() Line 60 C++

GuardedMain(const wchar_t * CmdLine) Line 190 C++

GuardedMainWrapper(const wchar_t * CmdLine) Line 123 C++

LaunchWindowsStartup(HINSTANCE__ * hInInstance, HINSTANCE__ * hPrevInstance, char * __formal, int nCmdShow, const wchar_t * CmdLine) Line 277 C++

WinMain(HINSTANCE__ * hInInstance, HINSTANCE__ * hPrevInstance, char * pCmdLine, int nCmdShow) Line 335 C++

[External Code]

[Attachment Removed]

4

Hello! I remember the crash report that you linked, that was from the other user. In the end, we weren’t able to figure out the problem yet with the information that was provided.

The angle that I want to investigate with your crash report is whether the ~FAggregator destructor is executed twice, and what’s causing that. In general, the FAggregator lifetimes are managed by FAggregatorRef/TSharedPtr<FAggregator> and in-engine I couldn’t find any obvious problems with how lifetime of those is managed. With no clear suspects, I think the best course of action is to add some debug logic on your end in engine code to the ~FAggregator destructor. Let me provide some context and them a suggestion.

Thoughts so far

The callstack you shared shows the crash happening during the FAggregator’s destructor, when destructing elements in TArray<FActiveGameplayEffectHandle> Dependents. There it appears to be writing to memory that’s either already freed (FAggregator destructed twice), or that it doesn’t own (FAggregator data got overwritten in memory).

FAggregator double destruct “detection”

For starters, can you make the following engine modification in FAggregator in GameplayEffectAggregator.h/.cpp?

  • Add a field ‘uint32 DebugAliveState’
  • Define some magic numbers that represent the state of the FAggregator:
    • int32 AGG_ALIVE = 42
    • int32 AGG_DEAD = 1234
  • Initialize DebugAliveState as AGG_ALIVE, set it to AGG_DEAD in destructor
  • Ensure / log-fatal if DebugAliveState isn’t AGG_ALIVE on entering destructor
FAggregator::~FAggregator()
{
#if !UE_BUILD_SHIPPING
	if (DebugAliveState  != AGG_ALIVE)
	{
		if (DebugAliveState  == AGG_DEAD)
		{
			UE_LOG(LogTemp, Fatal, TEXT("AGG_DEAD detected: FAggregator for '%s' ran destructor twice."))
		}
		else
		{
			UE_LOG(LogTemp, Fatal, TEXT("Garbage state detected: FAggregator for '%s' either ran destructor twice, or got mem corrupted before destruction."))
		}
	}
 
	DebugAliveState  = AGG_DEAD;
#endif
 
	int32 NumRemoved = FScopedAggregatorOnDirtyBatch::DirtyAggregators.Remove(this);
	ensure(NumRemoved == 0);
}

I’m hoping to see AGG_DEAD twice, which would clearly illustrate an ~FAggregator’s destructor being called twice and then we’d just need to narrow down which engine code or game code is calling it incorrectly by capturing and investigating the callstacks.

Either way, if you add it then any outcome will give a strong hint on next steps. Can you add that, wait for the crash to happen again in a Development build (or in a shipping build, if you remove the #if !UE_BUILD_SHIPPING conditions). And then report back on whether the !AGG_ALIVE condition triggered at all?

If the condition triggers, try to get the callstack for both destructions of the same FAggregator. You may want to modify FActiveGameplayEffectsContainer::FindOrCreateAttributeAggregator too, to debug store the attribute’s name on the FAggregator which could prove helpful, but based on your story I suspect it’s going to be IncomingDuration.

[Attachment Removed]

5

Hey William, I’m glad to hear you found a clear answer to these crashes! Thanks for the update. I’ll close this.

[Attachment Removed]

6

Hi Zhi,

Thanks for getting back to me. Turns it out was an issue in our game code causing a heap corruption, which happened to nuke memory in GAS and triggered this callstack.

Appears fixed now after some fixes on our end.

[Attachment Removed]