Deep Dive into UE5 Transaction/Undo System Issues in Plugin Development – Need Community Guidance

Hello everyone,

I’m developing a plugin for Unreal Engine 5 and I’m running into major challenges with the undo/transaction system. I’ve spent a long time troubleshooting how to integrate my code with this system, but I still can’t figure out a reliable approach. I’m hoping others who’ve faced similar issues can shed some light on this.

Background

My plugin performs several undoable operations:

  • Modifying custom UObjects: For example, I have a class UCoinPileSeedEditor that manages certain properties. I mark it as transactional in its constructor:
UCoinPileSeedEditor::UCoinPileSeedEditor()
{
    SetFlags(RF_Transient | RF_Transactional);
}
  • Spawning preview actors: I spawn actors (e.g., ACoinPileSeedLevelPreview) to visualize changes. Their constructor also sets:
ACoinPileSeedLevelPreview::ACoinPileSeedLevelPreview()
{
    PrimaryActorTick.bCanEverTick = false;
    mStaticMeshComponent = CreateDefaultSubobject<UStaticMeshComponent>(TEXT("StaticMeshComponent"));
    RootComponent = mStaticMeshComponent;
    SetFlags(RF_Transient | RF_Transactional);
    bIsEditorOnlyActor = true;
}
  • I use FScopedTransaction to bracket modifications and expect the changes (including actor creation) to be undoable.

The Core Problem

Unreal’s transaction system has several limitations that make things difficult:

  1. No Nested Transactions:
    UE5 does not support nested transactions. For instance, inside the constructor of FScopedTransaction you have:
void FScopedTransaction::Construct(const TCHAR* TransactionContext, const FText& SessionName, UObject* PrimaryObject, const bool bShouldActuallyTransact)
{
    if(bShouldActuallyTransact && GEditor && GEditor->CanTransact() && ensure(!GIsTransacting))
    {
        Index = GEditor->BeginTransaction(TransactionContext, SessionName, PrimaryObject);
        check(IsOutstanding());
    }
    else
    {
        Index = -1;
    }
}

The ensure(!GIsTransacting) call forces a crash if a transaction is already active. But—and this is my main issue—I often cannot know in advance whether my function is being called inside an existing transaction or not.

  1. Conflicting Transactions with GEditor->AddActor:
    When spawning an actor with GEditor->AddActor, the code starts its own transaction:
FScopedTransaction Transaction(NSLOCTEXT("UnrealEd", "AddActor", "Add Actor"));
if (!(InObjectFlags & RF_Transactional))
{
    // Don't attempt a transaction if the actor we are spawning isn't transactional
    Transaction.Cancel();
}

If I’ve already started a transaction in my higher-level code, this internal transaction in AddActor gets canceled or conflicts, making it unpredictable whether my actor creation is properly recorded for undo.

  1. Lack of Visibility:
    There is no reliable API to detect if a transaction is already in progress (beyond checking the global GIsTransacting flag), nor any way to know the call hierarchy that might be wrapping my function. This makes it nearly impossible to decide whether to create a new transaction or let an existing one handle the changes.

  2. Actor Spawning Not Integrated with Undo:
    I tried two approaches for spawning preview actors:

  • Using the basic SpawnActor method:
UWorld* World = EditedCoinPile->GetWorld();
if (World)
{
    FTransform SpawnTransform(FRotator::ZeroRotator, NewSeedLocation);
    ACoinPileSeedLevelPreview* PreviewActor = World->SpawnActor<ACoinPileSeedLevelPreview>(ACoinPileSeedLevelPreview::StaticClass(), SpawnTransform);
    // ... attach, scale, and initialize actor ...
}
  • Using GEditor->AddActor:
ULevel* Level = EditedCoinPile->GetLevel();
if (Level)
{
    Level->Modify();
    FTransform SpawnTransform(FRotator::ZeroRotator, NewSeedLocation);
    AActor* NewActor = GEditor->AddActor(Level, ACoinPileSeedLevelPreview::StaticClass(), SpawnTransform, false, true);
    ACoinPileSeedLevelPreview* PreviewActor = Cast<ACoinPileSeedLevelPreview>(NewActor);
    // ... attach, scale, and initialize actor ...
}

Questions for the Community

  1. Handling Active Transactions:
    How do you detect or manage situations where your functions might be called inside an existing transaction? Do you use a wrapper or some custom logic to adjust behavior based on transaction state?

  2. Best Practices for Editor Operations:
    For complex editor tools, is it better to rely on a single high-level transaction to encapsulate all operations, or should individual actions always attempt to create their own transactions? How do you avoid conflicts when internal functions (like those in GEditor) also try to create transactions?

  3. Making Actor Spawning Undoable:
    Has anyone successfully ensured that actor spawns (especially for transient preview actors) are fully integrated into the undo system? What additional steps are required so that an undo not only reverts property changes but also removes the spawned actors?

  4. Further References and Workarounds:
    Are there any official guidelines, detailed examples, or community patterns that explain how to design undoable editor operations—particularly regarding the transaction system—in complex plugins?

Thank you for your time and any help you can offer!

Koromire

Why not make you own transaction manager?

The sprite editor seems to implement this in a way


void FSpriteEditorViewportClient::BeginTransaction(const FText& SessionName)
{
	if (ScopedTransaction == nullptr)
	{
		ScopedTransaction = new FScopedTransaction(SessionName);

		UPaperSprite* Sprite = GetSpriteBeingEdited();
		Sprite->Modify();
	}
}

You could keep an internal array of transaction actions (a custom struct) that could hold the transaction name, it’s action (begin or end) and then you could walk up the array and check if there is an ongoing transaction or if you are clear to start a new one.

@3dRaven Thank you the tips, that’s a cool way to handle a reentrant system.

I achieved avoiding crash with :

    FScopedTransaction Transaction(NSLOCTEXT("CoinPileSeedLevelPreview", "CoinPileSeedLevelPreviewDettachedFromEditor", "Coin pile seed level preview dettached from editor"), !GIsTransacting);
    Modify();

Which do something similar in the way it checks first if a transaction is opened, but also higher than the current class.

However, function like AddActor use a classic FScopedTransaction, so overriding the system doesn’t prevent the crash unfortunately.

ULevel* Level = EditedCoinPile->GetLevel();
if (Level)
{
    Level->Modify();
    FTransform SpawnTransform(FRotator::ZeroRotator, NewSeedLocation);
    AActor* NewActor = GEditor->AddActor(Level, ACoinPileSeedLevelPreview::StaticClass(), SpawnTransform, false, true);
}

I also tried a modified version of GEditor->AddActor with :

FScopedTransaction Transaction(NSLOCTEXT("UnrealEd", "AddActor", "Add Actor"), !GIsTransacting || !(InObjectFlags & RF_Transactional));
//if (!(InObjectFlags & RF_Transactional))
//{
//	// Don't attempt a transaction if the actor we are spawning isn't transactional
//	Transaction.Cancel();
//}

And that fixes the issue of the crash, but the spawned actor aren’t undoed, so i don’t have any solution for the moment.

So I found a solution to my spawning issue: when undoing, there was a specific execution path that caused the spawning to happen again immediately after the undo operation, creating the illusion that the spawning was not undone. In reality, it was correctly undone but then spawned once more.

I used World->SpawnActor and manually called Modify() on both the level and the UObject performing the spawn. This allowed me to avoid the transaction issue related to UEditorEngine::AddActor.

However, I am still unsure about this part, and it might be a bug:

FScopedTransaction Transaction(NSLOCTEXT("UnrealEd", "AddActor", "Add Actor"));
if (!(InObjectFlags & RF_Transactional))
{
	// Don't attempt a transaction if the actor we are spawning isn't transactional
	Transaction.Cancel();
}

The issue is that Cancel() is called after the ensure() crash in:

void FScopedTransaction::Construct(const TCHAR* TransactionContext, const FText& SessionName, UObject* PrimaryObject, const bool bShouldActuallyTransact)
{
	if (bShouldActuallyTransact && GEditor && GEditor->CanTransact() && ensure(!GIsTransacting))
	{
		Index = GEditor->BeginTransaction(TransactionContext, SessionName, PrimaryObject);
		check(IsOutstanding());
	}
	else
	{
		Index = -1;
	}
}

A potential fix could be:

FScopedTransaction Transaction(NSLOCTEXT("UnrealEd", "AddActor", "Add Actor"), !GIsTransacting || !(InObjectFlags & RF_Transactional));

This ensures that the transaction does not start at all if a higher-level transaction is already in progress.

It seems like a convenient solution to simply apply !GIsTransacting by default to every FScopedTransaction. However, since I am not very familiar with the transaction system, I might be overlooking deeper issues.

That being said, until I find a better approach, I will likely use this method moving forward.

1 Like