So I found a solution to my spawning issue: when undoing, there was a specific execution path that caused the spawning to happen again immediately after the undo operation, creating the illusion that the spawning was not undone. In reality, it was correctly undone but then spawned once more.
I used World->SpawnActor and manually called Modify() on both the level and the UObject performing the spawn. This allowed me to avoid the transaction issue related to UEditorEngine::AddActor.
However, I am still unsure about this part, and it might be a bug:
FScopedTransaction Transaction(NSLOCTEXT("UnrealEd", "AddActor", "Add Actor"));
if (!(InObjectFlags & RF_Transactional))
{
// Don't attempt a transaction if the actor we are spawning isn't transactional
Transaction.Cancel();
}
The issue is that Cancel() is called after the ensure() crash in:
void FScopedTransaction::Construct(const TCHAR* TransactionContext, const FText& SessionName, UObject* PrimaryObject, const bool bShouldActuallyTransact)
{
if (bShouldActuallyTransact && GEditor && GEditor->CanTransact() && ensure(!GIsTransacting))
{
Index = GEditor->BeginTransaction(TransactionContext, SessionName, PrimaryObject);
check(IsOutstanding());
}
else
{
Index = -1;
}
}
A potential fix could be:
FScopedTransaction Transaction(NSLOCTEXT("UnrealEd", "AddActor", "Add Actor"), !GIsTransacting || !(InObjectFlags & RF_Transactional));
This ensures that the transaction does not start at all if a higher-level transaction is already in progress.
It seems like a convenient solution to simply apply !GIsTransacting by default to every FScopedTransaction. However, since I am not very familiar with the transaction system, I might be overlooking deeper issues.
That being said, until I find a better approach, I will likely use this method moving forward.