Crash (EXCEPTION_ACCESS_VIOLATION): Mover RollbackBlackboard race between anim-thread trajectory prediction and game-thread rollback - reproducible on pawn teleport

General Issues and Bug Reporting (Unreal Engine)
,
1

Summary

URollbackBlackboard (Mover plugin, Experimental) has no locking between its prediction bracket and its simulation/rollback brackets.

A Motion-Matching character whose AnimBP thread-safe update calls Pose Search Generate Transform Trajectory (With Predictor) runs UMoverComponent::GetPredictedTrajectory on a parallel animation worker thread. The prediction loop mutates the live simulation’s RollbackBlackboard via TrySet_Predictive — every GenerateMove ends with SimContext.Blackboard.TrySet(DidGenerateMoveEntry, true) (SimpleWalkingMode.cpp:123).

When the game thread concurrently invalidates / rolls back the same blackboard — which is exactly what happens when the pawn is teleported through the Mover simulation (instant movement effect / TeleportEffect) — both threads mutate the same BlackboardEntry storage and the editor crashes with EXCEPTION_ACCESS_VIOLATION inside TrySet_Predictive.

The phase brackets only guard against themselves: BeginRollback checks !bIsSimulationInProgress && !bIsRollbackInProgress but does not check bIsPredictionInProgress, so rollback and prediction are allowed to overlap. The engine source itself acknowledges the missing lock:

// TODO: Need locking mechanism (BeginRollback)
// TODO: need some kind of lock so we wait on any in-progress operations before advancing the CurrentSimTimeStamp (EndSimulationFrame)

In normal locomotion the overlap window is small (rare sporadic crashes); with teleports it becomes near-deterministic.

What Type of Bug are you experiencing?

AI

Steps to Reproduce

  1. Set up a Mover-based character (e.g. the Mover character from the UE 5.7/5.8 Game Animation Sample) whose Animation Blueprint calls Pose Search Generate Transform Trajectory (With Predictor) from its thread-safe update function.
  2. Keep parallel animation update enabled (the default: a.ParallelAnimUpdate=1).
  3. Add a teleporter that teleports the pawn through the Mover simulation: QueueInstantMovementEffect with a Teleport instant movement effect.
  4. Play In Editor and use the teleporter repeatedly.
  5. The editor crashes with the callstack below, usually within a handful of teleports.

Expected Result

Prediction, simulation and rollback on URollbackBlackboard are mutually excluded (or prediction runs against a snapshot copy), so calling trajectory prediction from a worker-threaded AnimBP update is safe alongside teleports/rollbacks. No crash.

Observed Result

Concurrent mutation of the same blackboard entries from two threads: the animation worker thread is inside TrySet_Predictive while the game thread invalidates/rolls back the same RollbackBlackboard on teleport. The entry pointer read in TrySet_Predictive dereferences freed/relocated memory → EXCEPTION_ACCESS_VIOLATION, editor crashes.

Near-100% reproducible with repeated teleports; rare sporadic crashes during normal locomotion.

Affects Versions

5.8

Platform(s)

Windows

For crash reports, include your callstack

Unhandled Exception: EXCEPTION_ACCESS_VIOLATION reading address 0x000001d3da863e00

UnrealEditor_Mover!URollbackBlackboard::TrySet_Predictive<bool>() [D:\build\++UE5\Sync\Engine\Plugins\Experimental\Mover\Source\Mover\Public\MoveLibrary\RollbackBlackboard.h:460]
UnrealEditor_Mover!USimpleWalkingMode::GenerateMove_Implementation() [D:\build\++UE5\Sync\Engine\Plugins\Experimental\Mover\Source\Mover\Private\DefaultMovementSet\Modes\SimpleWalkingMode.cpp:123]
UnrealEditor_Mover!UBaseMovementMode::GenerateMove() [D:\build\++UE5\Sync\Engine\Plugins\Experimental\Mover\Intermediate\Build\Win64\UnrealEditor\Inc\Mover\UHT\MovementMode.gen.cpp:153]
UnrealEditor_Mover!UMoverComponent::GetPredictedTrajectory() [D:\build\++UE5\Sync\Engine\Plugins\Experimental\Mover\Source\Mover\Private\MoverComponent.cpp:2439]
UnrealEditor_Mover!UMoverTrajectoryPredictor::Predict() [D:\build\++UE5\Sync\Engine\Plugins\Experimental\Mover\Source\Mover\Private\MoverPoseSearchTrajectoryPredictor.cpp:38]
UnrealEditor_Mover!UMoverTrajectoryPredictor::Predict() [D:\build\++UE5\Sync\Engine\Plugins\Experimental\Mover\Source\Mover\Private\MoverPoseSearchTrajectoryPredictor.cpp:21]
UnrealEditor_PoseSearch!UPoseSearchTrajectoryLibrary::PoseSearchGenerateTransformTrajectoryWithPredictor() [D:\build\++UE5\Sync\Engine\Plugins\Animation\PoseSearch\Source\Runtime\Private\PoseSearchTrajectoryLibrary.cpp:848]
UnrealEditor_PoseSearch!UPoseSearchTrajectoryLibrary::execPoseSearchGenerateTransformTrajectoryWithPredictor() [D:\build\++UE5\Sync\Engine\Plugins\Animation\PoseSearch\Intermediate\Build\Win64\UnrealEditor\Inc\PoseSearch\UHT\PoseSearchTrajectoryLibrary.gen.cpp:1298]
UnrealEditor_CoreUObject!UObject::execCallMathFunction() [D:\build\++UE5\Sync\Engine\Source\Runtime\CoreUObject\Private\UObject\ScriptCore.cpp:1128]
UnrealEditor_CoreUObject!ProcessLocalScriptFunction() [D:\build\++UE5\Sync\Engine\Source\Runtime\CoreUObject\Private\UObject\ScriptCore.cpp:1265]
UnrealEditor_CoreUObject!ProcessScriptFunction<void (__cdecl*)(UObject *,FFrame &,void *)>() [D:\build\++UE5\Sync\Engine\Source\Runtime\CoreUObject\Private\UObject\ScriptCore.cpp:1092]
UnrealEditor_CoreUObject!`ProcessLocalFunction'::`2'::<lambda_1>::operator()() [D:\build\++UE5\Sync\Engine\Source\Runtime\CoreUObject\Private\UObject\ScriptCore.cpp:1343]
UnrealEditor_CoreUObject!ProcessLocalFunction() [D:\build\++UE5\Sync\Engine\Source\Runtime\CoreUObject\Private\UObject\ScriptCore.cpp:1362]
UnrealEditor_CoreUObject!ProcessLocalScriptFunction() [D:\build\++UE5\Sync\Engine\Source\Runtime\CoreUObject\Private\UObject\ScriptCore.cpp:1265]
UnrealEditor_CoreUObject!ProcessScriptFunction<void (__cdecl*)(UObject *,FFrame &,void *)>() [D:\build\++UE5\Sync\Engine\Source\Runtime\CoreUObject\Private\UObject\ScriptCore.cpp:1092]
UnrealEditor_CoreUObject!`ProcessLocalFunction'::`2'::<lambda_1>::operator()() [D:\build\++UE5\Sync\Engine\Source\Runtime\CoreUObject\Private\UObject\ScriptCore.cpp:1343]
UnrealEditor_CoreUObject!ProcessLocalFunction() [D:\build\++UE5\Sync\Engine\Source\Runtime\CoreUObject\Private\UObject\ScriptCore.cpp:1362]
UnrealEditor_CoreUObject!ProcessLocalScriptFunction() [D:\build\++UE5\Sync\Engine\Source\Runtime\CoreUObject\Private\UObject\ScriptCore.cpp:1265]
UnrealEditor_CoreUObject!UObject::ProcessInternal() [D:\build\++UE5\Sync\Engine\Source\Runtime\CoreUObject\Private\UObject\ScriptCore.cpp:1394]
UnrealEditor_CoreUObject!UFunction::Invoke() [D:\build\++UE5\Sync\Engine\Source\Runtime\CoreUObject\Private\UObject\Class.cpp:7595]
UnrealEditor_CoreUObject!UObject::ProcessEvent() [D:\build\++UE5\Sync\Engine\Source\Runtime\CoreUObject\Private\UObject\ScriptCore.cpp:2234]
UnrealEditor_Engine!UAnimInstance::BlueprintThreadSafeUpdateAnimation() [D:\build\++UE5\Sync\Engine\Intermediate\Build\Win64\UnrealEditor\Inc\Engine\UHT\AnimInstance.gen.cpp:917]
UnrealEditor_Engine!FAnimInstanceProxy::UpdateAnimation_WithRoot() [D:\build\++UE5\Sync\Engine\Source\Runtime\Engine\Private\Animation\AnimInstanceProxy.cpp:1355]
UnrealEditor_Engine!FAnimInstanceProxy::UpdateAnimation() [D:\build\++UE5\Sync\Engine\Source\Runtime\Engine\Private\Animation\AnimInstanceProxy.cpp:1249]
UnrealEditor_Engine!USkeletalMeshComponent::PerformAnimationProcessing() [D:\build\++UE5\Sync\Engine\Source\Runtime\Engine\Private\Components\SkeletalMeshComponent.cpp:2553]
UnrealEditor_Engine!USkeletalMeshComponent::ParallelAnimationEvaluation() [D:\build\++UE5\Sync\Engine\Source\Runtime\Engine\Private\Components\SkeletalMeshComponent.cpp:4898]
UnrealEditor_Engine!FParallelAnimationEvaluationTask::DoTask() [D:\build\++UE5\Sync\Engine\Source\Runtime\Engine\Private\Components\SkeletalMeshComponent.cpp:378]
UnrealEditor_Engine!TGraphTask<FParallelAnimationEvaluationTask>::ExecuteTask() [D:\build\++UE5\Sync\Engine\Source\Runtime\Core\Public\Async\TaskGraphInterfaces.h:703]
UnrealEditor_Engine!UE::Tasks::Private::FTaskBase::TryExecuteTask() [D:\build\++UE5\Sync\Engine\Source\Runtime\Core\Public\Tasks\TaskPrivate.h:524]
UnrealEditor_Engine!LowLevelTasks::TTaskDelegate<LowLevelTasks::FTask * __cdecl(bool),48>::TTaskDelegateImpl<`LowLevelTasks::FTask::Init<`UE::Tasks::Private::FTaskBase::Init'::`2'::<lambda_1> >'::`15'::<lambda_1>,0>::CallAndMove() [D:\build\++UE5\Sync\Engine\Source\Runtime\Core\Public\Async\Fundamental\TaskDelegate.h:171]
UnrealEditor_Core!LowLevelTasks::FTask::ExecuteTask() [D:\build\++UE5\Sync\Engine\Source\Runtime\Core\Public\Async\Fundamental\Task.h:623]
UnrealEditor_Core!LowLevelTasks::FScheduler::ExecuteTask() [D:\build\++UE5\Sync\Engine\Source\Runtime\Core\Private\Async\Fundamental\Scheduler.cpp:403]
UnrealEditor_Core!LowLevelTasks::FScheduler::WorkerLoop() [D:\build\++UE5\Sync\Engine\Source\Runtime\Core\Private\Async\Fundamental\Scheduler.cpp:802]
UnrealEditor_Core!`LowLevelTasks::FScheduler::CreateWorker'::`2'::<lambda_1>::operator()() [D:\build\++UE5\Sync\Engine\Source\Runtime\Core\Private\Async\Fundamental\Scheduler.cpp:223]
UnrealEditor_Core!FThreadImpl::Run() [D:\build\++UE5\Sync\Engine\Source\Runtime\Core\Private\HAL\Thread.cpp:69]
UnrealEditor_Core!FRunnableThreadWin::Run() [D:\build\++UE5\Sync\Engine\Source\Runtime\Core\Private\Windows\WindowsRunnableThread.cpp:167]

Additional Notes

Root-cause analysis (from the 5.8 sources)

  1. SimpleWalkingMode.cpp:123 — every GenerateMove ends with SimContext.Blackboard.TrySet(DidGenerateMoveEntry, true). Inside a prediction bracket this routes to TrySet_Predictive, which does FindEntry(ObjName) → Entry->SetValueAtTime_Predictive(...) (RollbackBlackboard.h:460). The entry pointer read here is where the access violation occurs — the entry storage is being mutated by another thread.

  2. UMoverComponent::GetPredictedTrajectory (MoverComponent.cpp ~2400-2470) runs the prediction loop directly against the live simulation objects: Simulation->RollbackBlackboard->BeginPredictionFrame(...) and CurrentMovementMode->GenerateMove(...) per sample, with no synchronization against the actual simulation. When called from an AnimBP thread-safe update function (the documented Motion Matching setup — the Game Animation Sample’s Mover character does exactly this), all of it runs on an animation worker thread.

  3. RollbackBlackboard.cpp — the phase brackets only guard against themselves, never against the prediction bracket:

    • BeginRollback: check(!bIsSimulationInProgress && !bIsRollbackInProgress) — bIsPredictionInProgress is not checked, so rollback and prediction are allowed to overlap. BeginRollback then iterates the whole EntryMap calling RollBack() on each entry, mutating the same storage the predictor is touching.
    • Epic’s own comments acknowledge the gap: // TODO: Need locking mechanism (BeginRollback) and // TODO: need some kind of lock so we wait on any in-progress operations before advancing the CurrentSimTimeStamp (EndSimulationFrame).

  4. Teleporting maximizes the race window: a teleport invalidates blackboard state on the game thread (Blackboard->Invalidate + RollbackBlackboard->BeginRollback in the rollback-capable path), touching every entry at once, while the anim worker predicts every frame.

Workaround (verified)

a.ParallelAnimUpdate=0 — forces the AnimBP update (and therefore the trajectory prediction) onto the game thread, serializing it with the Mover simulation. With this cvar the teleport repro no longer crashes.

Suggested fix direction

Either take a lock across the three brackets (as the existing TODOs suggest), extend the bracket checks so BeginRollback / BeginSimulationFrame wait for (or assert against) bIsPredictionInProgress, or run GetPredictedTrajectory against a snapshot copy of the blackboard instead of the live one.

Environment

UE 5.8.0 (binary Launcher build), Windows 11 Pro. Project: Motion-Matching character on the Mover plugin (Game Animation Sample-style setup).