Guaranteed client crash shortly after joining a session with Chaos Mover resimulation (access violation in TestModeParticleDisabled)

Summary

In a packaged Win64 Development build using the experimental Chaos Mover plugin (pawn = Blueprint with a physics-simulated capsule root + ChaosCharacterMoverComponent), with Async Physics + Physics Prediction enabled and a listen-server session, the JOINING client crashes ~0.4 s after ClientTravel completes, in roughly 2 out of 3 networked runs, with EXCEPTION_ACCESS_VIOLATION reading 0xffffffffffffffff on a physics worker thread.

Log timeline on the crashing client:

  1. UPendingNetGame::TravelCompleted Pending net game travel completed
  2. LogChaosMover: ChaosMoverBackend: Setting Physics Replication Mode to Resimulation for BP_MoverPawn_C_... (x2)
  3. LogChaos: PBDRigidsSolver::EnableRewindCapture - Starting physics data history caching for rewind / resimulation. History Size: 32. Supported Latency: 1000.000000. TickRate: 30.
  4. ~0.4 s later: the access violation.

So post-travel world bring-up (particles added/removed in bursts) races the freshly-enabled rewind capture.

Root cause (verified against 5.8 sources, PBDRigidsEvolution.cpp, block under #if CHAOS_EVOLUTION_COLLISION_TESTMODE, ~lines 1348-1401):

  • FPBDRigidsEvolutionBase::TestModeParticleDisabled() (~line 1358) is the ONLY TestMode* function in that block MISSING the early-out if (!CVars::bChaos_Solver_TestMode_Enabled) { return; }. Its siblings TestModeSaveParticles (~1366), TestModeSaveParticle (~1379) and TestModeUpdateSavedParticle (~1397) all have the guard.
  • As a result, even with solver test mode DISABLED (the default), TestModeParticleDisabled still dereferences the incoming particle handle via CastToRigidParticle(). When DisableParticle is reached with a stale handle (see callstack: invoked from the TArrayCollectionArray::RemoveAtSwap lambda inside ProcessPushedData_Internal while ConditionalApplyRewind_Internal runs), that dereference reads freed memory and crashes.

Suggested fix: add the missing bChaos_Solver_TestMode_Enabled early-out to TestModeParticleDisabled (one line, consistent with its siblings), and separately investigate how a stale particle handle can reach DisableParticle from the pushed-data path during an active rewind.

No user-side workaround exists: the missing check is compile-time code (not CVar-gated), and CHAOS_EVOLUTION_COLLISION_TESTMODE is compiled into the stock binary engine build.

What Type of Bug are you experiencing?

Simulation

Steps to Reproduce

  1. UE 5.8.1 project with plugins Mover + ChaosMover (+ MoverExamples for the pawn base) enabled.
  2. Project Settings: Enable Async Physics (fixed tick 30 Hz) + Enable Physics Prediction (MaxSupportedLatencyPrediction = 1000).
  3. A pawn Blueprint: capsule root with Simulate Physics = true (collision preset Pawn) + ChaosCharacterMoverComponent; a replicated physics prop on the map with Physics Replication Mode = Resimulation.
  4. Package for Windows, Development, DX12/SM6.
  5. Run two instances; host a listen-server session on one, join from the other (we use EOS P2P sessions; the transport is likely irrelevant).
  6. In ~2 of 3 attempts the JOINING client crashes within ~1 s of travel completion.

Expected Result

Client joins and plays; world bring-up must not race the rewind system into dereferencing freed particle handles.

Observed Result

EXCEPTION_ACCESS_VIOLATION reading 0xffffffffffffffff on a physics worker thread (“Runnable thread Foreground Worker #1 crashed”), ~0.4 s after travel completion. Reproduced on multiple runs (~2 of 3). A similar sporadic AV inside Chaos.dll also occurs in-editor a few frames after PIE startup with the same plugin setup — same failure family (bring-up churn vs rewind).

Affects Versions

5.8

Platform(s)

Windows

For crash reports, include your callstack

LogThreadingWindows: Error: Runnable thread Foreground Worker #1 crashed.
Unhandled Exception: EXCEPTION_ACCESS_VIOLATION reading address 0xffffffffffffffff
[Callstack] EOS_OSS_Tutorial.exe!Chaos::FPBDRigidsEvolutionBase::TestModeParticleDisabled()
[Callstack] EOS_OSS_Tutorial.exe!Chaos::FPBDRigidsEvolutionBase::DisableParticle()
[Callstack] EOS_OSS_Tutorial.exe!Chaos::TArrayCollectionArray<TUniquePtr<Chaos::TGeometryParticleHandleImp<double,3,1>,TDefaultDelete<Chaos::TGeometryParticleHandleImp<double,3,1>>>>::RemoveAtSwap'::2’::<lambda_1>::operator()()
[Callstack] EOS_OSS_Tutorial.exe!Chaos::FPBDRigidsSolver::ProcessPushedData_Internal()
[Callstack] EOS_OSS_Tutorial.exe!Chaos::FPhysicsSolverProcessPushDataTask::ProcessPushData()
[Callstack] EOS_OSS_Tutorial.exe!Chaos::FSolverTasksPTOnly::AdvanceSolver()
[Callstack] EOS_OSS_Tutorial.exe!Chaos::FPBDRigidsSolver::ConditionalApplyRewind_Internal()
[Callstack] EOS_OSS_Tutorial.exe!Chaos::FPhysicsSolverRewindTask::RewindSolver()
[Callstack] EOS_OSS_Tutorial.exe!TGraphTaskChaos::FPhysicsSolverRewindTask::ExecuteTask()
[Callstack] EOS_OSS_Tutorial.exe!UE::Tasks::Private::FTaskBase::TryExecuteTask()
[Callstack] EOS_OSS_Tutorial.exe!LowLevelTasks::FTask::ExecuteTask()
[Callstack] EOS_OSS_Tutorial.exe!LowLevelTasks::FScheduler::ExecuteTask()
[Callstack] EOS_OSS_Tutorial.exe!LowLevelTasks::FScheduler::WorkerLoop()
[Callstack] EOS_OSS_Tutorial.exe!FThreadImpl::Run()
[Callstack] EOS_OSS_Tutorial.exe!FRunnableThreadWin::Run()
[Callstack] EOS_OSS_Tutorial.exe!FRunnableThreadWin::GuardedRun()

Additional Notes

System: UE 5.8.1 binary (Epic Games Launcher), Windows 11 Pro (10.0.26200), NVIDIA GeForce RTX 4090 (driver 610.88), DX12 SM6, packaged Win64 Development build (BuildCookRun -pak -iostore -compressed).

We reported a separate PSO-precache startup crash earlier today from this same account (r.PSOPrecache.KeepInMemoryUntilUsed on NVIDIA). Same machine and project.

Disclosure: the crash was found by the account owner while play-testing; the root-cause diagnosis and this write-up were done by Claude (Anthropic’s AI assistant) from log and engine-source analysis, and the account owner reviewed and authorized this submission.

that 0.4s-after-join crash on the physics worker with resim is almost always state arriving before the backend finished wiring up the client actor. the log line you pasted (Setting Physics Replication Mode to Resimulation x2) right before EnableRewindCa points at the rewind cache being touched during initial reconcile.

things i would check in order: make sure async physics framerate settings are identical in DefaultEngine.ini on both ends (fixed frame rate, same substep count), a mismatch makes the client allocate a different history size than the server expects. then test with p.Chaos.Solver.EnableRewindCapture 0 on the client for one run, if the crash goes away you confirmed its the rewind path and not your pawn logic. also guard your OnPreSimulate path for null rigid handles in the first few frames, joining clients tick physics before the replicated bodies exist and a raw handle read gives exactly that ffffffffffffffff read.

packaged dev build only makes it worse because the worker assert is compiled out, run one PIE listen server plus one packaged client if you can, you get a cleaner callstack. and if you are still on 5.7 try 5.8, there were resim stability fixes in that window.

background on why resim is picky here: Networked Physics - Fundamentals