Tutorial: Networked Physics - Pawn Tutorial

General Tutorial & Course Discussions
85

Hi @MBobbo*, thanks for the incredibly detailed response — it clarified a lot about the resimulation architecture and fundamentally changed how I think about the physics thread vs game thread boundary.*

I have two follow-up topics, one about scalability and one about discoverability of physics thread APIs.

Scalability — Listen Server at 60Hz with 8+ players

My game is a Chaos physics-based party game targeting 8 players minimum on a Listen Server (no dedicated server). Every player is a physics pawn using NetworkPhysicsComponent with resimulation, and there will also be passive physics objects in the level (push carts, seesaws, puzzle cubes) that interact with the players.

I’m considering raising the physics tick rate from 30Hz to 60Hz for better collision quality (less tunneling on thin surfaces and fast impacts), but I’m concerned about the cumulative cost on a single Listen Server host machine that has to maintain physics cache history and process resimulations for all 8 players plus level objects simultaneously.

Is there a practical ceiling where the resimulation cache memory and CPU cost becomes prohibitive for a Listen Server scenario at 60Hz with this many physics actors? Or is this manageable with reasonable optimization (keeping cache history short, limiting resimulation depth, etc.)? Any guidance on what levers to pull would be greatly appreciated.

Physics Thread API Discoverability

I’m currently studying the Chaos Mover plugin source (specifically FloorSweep_Internal) as you recommended, to understand how to do physics thread queries. However, I’m finding it difficult to discover the full set of available physics thread APIs since the internal Chaos interfaces (like FGenericPhysicsInterface_Internal::SpherecastMulti, OnContactModification_Internal callback patterns, probe collision setup, etc.) don’t appear in Epic’s public API documentation.

Is there a reference or internal document that lists the available physics thread query and modification APIs that are safe to use inside ISimCallbackObject callbacks? Or is the recommended approach to read through the Chaos Mover and Chaos Vehicle plugin source code as living documentation?

Beyond FloorSweep_Internal, are there other places in the engine source (plugins, samples, or test code) that demonstrate patterns for a networked physics game — things like physics thread overlap detection, contact modification for custom impulse responses, probe collision setup, or physics thread force application between multiple interacting pawns?

Any pointers would save a lot of engine source spelunking. Thanks!

86

About: Scalability — Listen Server at 60Hz with 8+ players

  • If your passive physics objects don’t need any special state replicated then don’t put a NetworkPhysicsComponent on them, the component has overhead from just being there and is not needed for dynamic physics objects that you interact with.

  • If tunneling is your concern you could try using CCD (continuous collision detection) before raising the tick-rate. You could also try MACD (movement away collision detection) if you get tunneling between moving objects.

  • A very common misconception is that the server is performing resimulations, but it’s not. Resimulations is a way to both correct a replication error and to get objects into forward prediction. The server is the source of truth it never needs to correct itself because it never does anything “wrong”, only clients get things wrong with replication so they need to correct themselves (via resimulation for example). And the client is the one that is forwrad predicted ahead of the server so the client is the one that needs resimulation predict replicated physics objects ahead of the server.

    • 60hz effects on the server: Outgoing network traffic will increase a lot, the server is sending and processing ALOT more network data than clients do.

    • 60hz effects on the client: Rewind history grows (memory consumpton), resimulation frames increase (CPU consumtion).

      • Memory: Lower the MaxSupportedLatencyPrediction in Project Settings → Physics → Replication. It defaults to 1000ms (so at 60hz that would be 60 frames of history cached) if you lower it to 500ms then it would only want to cache 30 frames of history but you would then also not support clients that have over 500ms latency (which is very bad latency anyway). One thing to know here is that the actual frames cached is ceiled to the closest power of two (2,4,8,16,32,64,128) so 1000 at 60hz will cache 64 frames, 600ms at 60hz will also cache 64 frames, you need to get below 32 for it to cache 32frames if you want to lower memory use. Note that the size of the rewind history doesn’t really affect CPU much unless for clients that actually do have really bad networking and require to go back further in time to perform resimulations.

      • CPU: There isn’t that much to do in 5.7 to lower CPU usage except that isn’t enabled by default, you could disable this CVar: p.Resim.AllowRewindToResimulatedFrames. You can also raise the resim thresholds in the projects settings to not trigger resimulations as often. You could also try enabling np2.Resim.RuntimeCorrectionEnabled with and without np2.Resim.RuntimeVelocityCorrection, which is a way to try keep a resim object in sync without resimulating and if it fails it triggers a resim. But it can cause replication issues depending on what is being replicated, a vehicle with bouncu suspension can go into a feedback loop with that enabled for example. In 5.8 you will get more options to clamp down on resimulation consuming too much CPU, for example p.Resim.CooldownFrames N which makes resim not able to trigger for N frames after a resim was performed. And p.Resim.GuardFromPerformanceCollapse true which forces the same amount of real time to pass between each resimulation as the last resim took.

About: Physics Thread API Discoverability

  • Unfortunately the documentation for these APIs are limited to non-existing indeed, part of the reason is that some of the API is still experimental and we might change it.

  • There are no tutorials or sample projects that showcase / teach async physics or Networked Physics outside of this tutorial (to my knowledge).

    • I want to make more tutorials and will do when I get some more time, i.e. after UE 5.8 is locked down.

  • One issue that exist is also that on the physics thread you should not do anything with UObjects, which means you can’t safely check if the thing you hit is an APawn or AActor to perform different logic based on what you hit easily.

    • There is a feature in the works to attach UserData to physics particles but I don’t know if it’s available already or if it will be in 5.8 or maybe later. I’ll try to get a tutorial made when it’s available, or get the one who wrote the solution to do a tutorial.

    • But before I’d say that within the same scope of you doing a sweep on the physics thread you can assume it’s fairly safe to read and cast the UObjects you get back, never write to them though!

      • And if you need to add a force to an object you hit, don’t do it via the AActor or BodyInstance. Grab the physics particle and apply you physics effects on that one when on the physics thread, there is also API for some things in the physics evolution, like FPBDRigidsEvolutionGBF::SetParticleTransform.

    • So currently unfortunately you will most likely need to explore what you can do in OnContactModification_Internal, you’ll need to cache the particle that the current ISimCallbackObject is for, i.e. your pawn for example and then look for that particle inside the collection of contact pairs you get as the parameter property.

1 Like
87

Addition to my post above, ContactModification is done by both Chaos Mover and the Chaos Modular Vehicle

  • FAsyncCallback::OnContactModification_Internal - ChaosMoverAsyncCallback.cpp

    • UChaosMoverSimulation::ModifyContacts
      • UChaosCharacterMovementMode::ModifyContacts

  • FChaosSimModuleManagerAsyncCallback::OnContactModification_Internal

    • …
      • ISimulationModuleBase.OnContactModification
        • Unfortunately none of the default modules use this so there doesn’t look to be an implementation to look at where it’s actually used
88

@MBobbo thanks again for sharing the details earlier. We have now modeled our vehicle runtime quite closely after the ChaosMover reference architecture: a single world-level async SimCallbackObject, a WorldSubsystem that owns backend registration and callback lifetime and a per-vehicle backend that owns the corresponding simulation class and network physics input/state structs derived from FNetworkPhysicsData. In our case that is roughly the same split as UChaosMoverSubsystem, UChaosMoverBackendComponent, UE::ChaosMover::FAsyncCallback, UChaosMoverSimulation and FNetworkChaosMoverInputData / FNetworkChaosMoverStateData. Aside from the basic networked physics pawn tutorial, ChaosMover is really the only substantial reference we found for this area.

Because ChaosMover appears to be designed for a broader and more configurable use case than ours, we intentionally kept our implementation as small as possible and omitted pieces that did not seem essential for our game.

Our game is a multiplayer arcade car battle with vehicle abilities, combat effects and adapted collision/knockback model, so getting the async physics + networked physics architecture right is critical for us.

If you have time, we would really appreciate guidance on a few points that are still unclear to us:

  1. Contact handling / gameplay reaction
    Right now we use one central OnContactModification_Internal callback in the shared sim callback. There we identify our vehicles via the particles’ UniqueIdx, map the pair back to our registered backends and forward the contact to both UVehicleSimulation instances. Inside UVehicleSimulation::OnContact we currently compute the collision response and immediately apply the resulting impulse to the rigid particle and we also update our health/damage state there.
    Does this sound like the right place / pattern for vehicle-to-vehicle and ability-to-vehicle collision gameplay? In particular, is ContactModification the right hook for identifying these contacts and is it reasonable to apply gameplay consequences such as impulse and damage immediately there or would you recommend separating detection from gameplay state mutation?

  2. TimeStep usage in ChaosMover
    We noticed ChaosMover uses FMoverTimeStep throughout the flow and we are not fully sure what role it plays in practice. We are looking at things like UChaosMoverSubsystem::InjectInputs_External, UE::ChaosMover::FAsyncCallback::GetCurrentMoverTimeStep, UChaosMoverSimulation::ProcessInputs and UChaosMoverSimulation::SimulationTick.
    More specifically, how important are fields such as ServerFrame, BaseSimTimeMs, StepMs and bIsResimulating for a setup like ours? If you are not doing kinematic movement or more advanced movement-mode/state-machine logic, are these fields still something you should model from the beginning for a networked physics vehicle game?

  3. Interpolated outputs / finalized frame approach
    In ChaosMover we saw that outputs are interpolated in FNetworkChaosMoverStateData::InterpolateData and then later consumed through the finalized-frame path in UChaosMoverBackendComponent::FinalizeFrame, where the interpolated result is created and the transform is then overwritten with the actual particle transform.
    Our question is less about that split itself and more about the overall concept: is this finalized-frame approach generally the intended pattern for networked physics? In other words, should we generally think of sim outputs as something that gets interpolated for presentation while transform-related data should ultimately be taken from the finalized particle state at the end of the frame? We are mainly trying to understand what problem this pattern is solving and whether it is something we should follow as well.

  4. On/off abilities or one-shot effects like an instant heal
    How do you usually handle things like an instant heal? Our car health currently lives in the physics thread and gets sent to the game thread for UI but once networking comes into play we are unsure what the best approach is with rewind + resimulation. Could something like that accidentally get triggered twice during resim or can we trust the UE networked physics internals to make sure there are no cases where the same effect gets applied twice?

Any pointers would be hugely appreciated. This part of the codebase is by far the most important and also the hardest to validate confidently because there is so little documentation around these patterns.

1 Like
90

Hello @FerAram, nice to see that you are making progress.

  1. That’s a valid approach yes, OnContactModification_Internal handles resimulations so you will get the call both during normal simulation frames and during resim so you can act the same during both. If you have damage / health etc. in your networked state that gets replicated via NetworkPhysicsComponent, then it is also part of the state that gets corrected during a rewind and resim so then it’s viable to do damage prediction in OnContact during resim. Though it can also be a bit dangerous predicting damage on the clients but that’s a design choice, usually it’s just server executed, and if that’s your approach then it’s still valid to do in OnContact, the server never resim so there will be no issues with double application etc.
  2. I’m not directly working on the ChaosMover solution so I don’t know all the reasons but I expect the FMoverTimeStep to be mainly a helper to easily get the ServerFrame, DeltaTime and resim bools instead of always getting them from their normal API locations. So it can be a help to have such a struct but not required no.
  3. Again I don’t have that much ChaosMover knowledge, but from my understanding the reason here is to align state dictated by the physics thread to align with the game thread. Game thread is always representing physics behind in time, there are usually 2 physics results already simulated and ready which the game thread then interpolate between to get smooth movement. For example if you tick physics at 30hz and your game renders at 120hz then the game thread will interpolated between two physics results 4 times, else you would see things standing still for 3 frames and then snap 1 frame. To align the state that the chaos mover decided on during the physics thread to where the game thread is actually showing things the state can be interpolated to match. This is not really required, but for a solution like ChaosMover which should handle a vast amount of features and be able to be expanded etc. this type of accuracy and alignment is important. No matter if you do this or not the physics movement will still be smooth since that’s handled by Chaos and Unreal Engine. One thing that is a bit linked to this and questio 2 is that Chaos Mover perform deduplication of internal mover events that gets triggered from the physics thread, it does that in UChaosMoverSimulation::UpdateEvents, so if you also have events that should get triggered on the physics thread of game thread only once no matter resims then looking at how dedupe is done might be good.
  4. This is linked to question 1, if the heal is just server-side then you will not run any risk of getting duplicates (as long as you guard it correctly, don’t trust inputs from a client and apply the heal without checking if you actually can/should apply a heal). If you want to predict health (healing and damage) then I’d recommend having the health as part of the networked state so it’s on the physics thread and part of the rewind / resim. The NetworkPhysicsComponent only handles inputs and states in UE 5.7 and on the server it can reuse last, interpolate or merge inputs depending on what happens with the input buffer, and on sim-proxies it will also reuse last, interpolate and merge inputs if they come irregularly and out of order.. So don’t trust inputs to always be applied in the exact same order as done on the autonomous proxy player, your state-machine should be robust enough to handle getting thrown any input combination without producing the wrong state (expect players to try to cheat via altering inputs, that’s the only way they can attempt to cheat with the NetworkPhysicsComponent). With UE 5.8 there is a new Actions system in the NetworkPhysicsComponent, also cheat proof as long as you use it correctly but it allows easier synchronization of one-off events. I’ll make a tutorial for that when 5.8 releases.
1 Like
91

Hi @MBobbo

I saw that recently branch for 5.8 was created, what are the core updates to expect in comparison with current stable 5.7 release? Super exited to try and check what you are cooking guys

1 Like
92

Hello, I’ll make a post later when 5.8 is locked down listing things that have changed / been added to networked physics.

I also aim to get two new tutorials up when 5.8 is released, one more of an information dump about fundamentals around networked physics and the other about networked physics gameplay implementations as a continuation on the current custom pawn tutorial.

5 Likes
93

Hi! @MBobbo

I’m doing some extra experiments with network physics and I just wanted to revisit some basics of the current approach, could you confirm these are true or I get them wrong?

  1. Resimulation works for the whole physics scene, so there is no difference in cost regarding what triggered this resim

  2. In a current approach resimulation is about to happen quite often in the game and it’s normal, since every time the input changes in a way we can’t predict (which is almost everytime) we have state divergence and we have to trigger the resim.

  3. With some fine tuning of the decay parameter there is a small chance to reduce the amount of resims for some simple movement patterns, but mostly it has to help with visual overshoot-correction phase

  4. Both predictive interpolation and resimulation modes can work together. It’s expected that for pawns with complex inputs we use resimulation and for regular objects in the world with physics we go with predictive interpolation

  5. Switch between these in runtime could be done with Physics Replication LOD but was not really ready at 5.7. Maybe there were some updates in 5.8? Are we able to switch between them manually to specify some extra priority on the object we are interacting with for example?

  6. p.Resim.AllowRewindToResimulatedFrames changing that to false basically allows us to reduce the amount of resims but at the same time we will see less smooth motion of complex pawns

Overall, seems like I’ve managed the system to work with quite a number of objects when I changed the physics mode for non-pawns to predictive interpolation and disabled p.Resim.AllowRewindToResimulatedFrames (but for sure it does a great job smoothing the experience, but it drains the CPU with large number of objects quite fast and clients are suffering)

Looking forward to see the new stuff from 5.8! Thank you guys again

UPDATE:

Built UE5.8 from source and exported the project with network physics from 5.7. No issues during compilation, but without adjusting anything now it looks like there is no client immediate simulation or something like that, so I feel heavy delay on my actor before it moves. I wonder may be there is some important parameter set to different value by default in a new version?

94

Hi @MBobbo! First of all thank you very much for this tutorial, and in helping us with using physics in network play. Now that 5.8 preview is up, I can’t wait for further documentation on this topic.

If I may ask, could we have some kind of flowchart to better understand it? At the moment it is quite daunting with so many _internal and internally called functions.

Another question, is it good practice to have multiple PhysicsObject being handled in a single Async class? In my project I have a pawn with multiple physics-enabled static meshes, and I’m not sure if I should pass each static mesh with their own inputs and resimulation logics.

Again, thanks for writing these guides. As someone that has been trying to use physics in unreal engine for a few years, I can say this has been a boon for me and many others.

95

Hello @Houdsonin, I can probably put together a flowchart from Autonomous Proxy input being pressed to the server receiving and applying it to the Simulate Proxy receiving and applying input and state. Covering the flow between game thread / physics thread and networking flow client-server-client. I’ll add that on my todo for the “fundamentals” post / tutorial I’m planning.

About multiple physics objects for a single pawn, it’s generally best to have as few NetworkPhysicsComponents as possible since they consume a bunch of network bandwidth etc. and if you have a setup where you have one async TSimCallbackObject for the pawn managing multiple objects that also sounds like you have one NetworkPhysicsComponent. I’d say that’s a good setup.

On another note, If the simulated objects are child components of the APawn then their physics states will not be replicated correctly and used in resimulation, only the root component is properly replicated with physics state from the server to clients. So you’d need to replicate each simulated child object yourself inside your custom State that replicates via the NetworkPhysicsComponent.

If the physics objects are all separate actors but registered to one pawn, hooked up via constraints for example (or just freely simulating) then you are already covered, each of them will replicate their physics state via their root components (AActor::ReplicatedMovement) and they will rewind and resimulate correctly. Only the APawn or AActor that you run your logic on needs to be possessed / owned by your client, other physics objects that you want to control can be owned by the server (default for any non pawn) since you can still modify them predictively client-side from your inputs. The server should do the exact same thing to them when applying the same inputs.

1 Like
96

Hello @mizarates

  1. True (but we have a first iteration of a “bubble resim” system coming in 5.8 which would not resim the whole physics scene)
  2. Sim-proxy pawns rely on constant resimulations to stay in sync yes. Autonomous proxy and sim-proxy objects can simulate predictively without triggering resimulations. There is a “runtime correction” system also which can try to correct a desync before it causes a resim, but it’s not enabled by default since it can cause replication inconsistencies. You can enable it with np2.Resim.RuntimeCorrectionEnabled 1 It runs when the desync is less than the resim trigger limit you set in the Project Settings → Physics → Replication →
  3. Performing input decay will cause more resimulations than not applying a decay, the goal of it is to lower the average desync at the end of a resim, which makes sim-proxy players look better since they don’t need to cover up as large desyncs after resimulations.
  4. In 5.7 Resim and PI doesn’t work well together because resimulation applies on all physics objects, which brings the PI replicated object into the forward predicted timeline and then after a resim PI will bring it back into the interpolated timeline, so PI replicated objects can look jittery moving forwards and backwards when resim happens. In 5.8 this is somewhat improved, PI objects will still simulate during resimulation but at the end of the resim the PI object is set back to their state as it was before the resim, which makes them not get affected by resim in the same way, they don’t jitter back and forth etc. The more proper approach is to use Physics Replication LOD along with Bubble Resim where the LOD dictates the replication mode (Resim or PI) and the timeline they should replicate in and Bubble resim only affect objects that close to a desynced object that can trigger resimulation. LOD exists, Bubble resim exists, but they are not yet piped up to actively take eachother into consideration. But in 5.8 you should theoretically be able to use that system.
  5. You can switch manually between the replication modes at runtime, there is API for that on the game thread (AActor::SetPhysicsReplicationMode). PhysicsReplicationLOD works but it can only transition timelines by extrapolating targets that are used by predictive interpolation, and this extrapolation doesn’t produce good replication in many scenarios like when a ball bounces for example, the extrapolation will put the target through the ground and then when it receives a target just after the bounce it will extrapolate that up into the air, so the replication becomes erratic inside the transition zone. This is better in 5.8 but at the cost of CPU, you make the LOD timeline transition to use SimulationDecay which relies on resimulations so while an object is inside the transition zone it will constantly trigger resimulations and it will decay the simulation during those resims to end up in the correct timeline, to gradually transition from the interpolated timeline to the forward predicted timeline.
  6. Yes that will reduce resimulations but at the cost of more desync that needs to be covered up visually by render interpolation. In 5.8 there is also other systems to lower the cost of resimulation, there is a cooldown system so you can choose to not allow multiple resimualtions in a row. There is also a resim frame coalescing system where if it predicts that this whole resim will overshoot the allocated CPU budget it starts to take double or tripple steps (one step with double or tripple delta time) to reduce the CPU cost, but at the cost of less determinism (more desync), this is not enabled by default.

About your 5.8 conversion, I don’t know what could be going on there, first thing I’d check is if the replication mode is using resim or not: p.Net.DebugDraw.ShowRepMode 1 and p.Chaos.DebugDraw.Enabled 1 to see the actual mode that gets applied as debug drawing, red = Resim, yellow = Predictive Interpolation.

97

Thanks for clarification!

I have one extra question though with some specific case.

I have a cargo vehicle that consists of body + 4 wheels + 4 constraints, body and wheels are set to resimulation, and the vehicle itself works fine both on server and clients.

However as soon as I put a cargo on it (BP actor with static mesh with physics enabled, replicated, replicate movement and always relevant, resimulation/predictive interpolation (doesn’t really matter) I see that at some point it start sliding or jumps into the air and then being corrected to a normal position.

For cargo I also verified that physics body doesn’t sleep (set it in physics material). Tested that in case when vehicle is moving with stable max throttle without input decay with average network profile (on bad it’s obviously worse and almost unplayable)

Is there anything possible to check/to do with that without changing game mechanics? Since on server that physics calculates fine and stable, I would expect it to be more less fine for simple scenarios when only few bodies are involved.