Tutorial: Networked Physics - Pawn Tutorial

I found the setting in the Physic Settings Data Asset. Compare State To Trigger Rewind. So that one is solved. Last issue is the fact that in standalone net mode launching the ball clears ownership but in multiplayer the server never clears the ownership somehow after it launches the ball.

UPDATE 1

I think I found the root cause of the launch in multiplayer. It seems like Scheduled instant movement effects have an issue.

This might also be my design. Basically when the client’s input requests a possession I route that to an instant movement effect the movement effect changes the state. Works fine in single player. In multiplayer it seems the instant effect is never removed so its constantly claiming the ball and when its launched there is a frame where the state is correct then the next frame the player claims the ball through the instant effect

The easy solution is to not route through instant effect and instead let send the winner of the ball possession through the input since I do the query in ProduceInputData anyways (it runs on the game thread allowing me to gather all of the other player’s sync states and see if they’ve made a claim for the ball.

I’m not sure if this will open things up for exploits.

UPDATE 2
A direct effect of turning on compare state to trigger rewind is now the pawn is constantly in resim. Almost every frame is a resim. This is probably what is causing the most issues. I’m guessing code isn’t truly rollback safe. I have to do some more digging.**

UPDATE 3
After more digging, the correction are not coming from my own custom sync state structs. FMoverDefaultSyncState is always out of sync or at least outside of the threshold. Quick solution is to bypass the reconcile there since physics resimulation should already handle positional errors. I do wonder if that will come back to haunt me down the line since I do use the mover sync state quite a bit to determine physics. I wonder if its better to use the controlled particle transforns when doing calculations based on player transforms. I still feel like my code needs to be improved to handle rollback. That way corrections don’t break the game. Right now, I can’t pin point the exact “thing” that is unstable in rollback but I will definitely keep digging tomorrow.

UPDATE 4
Today I found something odd. I have a counter in the sync state that is just counting how many frames have passes and for some reason the client is 3 frames ahead on the counter. The counter is incremented in the fixed tick of the physics thread. This might have something to do with the client dropping frames in order to keep the frame offset but I’m just guess. If I want to count time in this way what is the correct way to do it? My thoughts were it was safe to use the fixed tick to track elapsed frames but any code I have that is dependent on it is now fragile because the frame counter is not the same on client and server.

UPDATE 5
Okay, I’m realizing that command frames are not accurate on the client and server. Which is what I thought was required for deterministic physics. For example I’m sending inputs from the client to the server in the fixed tick. When the client hits frame 345 something in my input triggers an action to happen. That same action doesn’t happen on the server until frame 362. How is this even possible? Looking at the source code the TimeStep is filled out here
void FAsyncCallback::GetCurrentMoverTimeStep(const FAsyncCallbackInput* AsyncInput, FMoverTimeStep& OutMoverTimeStep) const
{
OutMoverTimeStep.BaseSimTimeMs = GetSimTime_Internal() * 1000.0;
OutMoverTimeStep.StepMs = GetDeltaTime_Internal() * 1000.0f;
if (AsyncInput && AsyncInput->PhysicsSolver)
{
OutMoverTimeStep.ServerFrame = AsyncInput->PhysicsSolver->GetCurrentFrame() + AsyncInput->NetworkPhysicsTickOffset;
OutMoverTimeStep.bIsResimulating = AsyncInput->PhysicsSolver->GetEvolution()->IsResimming();
}
}

With the offset applied so my assumption was that the client frame in the time step is the corresponding server frame. Maybe I’m getting NPP & Networked Physics confused in how they work. Another assumption I had was that if the client was handling an input on frame X then the server would also handle that input on the same frame since the client is running ahead of the server and sending inputs from the future. We definitely need some guidance on this subject because all I am making are assumptions.

UPDATE 6
I’ve decided that until I understand why time is not consistent between client and server to bypass and state reconciliation other than the basketball ownership. Unfortunately, I’m not able to keep the simulation consistent when state comparison is on. When its off the game play’s smooth and the playback in CVD is identical almost all the time. I’m not sure why the state is always out of sync when comparing. I still think the time dilation plays a part in it but I can’t be sure and trying to step through the code while it’s multithreaded feels impossible. The debugger is jumping in between worlds and each one is in a different position in the code. I might try to force physics to be single threaded with the console commands but I’m not sure if that will defeat the purpose of async physics. **EDIT Seems line this console variable is set to true by default.

Hello @MBobbo
I’ve been thinking about an interesting topic, but unfortunately I do not know how to implement it. Perhaps you can tell me the right way. I would like my physical pawn, which uses reslimulation, to be able to move pseudorandom and deterministically without client input. That’s what I mean. You can imagine a helicopter hovering in the air. When the client is not pressing the input buttons, the helicopter may tilt slightly to the left or right under the influence of wind force. Or a ship on the waves. I tried using a deterministic pseudo-random function on the client and server and applying the force multiplied by this value each iteration, but I get out of sync. Perhaps you can show me the right way. Thanks

void FHelicopterPawnAsync::OnPreSimulate_Internal() {
  	Seed += GetDeltaTime_Internal();
	FRandomStream Stream(Seed);
	
	FVector Force = Stream.VRand();
	
	Force.Normalize();
	
	Force *= 20000;
    ParticleHandle->AddTorque(Force, true);
}

@apelsinovi_sok
Yeah, usually I would suggest using something like the command frame as the seed since client and server are supposed to agree on that value but sometimes its out of sync. However, that’s only when I try to use it like a time stamp cached in history. You might be able to get away with it here. It should be the server frame + offset as the seed and technically the randomized seed should be the same.

1 Like

Hello, As @Greggory_Addison mentioned, basing the seed on a value that is deterministically correct between cleint and server is the correct approach here. A small note though that you should do CurrentFrame + TickOffset (both client and server).

CurrentFrame is the local frame and LocalFrame + TickOffset becomes ServerFrame on both client and server. And that ServerFrame is the correct value to seed with.

You get CurrentFrame from RigidsSolver→GetCurrentFrame()

For TickOffset I don’t think there is a good clean way in 5.7 of getting it on the physics thread (there will be in 5.8 via namespace UE::NetworkPhysicsUtils) so I think you’ll need to marshal it from the game thread to physics thread via the AsyncInput you have for your FHelicopterPawnAsync. And on the game thread you get it from APlayerController.GetNetworkPhysicsTickOffset.

Hello, it was good that you found NetworkPhysicSettingsDataAsset::CompareStateToTriggerRewind
But it should not trigger resimulations constantly from the mover state, that’s would be a bug.

Since it seems you have mover specific issues so I’ll see if I can get one of the mover engineers to look at your comments here. I’m mainly working with the core network physics solutions, not the implementations like Chaos Mover so I don’t have the answers for what might be going wrong with the state and the timestamps.

But I don’t think it’s linked to the core solutions like time dilation and frame offset since those have been working bug free for a long time now.

One note is also that you can’t compare CurrentFrame numbers directly, you mention that an action happens on frame 345 on the client while at frame 362 on the server. If those frames are from PhysicsSolver->GetCurrentFrame() then that can be correct. In that case I’d expect NetworkPhysicsTickOffset to be 17 since 345 + 17 = 362. So that’s the number you should be able to compare CurrentFrame + TickOffset = ServerFrame on both client and server.

And if TickOffset is 17 in this case then client frame 345 correspond to server frame 362.

Hi @MBobbo,

I have been looking forward to a Networked Physics solutions and happily followed along your Pawn tutorial. Thanks for providing a great starting point with insightful explanation.

Given your starting point I wanted to challenge myself to recreate some of the movement mechanics from Rocket League.
Creating the general movement (throttle, steering, reverse, boosting) was easy enough from your tutorial, however jumping, sliding and flipping did not come to me as easily.

The jump would be a discrete event, as opposed to a constant force. I read your previous comments on this and how it would be wise to avoid discrete events, I do however feel like it it a very common thing in games to have a one off event, so it would feel quite limiting to not incorporate those. Could you guide me in the right direction when it comes to creating discrete events? If that is something that is going to receive more support soon (you hinted at it during your talk) would that be aimed at 5.8 or way further down the road?

Secondly, I was feeling uncertain of the best way to determine whether the pawn is grounded or not, as that determines the jumping behaviour. Should that be traced by the game thread and passed on as an input? Can the physics thread perform the trace or determine whether the pawn is grounded is another way?

Hello @50berry, discrete actions are ofcourse often needed in games but they are problematic when it comes to a forward predicted networking solution so if you can design around it that’s the “best” approach.

For a Rocket League like implementation the jump should be quite straight forward though you should not need any special logic around it. Apply the jump impulse based on a jump input and the autonomous proxy will apply that jump on the same frame as the server applies it on and they should stay in sync.

When it comes to simulated proxies they will jump when they receive the input to jump, which happens on the wrong frame and they will then desync which triggers a resim which goes back in time corrects them and the replay physics and applies the jump on the correct frame.

After the resim they will get corrected in the air via render interpolation which could look a bit awkward but they should generally just move upwards a bit faster since in vehicle games, like Rocket League sim-proxy vehicles are most often forward predicted almost 100% so you can reliably interact with them at high speed. So when sim-proxies jump they are most likely in the correct XY location but wrong Z (height) location essentially so the main correction happens along Z.

If you do a character instead of a vehicle then balancing is generally a bit different, you should not have them 100% forward predicted because characters tend to act much more instant than a vehicle which has acceleration and steering to adhere to. A character can turn on a dime and stop anytime from full speed without deceleration. If you have a character 100% forward predicted and they stop then they need a large XY correction, meaning they need to slide backwards to correct the stopping point. Which I showed in the talk at UnrealFest.

Currently the best way to approach how forward predicted something is, is via InputDecay, so your input decay should be much more aggressive for a character compared to for a vehicle. Depending on how your character acts, if it has proper momentum you might not need to be as aggressive.

When it comes to jumping with a character (which is maybe 50% forward predicted) there is a larger issue because input decay works on inputs, and generally when jumping your inputs doesn’t control your movement anymore or they control much less of you movement. Meaning input decay can’t keep the sim-proxy at 50% forward prediction and the resimulation that happens as a result of jumping at the incorrect time will simulate the character physics object along its trajectory without limiting the forward prediction and the character ends up in the air 100% forward predicted. And depending on your latency the sim-proxy might be halfway or further into the jump after that first resimulation. Making render interpolation move the character forward and slightly up and you don’t actually see the jump.

So instead of input decay I’ve made a new type of solution to control how much forward predicted an object should be, which i call “Simulation Decay” and it controls how much of the physics acceleration / velocity integration gets applied onto the physics particles position and rotation during physics integration (before the solver moves on with the evolution from there).

That means that you can apply 100% of the input but the physics object will not move from 100% of the accelerations / velocities applied to it. But the velocities are unaltered, so you don’t lose any gravity or applied forces etc. the object just moves a bit slower through the simulation.

This is hard to describe without proper visualization and description, my goal is to write another tutorial for 5.8 about a lot of the different fundamentals around networked physics and hopefully I can do a talk about it also this year.

We don’t use the simulation decay yet anywhere, I was hoping on using it for the Fall Guys: Crown Jam game we released earlier this year which was using networked physics resimulation, but instead when a sim-proxy jump in that game we switch the replication mode to Predictive Interpolation which is a bit of a hacky way of making the same thing happen while in air since that replication mode runs in the “interpolated timeline” so not fully forward predicted..

When it comes to finding if you are grounded, you could rely on game thread traces and marshal the bIsGrounded bool to physics thread but it’s recommended to do everything that can affect your characters state on the physics thread so there are ways of tracing on the physics thread, though they are limited and it’s easy to do things that are not thread-safe. But there is implementations of it in the Chaos Mover plugin for example. UE.ChaosMover.Utils.FloorSweep_Internal is a funtion that uses physics thread trace.
The trace call itself is here: Chaos::Private::FGenericPhysicsInterface_Internal::SpherecastMulti();

3 Likes

Hi Markus,

Thank you for the excellent tutorial. Learned a tonne going through it, and the results are really promising.

I’ve been trying to extend the implementation to introduce temporary modifiers that can affect my pawn, e.g. directional repellent forces, multipliers for linear/angular movement forces, jump force multiplier, gravity effectors, etc… The hope is that I might eventually use GAS to store these as attributes, and have gameplay effects modify their values, but for now I’m just simply placing static brush volumes around my level which declare their values (i.e. pawn enters volume → modifiers get applied, pawn exits volume → modifiers removed).

However, I’m struggling a bit with understanding how this kind of data is supposed to be passed in such that everything still works nicely with prediction & resimulation, since the tutorial mainly just focuses on inputs which are being produced by autonomous proxies. I’ve tried modifying the pawn’s tick function to just always include these extra inputs on both server & client, but as you mentioned in the tutorial this just ends up squashing the networked inputs via ApplyInput_Internal().

Do you have any recommendations/suggestions for how I might start implementing this? I could quite easily just include this data as part of my current inputs from the locally controlled pawn, but it feels wasteful to network all this extra data for a simple case like entering a volume (which has a predictable presence on both client & server already). Apologies if this is a simple question, but I feel like I’m missing something painfully obvious here and haven’t been able to solve it by myself. :sweat_smile:

Thanks!

Hi,

I am currently building a deterministic multiplayer gameplay/physics architecture in UE 5.7 based on the Networked Physics Pawn tutorial and Chaos fixed-step async simulation / resimulation.

My goal is strict determinism for multiplayer and as little pain as possible during resimulation. The Game Thread should only handle visuals, UI and input. Actual gameplay logic and physics should live in the async/physics simulation.

Right now I started with a per-object setup like this:
AVehicle → FVehiclePawnAsync

Later this would also become:
ATrap → FTrapAsync
AFireBall → FFireBallAsync

But the more I think about cross-entity interactions, the more I wonder if this is the wrong architectural level for a deterministic networked physics game.

My use case is:

  • Cars have internal async state such as speed, acceleration, health, buffs, debuffs, max speed modifiers, etc.
  • Other gameplay objects like fireballs, traps, gravity zones, etc. can affect those internal states.
  • Some effects are pure physics, such as impulses / forces.
  • Some effects are gameplay-state changes, such as damage, temporary max-speed buffs for 10 seconds, debuff arrays, etc.
  • All of this should remain deterministic and resim-friendly.

What I am considering now is a different architecture:

  • One central async simulation manager as the only actual Chaos sim callback object
  • GameThread actors such as AVehicle / ATrap / AFireBall register themselves there
  • The central async manager owns internal simulation entities / representations for cars, traps, fireballs, etc.
  • These are still encapsulated types with their own logic, state and physics handle, but they are no longer separate sim callback objects
  • The central manager controls update order, collision/effect resolution, and cross-entity application in one deterministic pipeline

So effectively:

  • The GameThread side stays object-based
  • The async side also stays structured into dedicated simulation types for organization
  • But orchestration is centralized inside one async manager

The reason I am considering this is that with many separate async callback objects, cross-entity communication, ordering, and internal state mutation seem much harder to reason about when the goal is absolute determinism, multiplayer prediction, and resimulation.

My question is:
For a Chaos-based networked physics game with strict determinism requirements, is it a better architecture to have a single central async simulation manager as the only actual sim callback object, while all cars, traps, projectiles, etc. exist only as internal simulation entities managed and updated by that manager?

I am especially interested in whether this is the more appropriate direction for:

  • deterministic update ordering
  • cross-entity gameplay interactions
  • internal gameplay state changes on async entities
  • multiplayer resimulation / rollback friendliness

If there is a better established pattern for this in Chaos / Networked Physics / resim workflows, I would appreciate pointers.

I had the issue with discrete jump in that approach before and I figured out that it’s quite easily replaceable with linear interpolation of force via time.

Let’s say you send JumpForce each tick as a regular float input, and calculate this JumpForce from something like: LookupTable1d(CurrentTime-TimeOfJumpStart, TimeBreakPoints, ForcePoints), where TimeBreakPoints is something like [0, 0.1, 0.3, 0.5] and ForcePoints is [0 1000 400 10] or similar pattern. By fine tuning the pattern you can achieve pretty much discrete feel of the jump without discrete impulse application.

When I switched to this approach in my game there were no more issues with desync on jumps.

2 Likes

Hello @lukecpowell, the easy approach is to do this client and server-side and store the multipliers inside the networked state, not the input.

For this to be properly working with resimulation you need to do things on the physics thread but gameplay logic is still quite hard to do there. In my post above I mention where to find the physics thread trace logic that you could use. You can also use other approaches like the contact modifier callback via ISimCallbackObject.

But for the simplest approach do the overlap trigger on the game thread and then pass the effect, like a movement input modifier, from game thread to physics thread via AsyncInput. AsyncInput is not networked itself, it’s just done locally on client (autonomous and sim-proxy) as well as server.
On the physics thread you can then update your multipliers stored on your pawn, this way you predict the modifier changing on clients. Then also add the modifiers into your networked state struct and populate them via BuildState_Internal and apply via ApplyState_Internal.
Only the server replicate states, so it will replicate the correct modifier values for the correct frames down to clients so when clients mispredict and desync from using the wrong modifiers then the correct modifiers will be applied during resimulation since we apply states from the server during resim.

You can have issues with this though since if you do the overlap on the game thread and you mispredicted that then during resim you will still get the same AsyncInput’s applied that were mispredicted on the game thread so that might stomp what you applied from the server.
That’s why anything that affects your physics simulation should really be done on the physics thread since it will also resimulate then and act on the correction.

When it comes to applying forces when entering a trigger, like walking into a fan that blows you upwards then that can’t be viably done on the game thread with good results. You’ll need to do a physics thread approach with a sweep or probe collision that you detect and then apply the force directly on the physics thread.

My plan is to write a few more tutorials that show some of these things that you need to actually make a game. But first out will be fundamentals around networked physics and features new to 5.8 and after that tutorial on how to approach gameplay.

2 Likes

hello @FerAram, setting you logic up with a manager that steps your pawns / objects is a valid approach. That’s how the Chaos Mover works for example. It also allows you to potentially parallelize the updates.

And Chaos does not have any notion of execution order within callbacks at the moment so there is no good way of making objects tick before pawns or to even make PawnA tick before PawnB on all clients and server. You can solve that via your own manager if you need it.

Note though that you don’t get full determinism from Chaos itself, especially not with networking and resimulation. For example AActor::ReplicatedMovement quantizes the transform data to between whole number and 2 decimal points depending on what setting you choose for it on your actor. So you don’t receive 100% correct position, rotation, linear velocity, angular velocity from the server which then breaks determinism. And if you send you own custom data and manage full floatingpoint determinism with the replication then when a resim is needed we clear friction points on the rewind event and recalculate them which is another source of non-determinism during resim. You’d need to network friction constraints from the server to fix that which is too much data to network viably.

Or you can force quantize all physics particles to the quantization that you network with each frame so each solve starts from a state that could just as well have been networked. And recalculate friction points each frame but I don’t know if that logic itself is deterministic.

The point is, don’t strive for 100% determinism becauase the only time you actually need it for a networking solution is if you use lockstep based on inputs.

If you rely on forward prediction via resimulation you just need “good enough” results so that one resim doesn’t trigger a new resim from desyncing during the first.

Note that whenever a sim-proxy changes their inputs (which normally happens continuously for most games) they will desync and require a resim and that resim can’t be 100% predicted so determinism doens’t matter at that point either.

When it comes to handling interactions between your objects you should track collisions / do sweeps on the physics thread and you could have a manager that maps what each physics particle is, so you can get a hit for an FPhysicsObject and then get the FFireBallAsync for that from your manager for example. And then run the logic you need.

There is no good way of getting components / ISimCallbackObjects based on an owning physics particle in Chaos.

1 Like

Thanks, I appreciate it. The reason I was pointing at the core solutions was because in the source code the FMoverTimeStep is filled via a function called GetCurrentMoverTimeStep() This is located in the ChaosMoverAsyncCallback.cpp file. In that function they apply the offset on this line here OutMoverTimeStep.ServerFrame = AsyncInput->PhysicsSolver->GetCurrentFrame() + AsyncInput->NetworkPhysicsTickOffset; So, when I was getting rollback for mismatch frames, I thought it was another issue. I have rollbacks turned off for now except for really important things that can’t be corrected via movement, but I would like to turn them back on for more security. I’m looking forward to hearing from anyone with some information. I’m hoping I can catch up with some of you guys at Unreal Fest.

@MBobbo

I wanted to bring up another question I had pertaining to the input decay method. If I’m predicting inputs with decay wouldn’t that in itself cause the simulation to be non-deterministic creating a per frame resimulation. I’m noticing in my tests when I have a test pawn in the level where the server is randomly selecting from 2 input vectors (Forward & Backward simulating random client input) I get rollback pretty much every frame. This test is done with my controlled pawn standing still not changing the state/input whatsoever and another version of my pawn placed in the world.

If I look at the CVD and check on the inputs during a resim I can see them decaying. My thoughts are the moment that we run physics with an input that is decayed that frame would essentially be out of sync causing a rollback to be applied which starts the cycle all over again.

Following your example from the talk you gave. Another player’s auto proxy is moving forward from frame 0-12 The server follows that and sends that down to the sim proxy on our machine. We get that update and if it differs from the server we resim N amount frames to get back on our predicted timeline. With input decay, N+0 is the correct input from the server, N+1 has some decay and so on and so forth. This means when we get the server update for frame N+1 its automatically going to be wrong meaning we have to correct.

Shouldn’t we only apply decay if the newly found input’s dot against our last known is <= some sensitivity value. This way we don’t end up getting out of sync just because the client corrected.

Basically, what I’m seeing is the first time you correct and apply decay you are essentially opening up pandoras box and now every N frames when you get the server snap shot you are forced to rollback because the previously applied decay altered your predicted inputs forcing you to always be out of sync with the server.

UPDATE #1
After some tests, turning off input decay does reduce the amount of corrections that happen. I still get the overshoot but that’s a small price to pay. I have a solution to make that a little easier which is separating the visual layer from the simulation layer. Essentially, I will set the visual’s transforms to absolute and manually handle its interpolation so that during resims we slowly interpolate the mesh towards the correct position. This will make the visual representation slightly inaccurate but you would never see it overshoot.

Hello @Greggory_Addison, a couple of things:

We don’t see constant resimulations when enabling NetworkPhysicSettingsDataAsset::CompareStateToTriggerRewind on the Chaos Manny in the MoverExamples. There could be occational ones though.
You mentioned you bypassed FMoverDefaultSyncState::ShouldReconcile but have you looked to see what is the culprit in causing a reconcile to be needed there for you?

You mentioned you were doing something like Frames++ OnPreSimulate_Internal or something and storing that in the SyncState. And you noticed a diff between frames stepped on client and server. I’m not reproed it but you can’t rely on client and server stepping the same amount of frames in all cases no. If the server or client has a CPU hitch it might drop physics frames as to not cause further CPU performance issues like going into a spiral.
When this happens it usually don’t cause a desync and resim because we have a buffer built into the TickOffset, instead the server will tell the client to speed up or slow down to align the TickOffset again. And eventually when it’s aligned the frames ticked should be the same.
But it can also be that the time dilation doens’t work due to bad performance or just not quick enough and then the TickOffset will be recalculated (worst case scenario) which causes a resim and frames to be lost.

Your mention “When the client hits frame 345 something in my input triggers an action to happen. That same action doesn’t happen on the server until frame 362“.
If those frames are both "ServerFrame” then that should not happen no except when things desync a lot.
Is this a reproducible issue for you? Is it happening constantly or randomly or when you do something specific? For example switching window focus from game to the editor might deprioritize the game which might then drop frames on client but not server which can cause issues like this to appear. Also when putting breakpoints for example.

“Maybe I’m getting NPP & Networked Physics confused in how they work”
I think you seem to grasp networked physics pretty well.
But don’t mix NPP and Networked Physics, they are not meant to work together and share no functionality. Mover uses NPP and Chaos Mover uses Networked Physics, so inside Mover things might seem to overlap a bit between them.

About input decay, yes it will cause non-determinism and require resimulation again for sim-proxies (not autonomous proxies since they don’t decay input).
Your idea of only applying decay if the input changes a certain amount (if I read that correct) does not solve what input decay is generally meant to solve. The idea is that the sim-proxy is forward predicted somewhere between the “extremes” of having the input pressed or not pressed. A vehicle for example turning left and then turning right, if we wait for the input to turn right the vehicle will have forward predicted wrongly towards left as much as possible until the input is changed to right, and at that point it doesn’t matter if we decay or not the vehicle has still predicted too far left and needs a larger correction towards the right than if we would have decayed the steering input.

Your approach with essentially running rendering behind in time is an approach that does work, but at the cost of decoupling physics and rendering timelines further than they already are which means it can be hard to aim at an object you want to collide with or perform logic based on overlaps or sweeps etc.

I have something similar on my todo list with the layer between physics and game states but it’s supposed to solve another issue but could potentially be used fr something similar to what you did also.

But any solution can be a good solution for the correct project / game though, there is no way of making a network physics solution that handles all cases with one set of solutions and default settings for example, there needs to be a palette of solutions and settings that you get to develop with. And we are still developing the palette :slight_smile:

One of my goals are to write a forward predicted replication (for both pawns and actors) that doesn’t need constant resimulations and that handles both full and partial forward prediction (which input and simulation decay is essentially doing). It’s in the works but will not be out for 5.8, it will be in a later release.

2 Likes

@MBobbo

You mentioned you bypassed FMoverDefaultSyncState::ShouldReconcile but have you looked to see what is the culprit in causing a reconcile to be needed there for you?

Yes, for me the culprit was the distance between the location of the server and client being greater than the static value there (5.f)

I’m not reproed it but you can’t rely on client and server stepping the same amount of frames in all cases no. If the server or client has a CPU hitch it might drop physics frames as to not cause further CPU performance issues like going into a spiral.

This makes sense! Thanks for clearing that up. I have a lot of code that is highly dependent on the number of frames between actions. From now on I will just snapshot the server frame when the action starts and then when another action begins get the delta between the current server frame and the snapshotted frame to calculate how much time has passed.

Your approach with essentially running rendering behind in time is an approach that does work, but at the cost of decoupling physics and rendering timelines further than they already are which means it can be hard to aim at an object you want to collide with or perform logic based on overlaps or sweeps etc.

Understood. From my CVD captures the visual mesh doesn’t stray too far from the root component so it will do for now until I find a better approach. I’m looking forward to the updates. I’ve been checking up on UE5main almost every day to see where things are headed. I ran my first dedicated server test using Chaos Mover today and there are a ton of kinks to work out on my end but it’s super exciting to see things somewhat working properly between multiple clients over the network. Once I’m able to reduce the number of corrections and hide the ones that do occur my game will be extremely polished thanks to you guys! :heart:

Thanks for taking the time to reply!

Hi @MBobbo, I’m building a physics-based multiplayer party game (think chaotic soap/physics objects knocking each other around) using the NetworkPhysicsComponent resimulation system and I have some architectural questions before committing to this design.

Planned Architecture:

Each physics pawn has two collision layers. The inner layer is the actual mesh collider which handles real environment collisions like walls and static meshes via normal Chaos physics. The outer layer is a larger trigger box which handles pawn-to-pawn interactions only. When two pawns’ trigger boxes overlap, instead of relying on default PhysX contact resolution, I calculate and apply velocity changes inside OnPreSimulate_Internal based on the interacting objects’ current velocity, mass, and type. This gives me full control over game feel tuning similar to how Rocket League tuned their impulse responses. Mesh colliders between pawns are set to ignore each other to avoid double-resolution conflicts.

Questions:

First, overlap events are detected on the Game Thread but OnPreSimulate runs on the Physics Thread. Is there a safe and resimulation-compatible way to pass overlap data into OnPreSimulate, or should I be doing the overlap query directly inside OnPreSimulate using a physics scene query?

Second, when a pawn bounces off a static mesh via real Chaos collision and then immediately enters another pawn’s trigger zone within the same frame, is there a guaranteed execution order between the contact resolution and my OnPreSimulate callback? I’m concerned about reading stale velocity data in the chain reaction scenario.

Third, does applying velocity changes to other pawns inside my own OnPreSimulate_Internal properly get recorded into those pawns’ rewind history? Or will the resimulation system on those other pawns be unaware of the externally applied velocity, causing desyncs on correction?

Fourth, we’re considering raising the physics tick rate from 30hz to 60hz specifically to reduce the window between game thread overlap detection and physics thread processing. Is this a reasonable trade-off or is there a better architectural solution to this timing gap?

Thanks in advance.

1 Like

Hello @Getmoonlight, one of the main things to think about with physics resimulation is that anything affecting physics needst to be both decided and applied on the physics thread. The game thread can’t be used to decide if you should add a force for example since during a resimulation only the physics thread will tick, not the game thread so if your game thread made the wrong decision it will not be corrected during a resim.

Something to know also then is that things done to physics on the game thread will be reapplied during resim. So if you decide that you want to add a force on the game thread then that force will be added again during resim but the decision to add the force can’t be corrected and that’s the main point of doing a resimulation to make new decisions for each of the resim frames.

Question 1
You have three options here instead of doing game thread overlaps.

  1. Physics Thread sweep trace, which you can do inside OnPreSimulate_Internal to find overlaps. The API is here: Chaos::Private::FGenericPhysicsInterface_Internal::SpherecastMulti() and you can look in Chaos Mover (plugin) which uses this for the ground trace in UE.ChaosMover.Utils.FloorSweep_Internal.
  2. You can rely on collisions instead and modify the collision when it happens via OnContactModification_Internal which you can override same as you do for OnPreSimulate_Internal. Note that you also need to pass in Chaos::ESimCallbackOptions::ContactModification to your Chaos::TSimCallbackObject like we do for Presimulate and PhysicsObjectUnregister in the tutorial. The callback will trigger with a collection of all the contacts that happen in physics this step, so not specific to your own physics object, you need to look through the collection to see if the physics object of your pawn is there and then modify the contact of that.
  3. There is way of doing “probe collisions” also where you get the above contact modification callback but for a larger overlap collision box which doesn’t have any collisions set, so you just notice an overlap and you can do whatever you want during that overlap on the physics thread.

Question 2
There is no deterministic callorder within a certain callbacks, but all OnPreSimulate will happen before OnContactModification_Internal is called. If you need determinism in callorder for your pawns then you need to make a manager that has an ISimCallbackObject and that manager is the one that calls into your pawns to make them update and you can then make that manager deterministic between clients and server.

But with above solutions for Physics Thread contact modifications I don’t think this is much of an issue?

Question 3
The resimulation system caches particle state (position, rotation, linear velocity, angular velocity, etc.) at the very beginning of each physics tick, before game thread data is applied and before
OnPreSimulate_Internal runs. This is by design, the cached state is used to rewind to a specified frame and then we run all logic for that frame again and make new decisions there based on changes that we might have made from server data after the Rewind.

The actions you do during a physics tick is not being cached in rewind history, only the starting state of physics for that tick is cached. So that you can resimulate and choose not to apply a force where you previously did apply a force (which might have been what caused a desync).

Question 4
The valid reason to raise tick rate is physics simulation quality: smaller time steps mean more accurate collision detection (less tunneling), smoother contact resolution, and better behavior
for fast-moving objects. For a chaotic physics party game with lots of high-speed collisions, going from 33ms to 16ms steps could noticeably improve collision reliability. But at the cost of CPU (twice as many frames to resimulate), Memory (twice as many frames to cache in history) and network bandwidth (twice as much data to send over the network). But it all depends on what your game can handle and how you optimize it. Some games can handle 120hz others can only do 20hz.

But don’t up the tick-rate to make the game thread overlap happen more often, it’s not a viable approach to drive physics interactions.

1 Like

Hi @MBobbo*, thanks for the incredibly detailed response — it clarified a lot about the resimulation architecture and fundamentally changed how I think about the physics thread vs game thread boundary.*

I have two follow-up topics, one about scalability and one about discoverability of physics thread APIs.

Scalability — Listen Server at 60Hz with 8+ players

My game is a Chaos physics-based party game targeting 8 players minimum on a Listen Server (no dedicated server). Every player is a physics pawn using NetworkPhysicsComponent with resimulation, and there will also be passive physics objects in the level (push carts, seesaws, puzzle cubes) that interact with the players.

I’m considering raising the physics tick rate from 30Hz to 60Hz for better collision quality (less tunneling on thin surfaces and fast impacts), but I’m concerned about the cumulative cost on a single Listen Server host machine that has to maintain physics cache history and process resimulations for all 8 players plus level objects simultaneously.

Is there a practical ceiling where the resimulation cache memory and CPU cost becomes prohibitive for a Listen Server scenario at 60Hz with this many physics actors? Or is this manageable with reasonable optimization (keeping cache history short, limiting resimulation depth, etc.)? Any guidance on what levers to pull would be greatly appreciated.

Physics Thread API Discoverability

I’m currently studying the Chaos Mover plugin source (specifically FloorSweep_Internal) as you recommended, to understand how to do physics thread queries. However, I’m finding it difficult to discover the full set of available physics thread APIs since the internal Chaos interfaces (like FGenericPhysicsInterface_Internal::SpherecastMulti, OnContactModification_Internal callback patterns, probe collision setup, etc.) don’t appear in Epic’s public API documentation.

Is there a reference or internal document that lists the available physics thread query and modification APIs that are safe to use inside ISimCallbackObject callbacks? Or is the recommended approach to read through the Chaos Mover and Chaos Vehicle plugin source code as living documentation?

Beyond FloorSweep_Internal, are there other places in the engine source (plugins, samples, or test code) that demonstrate patterns for a networked physics game — things like physics thread overlap detection, contact modification for custom impulse responses, probe collision setup, or physics thread force application between multiple interacting pawns?

Any pointers would save a lot of engine source spelunking. Thanks!

About: Scalability — Listen Server at 60Hz with 8+ players

  • If your passive physics objects don’t need any special state replicated then don’t put a NetworkPhysicsComponent on them, the component has overhead from just being there and is not needed for dynamic physics objects that you interact with.

  • If tunneling is your concern you could try using CCD (continuous collision detection) before raising the tick-rate. You could also try MACD (movement away collision detection) if you get tunneling between moving objects.

  • A very common misconception is that the server is performing resimulations, but it’s not. Resimulations is a way to both correct a replication error and to get objects into forward prediction. The server is the source of truth it never needs to correct itself because it never does anything “wrong”, only clients get things wrong with replication so they need to correct themselves (via resimulation for example). And the client is the one that is forwrad predicted ahead of the server so the client is the one that needs resimulation predict replicated physics objects ahead of the server.

    • 60hz effects on the server: Outgoing network traffic will increase a lot, the server is sending and processing ALOT more network data than clients do.

    • 60hz effects on the client: Rewind history grows (memory consumpton), resimulation frames increase (CPU consumtion).

      • Memory: Lower the MaxSupportedLatencyPrediction in Project Settings → Physics → Replication. It defaults to 1000ms (so at 60hz that would be 60 frames of history cached) if you lower it to 500ms then it would only want to cache 30 frames of history but you would then also not support clients that have over 500ms latency (which is very bad latency anyway). One thing to know here is that the actual frames cached is ceiled to the closest power of two (2,4,8,16,32,64,128) so 1000 at 60hz will cache 64 frames, 600ms at 60hz will also cache 64 frames, you need to get below 32 for it to cache 32frames if you want to lower memory use. Note that the size of the rewind history doesn’t really affect CPU much unless for clients that actually do have really bad networking and require to go back further in time to perform resimulations.

      • CPU: There isn’t that much to do in 5.7 to lower CPU usage except that isn’t enabled by default, you could disable this CVar: p.Resim.AllowRewindToResimulatedFrames. You can also raise the resim thresholds in the projects settings to not trigger resimulations as often. You could also try enabling np2.Resim.RuntimeCorrectionEnabled with and without np2.Resim.RuntimeVelocityCorrection, which is a way to try keep a resim object in sync without resimulating and if it fails it triggers a resim. But it can cause replication issues depending on what is being replicated, a vehicle with bouncu suspension can go into a feedback loop with that enabled for example. In 5.8 you will get more options to clamp down on resimulation consuming too much CPU, for example p.Resim.CooldownFrames N which makes resim not able to trigger for N frames after a resim was performed. And p.Resim.GuardFromPerformanceCollapse true which forces the same amount of real time to pass between each resimulation as the last resim took.

About: Physics Thread API Discoverability

  • Unfortunately the documentation for these APIs are limited to non-existing indeed, part of the reason is that some of the API is still experimental and we might change it.

  • There are no tutorials or sample projects that showcase / teach async physics or Networked Physics outside of this tutorial (to my knowledge).

    • I want to make more tutorials and will do when I get some more time, i.e. after UE 5.8 is locked down.
  • One issue that exist is also that on the physics thread you should not do anything with UObjects, which means you can’t safely check if the thing you hit is an APawn or AActor to perform different logic based on what you hit easily.

    • There is a feature in the works to attach UserData to physics particles but I don’t know if it’s available already or if it will be in 5.8 or maybe later. I’ll try to get a tutorial made when it’s available, or get the one who wrote the solution to do a tutorial.

    • But before I’d say that within the same scope of you doing a sweep on the physics thread you can assume it’s fairly safe to read and cast the UObjects you get back, never write to them though!

      • And if you need to add a force to an object you hit, don’t do it via the AActor or BodyInstance. Grab the physics particle and apply you physics effects on that one when on the physics thread, there is also API for some things in the physics evolution, like FPBDRigidsEvolutionGBF::SetParticleTransform.
    • So currently unfortunately you will most likely need to explore what you can do in OnContactModification_Internal, you’ll need to cache the particle that the current ISimCallbackObject is for, i.e. your pawn for example and then look for that particle inside the collection of contact pairs you get as the parameter property.

1 Like