Hello, I take it you found a solution to your problems?
I read the posts the other day but didn’t have time to respond yet.
As a note, there is a gap between what Inputs and States solve, so I’m working on filling that gap which would probably make the ball possession, for example, a bit easier to address.
I’m aiming at getting that solution for 5.8 at the moment.
Hey, thanks for responding.
Yes I found a work around which was to queue an instant movement effect that would let the physics thread know when the player was requesting possession. I also had to redesign the entire system since the global/singleton basketball state was on the interpolated time even with setting NPP to forward predicted. I’m running into other issues having to do with multiplayer and I’m trying to track those down right now. From what I’m seeing in the CVD the state is not changing properly on the server even though the server is running the code. For example, when I launch the ball the state is supposed to clear my ownership. Instead the ball is launched and the resulting state is not cleared even though the code is written to where one is the result of the other
if (Event.EventTag == BasketballNativeTags::Event_Fire)
{
if (TryToLaunchTheBasketball())
{
FSyncState_BasketballStatus& BasketballStatus = OutputState.SyncState.SyncStateCollection.FindOrAddMutableDataByType<FSyncState_BasketballStatus>();
BasketballStatus.bPlayerHasBall = false;
UE_LOG(LogBasketball, Warning, TEXT("Player Has Successfully Launched The Ball"))
}
}
But in CVD when I watch the ball get launched the server has this boolean set to true. I’m not sure how that’s possible but I’m doing some debugging to see if I’ve made any mistakes
Another huge issue is Resimulation is not triggered by the state mismatching. For example, I have a bug where sometimes the client doesn’t properly predict picking up or launching the ball (I’m not sure how but it happens more often than I would like) but the server does. The state differs but I don’t get a resim until I mispredict something movement wise. What I mean by than is players can execute dribble moves or shoot the ball when they have it. These actions trigger big movement changes (launching, root motion, etc) Only after starting one of these moves do I get a correction and align locally with the authoritative state.
There might be some settings I missed or some code paths that are called wrong. I’m still trying to figure out the right way to do things.
Overall, what you and your team has done is really great and has literally saved my game.
I found the setting in the Physic Settings Data Asset. Compare State To Trigger Rewind. So that one is solved. Last issue is the fact that in standalone net mode launching the ball clears ownership but in multiplayer the server never clears the ownership somehow after it launches the ball.
UPDATE 1
I think I found the root cause of the launch in multiplayer. It seems like Scheduled instant movement effects have an issue.
This might also be my design. Basically when the client’s input requests a possession I route that to an instant movement effect the movement effect changes the state. Works fine in single player. In multiplayer it seems the instant effect is never removed so its constantly claiming the ball and when its launched there is a frame where the state is correct then the next frame the player claims the ball through the instant effect
The easy solution is to not route through instant effect and instead let send the winner of the ball possession through the input since I do the query in ProduceInputData anyways (it runs on the game thread allowing me to gather all of the other player’s sync states and see if they’ve made a claim for the ball.
I’m not sure if this will open things up for exploits.
UPDATE 2
A direct effect of turning on compare state to trigger rewind is now the pawn is constantly in resim. Almost every frame is a resim. This is probably what is causing the most issues. I’m guessing code isn’t truly rollback safe. I have to do some more digging.**
UPDATE 3
After more digging, the correction are not coming from my own custom sync state structs. FMoverDefaultSyncState is always out of sync or at least outside of the threshold. Quick solution is to bypass the reconcile there since physics resimulation should already handle positional errors. I do wonder if that will come back to haunt me down the line since I do use the mover sync state quite a bit to determine physics. I wonder if its better to use the controlled particle transforns when doing calculations based on player transforms. I still feel like my code needs to be improved to handle rollback. That way corrections don’t break the game. Right now, I can’t pin point the exact “thing” that is unstable in rollback but I will definitely keep digging tomorrow.
UPDATE 4
Today I found something odd. I have a counter in the sync state that is just counting how many frames have passes and for some reason the client is 3 frames ahead on the counter. The counter is incremented in the fixed tick of the physics thread. This might have something to do with the client dropping frames in order to keep the frame offset but I’m just guess. If I want to count time in this way what is the correct way to do it? My thoughts were it was safe to use the fixed tick to track elapsed frames but any code I have that is dependent on it is now fragile because the frame counter is not the same on client and server.
UPDATE 5
Okay, I’m realizing that command frames are not accurate on the client and server. Which is what I thought was required for deterministic physics. For example I’m sending inputs from the client to the server in the fixed tick. When the client hits frame 345 something in my input triggers an action to happen. That same action doesn’t happen on the server until frame 362. How is this even possible? Looking at the source code the TimeStep is filled out here
void FAsyncCallback::GetCurrentMoverTimeStep(const FAsyncCallbackInput* AsyncInput, FMoverTimeStep& OutMoverTimeStep) const
{
OutMoverTimeStep.BaseSimTimeMs = GetSimTime_Internal() * 1000.0;
OutMoverTimeStep.StepMs = GetDeltaTime_Internal() * 1000.0f;
if (AsyncInput && AsyncInput->PhysicsSolver)
{
OutMoverTimeStep.ServerFrame = AsyncInput->PhysicsSolver->GetCurrentFrame() + AsyncInput->NetworkPhysicsTickOffset;
OutMoverTimeStep.bIsResimulating = AsyncInput->PhysicsSolver->GetEvolution()->IsResimming();
}
}
With the offset applied so my assumption was that the client frame in the time step is the corresponding server frame. Maybe I’m getting NPP & Networked Physics confused in how they work. Another assumption I had was that if the client was handling an input on frame X then the server would also handle that input on the same frame since the client is running ahead of the server and sending inputs from the future. We definitely need some guidance on this subject because all I am making are assumptions.
UPDATE 6
I’ve decided that until I understand why time is not consistent between client and server to bypass and state reconciliation other than the basketball ownership. Unfortunately, I’m not able to keep the simulation consistent when state comparison is on. When its off the game play’s smooth and the playback in CVD is identical almost all the time. I’m not sure why the state is always out of sync when comparing. I still think the time dilation plays a part in it but I can’t be sure and trying to step through the code while it’s multithreaded feels impossible. The debugger is jumping in between worlds and each one is in a different position in the code. I might try to force physics to be single threaded with the console commands but I’m not sure if that will defeat the purpose of async physics. **EDIT Seems line this console variable is set to true by default.
Hello @MBobbo
I’ve been thinking about an interesting topic, but unfortunately I do not know how to implement it. Perhaps you can tell me the right way. I would like my physical pawn, which uses reslimulation, to be able to move pseudorandom and deterministically without client input. That’s what I mean. You can imagine a helicopter hovering in the air. When the client is not pressing the input buttons, the helicopter may tilt slightly to the left or right under the influence of wind force. Or a ship on the waves. I tried using a deterministic pseudo-random function on the client and server and applying the force multiplied by this value each iteration, but I get out of sync. Perhaps you can show me the right way. Thanks
void FHelicopterPawnAsync::OnPreSimulate_Internal() {
Seed += GetDeltaTime_Internal();
FRandomStream Stream(Seed);
FVector Force = Stream.VRand();
Force.Normalize();
Force *= 20000;
ParticleHandle->AddTorque(Force, true);
}
@apelsinovi_sok
Yeah, usually I would suggest using something like the command frame as the seed since client and server are supposed to agree on that value but sometimes its out of sync. However, that’s only when I try to use it like a time stamp cached in history. You might be able to get away with it here. It should be the server frame + offset as the seed and technically the randomized seed should be the same.
Hello, As @Greggory_Addison mentioned, basing the seed on a value that is deterministically correct between cleint and server is the correct approach here. A small note though that you should do CurrentFrame + TickOffset (both client and server).
CurrentFrame is the local frame and LocalFrame + TickOffset becomes ServerFrame on both client and server. And that ServerFrame is the correct value to seed with.
You get CurrentFrame from RigidsSolver→GetCurrentFrame()
For TickOffset I don’t think there is a good clean way in 5.7 of getting it on the physics thread (there will be in 5.8 via namespace UE::NetworkPhysicsUtils) so I think you’ll need to marshal it from the game thread to physics thread via the AsyncInput you have for your FHelicopterPawnAsync. And on the game thread you get it from APlayerController.GetNetworkPhysicsTickOffset.
Hello, it was good that you found NetworkPhysicSettingsDataAsset::CompareStateToTriggerRewind
But it should not trigger resimulations constantly from the mover state, that’s would be a bug.
Since it seems you have mover specific issues so I’ll see if I can get one of the mover engineers to look at your comments here. I’m mainly working with the core network physics solutions, not the implementations like Chaos Mover so I don’t have the answers for what might be going wrong with the state and the timestamps.
But I don’t think it’s linked to the core solutions like time dilation and frame offset since those have been working bug free for a long time now.
One note is also that you can’t compare CurrentFrame numbers directly, you mention that an action happens on frame 345 on the client while at frame 362 on the server. If those frames are from PhysicsSolver->GetCurrentFrame() then that can be correct. In that case I’d expect NetworkPhysicsTickOffset to be 17 since 345 + 17 = 362. So that’s the number you should be able to compare CurrentFrame + TickOffset = ServerFrame on both client and server.
And if TickOffset is 17 in this case then client frame 345 correspond to server frame 362.
Hi @MBobbo,
I have been looking forward to a Networked Physics solutions and happily followed along your Pawn tutorial. Thanks for providing a great starting point with insightful explanation.
Given your starting point I wanted to challenge myself to recreate some of the movement mechanics from Rocket League.
Creating the general movement (throttle, steering, reverse, boosting) was easy enough from your tutorial, however jumping, sliding and flipping did not come to me as easily.
The jump would be a discrete event, as opposed to a constant force. I read your previous comments on this and how it would be wise to avoid discrete events, I do however feel like it it a very common thing in games to have a one off event, so it would feel quite limiting to not incorporate those. Could you guide me in the right direction when it comes to creating discrete events? If that is something that is going to receive more support soon (you hinted at it during your talk) would that be aimed at 5.8 or way further down the road?
Secondly, I was feeling uncertain of the best way to determine whether the pawn is grounded or not, as that determines the jumping behaviour. Should that be traced by the game thread and passed on as an input? Can the physics thread perform the trace or determine whether the pawn is grounded is another way?
Hello @50berry, discrete actions are ofcourse often needed in games but they are problematic when it comes to a forward predicted networking solution so if you can design around it that’s the “best” approach.
For a Rocket League like implementation the jump should be quite straight forward though you should not need any special logic around it. Apply the jump impulse based on a jump input and the autonomous proxy will apply that jump on the same frame as the server applies it on and they should stay in sync.
When it comes to simulated proxies they will jump when they receive the input to jump, which happens on the wrong frame and they will then desync which triggers a resim which goes back in time corrects them and the replay physics and applies the jump on the correct frame.
After the resim they will get corrected in the air via render interpolation which could look a bit awkward but they should generally just move upwards a bit faster since in vehicle games, like Rocket League sim-proxy vehicles are most often forward predicted almost 100% so you can reliably interact with them at high speed. So when sim-proxies jump they are most likely in the correct XY location but wrong Z (height) location essentially so the main correction happens along Z.
If you do a character instead of a vehicle then balancing is generally a bit different, you should not have them 100% forward predicted because characters tend to act much more instant than a vehicle which has acceleration and steering to adhere to. A character can turn on a dime and stop anytime from full speed without deceleration. If you have a character 100% forward predicted and they stop then they need a large XY correction, meaning they need to slide backwards to correct the stopping point. Which I showed in the talk at UnrealFest.
Currently the best way to approach how forward predicted something is, is via InputDecay, so your input decay should be much more aggressive for a character compared to for a vehicle. Depending on how your character acts, if it has proper momentum you might not need to be as aggressive.
When it comes to jumping with a character (which is maybe 50% forward predicted) there is a larger issue because input decay works on inputs, and generally when jumping your inputs doesn’t control your movement anymore or they control much less of you movement. Meaning input decay can’t keep the sim-proxy at 50% forward prediction and the resimulation that happens as a result of jumping at the incorrect time will simulate the character physics object along its trajectory without limiting the forward prediction and the character ends up in the air 100% forward predicted. And depending on your latency the sim-proxy might be halfway or further into the jump after that first resimulation. Making render interpolation move the character forward and slightly up and you don’t actually see the jump.
So instead of input decay I’ve made a new type of solution to control how much forward predicted an object should be, which i call “Simulation Decay” and it controls how much of the physics acceleration / velocity integration gets applied onto the physics particles position and rotation during physics integration (before the solver moves on with the evolution from there).
That means that you can apply 100% of the input but the physics object will not move from 100% of the accelerations / velocities applied to it. But the velocities are unaltered, so you don’t lose any gravity or applied forces etc. the object just moves a bit slower through the simulation.
This is hard to describe without proper visualization and description, my goal is to write another tutorial for 5.8 about a lot of the different fundamentals around networked physics and hopefully I can do a talk about it also this year.
We don’t use the simulation decay yet anywhere, I was hoping on using it for the Fall Guys: Crown Jam game we released earlier this year which was using networked physics resimulation, but instead when a sim-proxy jump in that game we switch the replication mode to Predictive Interpolation which is a bit of a hacky way of making the same thing happen while in air since that replication mode runs in the “interpolated timeline” so not fully forward predicted..
When it comes to finding if you are grounded, you could rely on game thread traces and marshal the bIsGrounded bool to physics thread but it’s recommended to do everything that can affect your characters state on the physics thread so there are ways of tracing on the physics thread, though they are limited and it’s easy to do things that are not thread-safe. But there is implementations of it in the Chaos Mover plugin for example. UE.ChaosMover.Utils.FloorSweep_Internal is a funtion that uses physics thread trace.
The trace call itself is here: Chaos::Private::FGenericPhysicsInterface_Internal::SpherecastMulti();
Hi Markus,
Thank you for the excellent tutorial. Learned a tonne going through it, and the results are really promising.
I’ve been trying to extend the implementation to introduce temporary modifiers that can affect my pawn, e.g. directional repellent forces, multipliers for linear/angular movement forces, jump force multiplier, gravity effectors, etc… The hope is that I might eventually use GAS to store these as attributes, and have gameplay effects modify their values, but for now I’m just simply placing static brush volumes around my level which declare their values (i.e. pawn enters volume → modifiers get applied, pawn exits volume → modifiers removed).
However, I’m struggling a bit with understanding how this kind of data is supposed to be passed in such that everything still works nicely with prediction & resimulation, since the tutorial mainly just focuses on inputs which are being produced by autonomous proxies. I’ve tried modifying the pawn’s tick function to just always include these extra inputs on both server & client, but as you mentioned in the tutorial this just ends up squashing the networked inputs via ApplyInput_Internal().
Do you have any recommendations/suggestions for how I might start implementing this? I could quite easily just include this data as part of my current inputs from the locally controlled pawn, but it feels wasteful to network all this extra data for a simple case like entering a volume (which has a predictable presence on both client & server already). Apologies if this is a simple question, but I feel like I’m missing something painfully obvious here and haven’t been able to solve it by myself. 
Thanks!
Hi,
I am currently building a deterministic multiplayer gameplay/physics architecture in UE 5.7 based on the Networked Physics Pawn tutorial and Chaos fixed-step async simulation / resimulation.
My goal is strict determinism for multiplayer and as little pain as possible during resimulation. The Game Thread should only handle visuals, UI and input. Actual gameplay logic and physics should live in the async/physics simulation.
Right now I started with a per-object setup like this:
AVehicle → FVehiclePawnAsync
Later this would also become:
ATrap → FTrapAsync
AFireBall → FFireBallAsync
But the more I think about cross-entity interactions, the more I wonder if this is the wrong architectural level for a deterministic networked physics game.
My use case is:
What I am considering now is a different architecture:
So effectively:
The reason I am considering this is that with many separate async callback objects, cross-entity communication, ordering, and internal state mutation seem much harder to reason about when the goal is absolute determinism, multiplayer prediction, and resimulation.
My question is:
For a Chaos-based networked physics game with strict determinism requirements, is it a better architecture to have a single central async simulation manager as the only actual sim callback object, while all cars, traps, projectiles, etc. exist only as internal simulation entities managed and updated by that manager?
I am especially interested in whether this is the more appropriate direction for:
If there is a better established pattern for this in Chaos / Networked Physics / resim workflows, I would appreciate pointers.
I had the issue with discrete jump in that approach before and I figured out that it’s quite easily replaceable with linear interpolation of force via time.
Let’s say you send JumpForce each tick as a regular float input, and calculate this JumpForce from something like: LookupTable1d(CurrentTime-TimeOfJumpStart, TimeBreakPoints, ForcePoints), where TimeBreakPoints is something like [0, 0.1, 0.3, 0.5] and ForcePoints is [0 1000 400 10] or similar pattern. By fine tuning the pattern you can achieve pretty much discrete feel of the jump without discrete impulse application.
When I switched to this approach in my game there were no more issues with desync on jumps.
Hello @lukecpowell, the easy approach is to do this client and server-side and store the multipliers inside the networked state, not the input.
For this to be properly working with resimulation you need to do things on the physics thread but gameplay logic is still quite hard to do there. In my post above I mention where to find the physics thread trace logic that you could use. You can also use other approaches like the contact modifier callback via ISimCallbackObject.
But for the simplest approach do the overlap trigger on the game thread and then pass the effect, like a movement input modifier, from game thread to physics thread via AsyncInput. AsyncInput is not networked itself, it’s just done locally on client (autonomous and sim-proxy) as well as server.
On the physics thread you can then update your multipliers stored on your pawn, this way you predict the modifier changing on clients. Then also add the modifiers into your networked state struct and populate them via BuildState_Internal and apply via ApplyState_Internal.
Only the server replicate states, so it will replicate the correct modifier values for the correct frames down to clients so when clients mispredict and desync from using the wrong modifiers then the correct modifiers will be applied during resimulation since we apply states from the server during resim.
You can have issues with this though since if you do the overlap on the game thread and you mispredicted that then during resim you will still get the same AsyncInput’s applied that were mispredicted on the game thread so that might stomp what you applied from the server.
That’s why anything that affects your physics simulation should really be done on the physics thread since it will also resimulate then and act on the correction.
When it comes to applying forces when entering a trigger, like walking into a fan that blows you upwards then that can’t be viably done on the game thread with good results. You’ll need to do a physics thread approach with a sweep or probe collision that you detect and then apply the force directly on the physics thread.
My plan is to write a few more tutorials that show some of these things that you need to actually make a game. But first out will be fundamentals around networked physics and features new to 5.8 and after that tutorial on how to approach gameplay.
hello @FerAram, setting you logic up with a manager that steps your pawns / objects is a valid approach. That’s how the Chaos Mover works for example. It also allows you to potentially parallelize the updates.
And Chaos does not have any notion of execution order within callbacks at the moment so there is no good way of making objects tick before pawns or to even make PawnA tick before PawnB on all clients and server. You can solve that via your own manager if you need it.
Note though that you don’t get full determinism from Chaos itself, especially not with networking and resimulation. For example AActor::ReplicatedMovement quantizes the transform data to between whole number and 2 decimal points depending on what setting you choose for it on your actor. So you don’t receive 100% correct position, rotation, linear velocity, angular velocity from the server which then breaks determinism. And if you send you own custom data and manage full floatingpoint determinism with the replication then when a resim is needed we clear friction points on the rewind event and recalculate them which is another source of non-determinism during resim. You’d need to network friction constraints from the server to fix that which is too much data to network viably.
Or you can force quantize all physics particles to the quantization that you network with each frame so each solve starts from a state that could just as well have been networked. And recalculate friction points each frame but I don’t know if that logic itself is deterministic.
The point is, don’t strive for 100% determinism becauase the only time you actually need it for a networking solution is if you use lockstep based on inputs.
If you rely on forward prediction via resimulation you just need “good enough” results so that one resim doesn’t trigger a new resim from desyncing during the first.
Note that whenever a sim-proxy changes their inputs (which normally happens continuously for most games) they will desync and require a resim and that resim can’t be 100% predicted so determinism doens’t matter at that point either.
When it comes to handling interactions between your objects you should track collisions / do sweeps on the physics thread and you could have a manager that maps what each physics particle is, so you can get a hit for an FPhysicsObject and then get the FFireBallAsync for that from your manager for example. And then run the logic you need.
There is no good way of getting components / ISimCallbackObjects based on an owning physics particle in Chaos.
Thanks, I appreciate it. The reason I was pointing at the core solutions was because in the source code the FMoverTimeStep is filled via a function called GetCurrentMoverTimeStep() This is located in the ChaosMoverAsyncCallback.cpp file. In that function they apply the offset on this line here OutMoverTimeStep.ServerFrame = AsyncInput->PhysicsSolver->GetCurrentFrame() + AsyncInput->NetworkPhysicsTickOffset; So, when I was getting rollback for mismatch frames, I thought it was another issue. I have rollbacks turned off for now except for really important things that can’t be corrected via movement, but I would like to turn them back on for more security. I’m looking forward to hearing from anyone with some information. I’m hoping I can catch up with some of you guys at Unreal Fest.
I wanted to bring up another question I had pertaining to the input decay method. If I’m predicting inputs with decay wouldn’t that in itself cause the simulation to be non-deterministic creating a per frame resimulation. I’m noticing in my tests when I have a test pawn in the level where the server is randomly selecting from 2 input vectors (Forward & Backward simulating random client input) I get rollback pretty much every frame. This test is done with my controlled pawn standing still not changing the state/input whatsoever and another version of my pawn placed in the world.
If I look at the CVD and check on the inputs during a resim I can see them decaying. My thoughts are the moment that we run physics with an input that is decayed that frame would essentially be out of sync causing a rollback to be applied which starts the cycle all over again.
Following your example from the talk you gave. Another player’s auto proxy is moving forward from frame 0-12 The server follows that and sends that down to the sim proxy on our machine. We get that update and if it differs from the server we resim N amount frames to get back on our predicted timeline. With input decay, N+0 is the correct input from the server, N+1 has some decay and so on and so forth. This means when we get the server update for frame N+1 its automatically going to be wrong meaning we have to correct.
Shouldn’t we only apply decay if the newly found input’s dot against our last known is <= some sensitivity value. This way we don’t end up getting out of sync just because the client corrected.
Basically, what I’m seeing is the first time you correct and apply decay you are essentially opening up pandoras box and now every N frames when you get the server snap shot you are forced to rollback because the previously applied decay altered your predicted inputs forcing you to always be out of sync with the server.
UPDATE #1
After some tests, turning off input decay does reduce the amount of corrections that happen. I still get the overshoot but that’s a small price to pay. I have a solution to make that a little easier which is separating the visual layer from the simulation layer. Essentially, I will set the visual’s transforms to absolute and manually handle its interpolation so that during resims we slowly interpolate the mesh towards the correct position. This will make the visual representation slightly inaccurate but you would never see it overshoot.
Hello @Greggory_Addison, a couple of things:
We don’t see constant resimulations when enabling NetworkPhysicSettingsDataAsset::CompareStateToTriggerRewind on the Chaos Manny in the MoverExamples. There could be occational ones though.
You mentioned you bypassed FMoverDefaultSyncState::ShouldReconcile but have you looked to see what is the culprit in causing a reconcile to be needed there for you?
You mentioned you were doing something like Frames++ OnPreSimulate_Internal or something and storing that in the SyncState. And you noticed a diff between frames stepped on client and server. I’m not reproed it but you can’t rely on client and server stepping the same amount of frames in all cases no. If the server or client has a CPU hitch it might drop physics frames as to not cause further CPU performance issues like going into a spiral.
When this happens it usually don’t cause a desync and resim because we have a buffer built into the TickOffset, instead the server will tell the client to speed up or slow down to align the TickOffset again. And eventually when it’s aligned the frames ticked should be the same.
But it can also be that the time dilation doens’t work due to bad performance or just not quick enough and then the TickOffset will be recalculated (worst case scenario) which causes a resim and frames to be lost.
Your mention “When the client hits frame 345 something in my input triggers an action to happen. That same action doesn’t happen on the server until frame 362“.
If those frames are both "ServerFrame” then that should not happen no except when things desync a lot.
Is this a reproducible issue for you? Is it happening constantly or randomly or when you do something specific? For example switching window focus from game to the editor might deprioritize the game which might then drop frames on client but not server which can cause issues like this to appear. Also when putting breakpoints for example.
“Maybe I’m getting NPP & Networked Physics confused in how they work”
I think you seem to grasp networked physics pretty well.
But don’t mix NPP and Networked Physics, they are not meant to work together and share no functionality. Mover uses NPP and Chaos Mover uses Networked Physics, so inside Mover things might seem to overlap a bit between them.
About input decay, yes it will cause non-determinism and require resimulation again for sim-proxies (not autonomous proxies since they don’t decay input).
Your idea of only applying decay if the input changes a certain amount (if I read that correct) does not solve what input decay is generally meant to solve. The idea is that the sim-proxy is forward predicted somewhere between the “extremes” of having the input pressed or not pressed. A vehicle for example turning left and then turning right, if we wait for the input to turn right the vehicle will have forward predicted wrongly towards left as much as possible until the input is changed to right, and at that point it doesn’t matter if we decay or not the vehicle has still predicted too far left and needs a larger correction towards the right than if we would have decayed the steering input.
Your approach with essentially running rendering behind in time is an approach that does work, but at the cost of decoupling physics and rendering timelines further than they already are which means it can be hard to aim at an object you want to collide with or perform logic based on overlaps or sweeps etc.
I have something similar on my todo list with the layer between physics and game states but it’s supposed to solve another issue but could potentially be used fr something similar to what you did also.
But any solution can be a good solution for the correct project / game though, there is no way of making a network physics solution that handles all cases with one set of solutions and default settings for example, there needs to be a palette of solutions and settings that you get to develop with. And we are still developing the palette 
One of my goals are to write a forward predicted replication (for both pawns and actors) that doesn’t need constant resimulations and that handles both full and partial forward prediction (which input and simulation decay is essentially doing). It’s in the works but will not be out for 5.8, it will be in a later release.
You mentioned you bypassed FMoverDefaultSyncState::ShouldReconcile but have you looked to see what is the culprit in causing a reconcile to be needed there for you?
Yes, for me the culprit was the distance between the location of the server and client being greater than the static value there (5.f)
I’m not reproed it but you can’t rely on client and server stepping the same amount of frames in all cases no. If the server or client has a CPU hitch it might drop physics frames as to not cause further CPU performance issues like going into a spiral.
This makes sense! Thanks for clearing that up. I have a lot of code that is highly dependent on the number of frames between actions. From now on I will just snapshot the server frame when the action starts and then when another action begins get the delta between the current server frame and the snapshotted frame to calculate how much time has passed.
Your approach with essentially running rendering behind in time is an approach that does work, but at the cost of decoupling physics and rendering timelines further than they already are which means it can be hard to aim at an object you want to collide with or perform logic based on overlaps or sweeps etc.
Understood. From my CVD captures the visual mesh doesn’t stray too far from the root component so it will do for now until I find a better approach. I’m looking forward to the updates. I’ve been checking up on UE5main almost every day to see where things are headed. I ran my first dedicated server test using Chaos Mover today and there are a ton of kinks to work out on my end but it’s super exciting to see things somewhat working properly between multiple clients over the network. Once I’m able to reduce the number of corrections and hide the ones that do occur my game will be extremely polished thanks to you guys! 
Thanks for taking the time to reply!
Hi @MBobbo, I’m building a physics-based multiplayer party game (think chaotic soap/physics objects knocking each other around) using the NetworkPhysicsComponent resimulation system and I have some architectural questions before committing to this design.
Planned Architecture:
Each physics pawn has two collision layers. The inner layer is the actual mesh collider which handles real environment collisions like walls and static meshes via normal Chaos physics. The outer layer is a larger trigger box which handles pawn-to-pawn interactions only. When two pawns’ trigger boxes overlap, instead of relying on default PhysX contact resolution, I calculate and apply velocity changes inside OnPreSimulate_Internal based on the interacting objects’ current velocity, mass, and type. This gives me full control over game feel tuning similar to how Rocket League tuned their impulse responses. Mesh colliders between pawns are set to ignore each other to avoid double-resolution conflicts.
Questions:
First, overlap events are detected on the Game Thread but OnPreSimulate runs on the Physics Thread. Is there a safe and resimulation-compatible way to pass overlap data into OnPreSimulate, or should I be doing the overlap query directly inside OnPreSimulate using a physics scene query?
Second, when a pawn bounces off a static mesh via real Chaos collision and then immediately enters another pawn’s trigger zone within the same frame, is there a guaranteed execution order between the contact resolution and my OnPreSimulate callback? I’m concerned about reading stale velocity data in the chain reaction scenario.
Third, does applying velocity changes to other pawns inside my own OnPreSimulate_Internal properly get recorded into those pawns’ rewind history? Or will the resimulation system on those other pawns be unaware of the externally applied velocity, causing desyncs on correction?
Fourth, we’re considering raising the physics tick rate from 30hz to 60hz specifically to reduce the window between game thread overlap detection and physics thread processing. Is this a reasonable trade-off or is there a better architectural solution to this timing gap?
Thanks in advance.
Hello @Getmoonlight, one of the main things to think about with physics resimulation is that anything affecting physics needst to be both decided and applied on the physics thread. The game thread can’t be used to decide if you should add a force for example since during a resimulation only the physics thread will tick, not the game thread so if your game thread made the wrong decision it will not be corrected during a resim.
Something to know also then is that things done to physics on the game thread will be reapplied during resim. So if you decide that you want to add a force on the game thread then that force will be added again during resim but the decision to add the force can’t be corrected and that’s the main point of doing a resimulation to make new decisions for each of the resim frames.
Question 1
You have three options here instead of doing game thread overlaps.
Question 2
There is no deterministic callorder within a certain callbacks, but all OnPreSimulate will happen before OnContactModification_Internal is called. If you need determinism in callorder for your pawns then you need to make a manager that has an ISimCallbackObject and that manager is the one that calls into your pawns to make them update and you can then make that manager deterministic between clients and server.
But with above solutions for Physics Thread contact modifications I don’t think this is much of an issue?
Question 3
The resimulation system caches particle state (position, rotation, linear velocity, angular velocity, etc.) at the very beginning of each physics tick, before game thread data is applied and before
OnPreSimulate_Internal runs. This is by design, the cached state is used to rewind to a specified frame and then we run all logic for that frame again and make new decisions there based on changes that we might have made from server data after the Rewind.
The actions you do during a physics tick is not being cached in rewind history, only the starting state of physics for that tick is cached. So that you can resimulate and choose not to apply a force where you previously did apply a force (which might have been what caused a desync).
Question 4
The valid reason to raise tick rate is physics simulation quality: smaller time steps mean more accurate collision detection (less tunneling), smoother contact resolution, and better behavior
for fast-moving objects. For a chaotic physics party game with lots of high-speed collisions, going from 33ms to 16ms steps could noticeably improve collision reliability. But at the cost of CPU (twice as many frames to resimulate), Memory (twice as many frames to cache in history) and network bandwidth (twice as much data to send over the network). But it all depends on what your game can handle and how you optimize it. Some games can handle 120hz others can only do 20hz.
But don’t up the tick-rate to make the game thread overlap happen more often, it’s not a viable approach to drive physics interactions.