Physics (Bullet) Client-side prediction

Hey guys,
for the last weeks I have been working on Client-Side-Prediction for our vehicles, and tried many techniques, but somehow, nothing really satiesfied us.
We were looking for a solution that fulfills the following requirements:

-Server Authoritve | Client does not determine his own position autonomously
-No Input Latency | As a fast-paced racing multiplayer game, input latency is no option. When the player presses a key, he should see the results immeditaley
-Synchronousity (Don’t think I spelled that right, I apologize for my english) Although the Client has to predict his own future for the roundtriptime, all objects should be corrected to the same moment in time as fast as possible, to avoid confusion in aiming and driving mechanics, when ping changes occur

As I mentioned, I tried dozens of techniques, but then realized, we need Rewind&Replay for Physics.
And so I implemented it. Well, I currently am still implementing it, it required me to dig deep into the engine code, but the results look good so far, but far from being perfect yet (some errors in my implementation in the Replay, but the generall system is working).

I was wondering if anyone else of you ever implemented something similar and if you want to share your experience. As soon as I have a result good enough to show, I will of course try to share my 'knowledge", if there is any interest. :slight_smile:

**Edit:
**
I switched to bullet physics, to read why and how the results are, and when you want a “short” explanation (“tutorial”) on what I am doing for my client-side-prediction, check out page two.

Just a quick video I did showing my current progress. Note: The whole code is not optimized and properly synched yet, I just wanted to show that the generall physics fast-forward (rewind) is working :slight_smile:

@Rama has done one but hasn’t shown any source code. I’ve dabbled a bit but syncing timestamps was a pig…

Impressed with the stuff in the dev log btw. If people do get this figured out, I’d like to see some source. I’ve currently given up on syncing PhysX and decided to take Character Movement Components’ approach. Not a fan of it though :confused:

Thanks. I did find some stuf of Rama, but his latest attempt for physics based platformer seemed to be client authoritive, without any physics rewind, if I am not wrong.
I am still working on it, it still has major flaws where I need to determine where the inaccurarcys come from, but as soon as I have something acceptable working I will share my progress here :slight_smile:

I think his very latest version is server authoritative, but I can’t be sure. Like you I need server-authoritative physics, since it’s a competitive game and cheating would end up being all too easy.

This is what I was doing to try and test my solution. It was so close, but I could never get the timestamps to sync - so the client would try to predict it’s moves but the server would send a new one, and it would overwrite the wrong moves.

What I have found however, is that the best way to hide the transition is to apply the positional update to the collision first, THEN interpolate the relative transform of the visual mesh to that position. That way collision is always handled in the most up-to-date form which results in less resyncing / error - and minimal rubber-banding for the player.

Thanks for the tip!! But what do you think exactly causes problems in your attempt? Not sure if I understood it right.

To me it looked like you were applying forces on the Server to the object. In that case, thats not client-side-prediction. That only works for input that happened by the client, and is in fact just the prediction for the time it takes the players input to be send, processed and corrected back to the client, by the server. When a force is applied on the server side only, the client of course can’t know when this force will get applied.

And whhats your problem with syncing?

Read the description :slight_smile:

Nope, the simulation ran both Client and Server side, and the client sent Input to the Server only.

The problem I had was that the Timestamps of the moves wouldn’t stay in sync, so as the client moved forward, they’d receive a new / old update from the server with the wrong timestamp and it would therefore replace the wrong move in the client history, so it would then play subsequent moves incorrectly as well making the snapping even worse.

The above example was 50% packet loss, 500 ms latency I think… been a long time since i work on that project though (mostly out of frustration :wink:

I can’t see how they would not stay in sync…
My technique is:

Save local time on client, then immediatley send request to the server, who then sends his local time back to the client immedtialtely.
When the time arrives on the client, the client again gets his local time and checks the time it took this response to arrive. Half of that time is the Ping.
You substract the ping from the server time to get the offset between the timers. Then you can convert your clients local time by adding this offset + ping (roundtriptime / 2).

So, when a server correction arrives, you check locally which history snapshot has the closest timestamp, on the synced time.

That’s exactly what I did, but over time they drift (especially with PktLagVariance). Only by very tiny amounts, but it was enough. Synchronizing time between two machines isn’t a trivial task unfortunately, can’t even rely on the windows clock.

I used the same system that Unreal Tournament did to effectively bounce a packet off of the server. Have to admit it’s been ages since I’ve looked at it though.

Well, the key is to update the ping frequently. Another technique is to send all the input tick-based to the server with a timestamp of the client, and the server send its data back, together with the latest received timestamp of what the client send, so then the client can sync 100% all the time.

A good solution might be a mix of the two, relying on normal sync most of the time but every few seconds, do the timestamp sync to get it back into perfect sync.

Well if you’ve managed to do it, I’d love to see it :stuck_out_tongue: I will have to go back to working on it sooner or later…

Actually, I just made up the second technique while writing and have to test it myself yet, but at least I think that could work, but I will try it today or tomorrow and give you some feedback.

Can I ask how you did implement the actual physics recalculation?

Okay, for the last two days I was trying to figure out why my Rewind&Replay doesnt work properly, then I noticed… it does. I print all the last positions of 1.) Local calculated position 2.) Server Send positions and 3.) Fast Forwarded Positions (the positions between the substeps also drawn in the next tick), so that I get three lines of movement basically. And I expected, that my Red Line (The FastForward Calculations) differ from the Black Line (Server Positions), but they didnt. They almost exactly match all the time, meaning, that the FastForwarding Calculations calculate the same as the server does, but in advance. Neat!

No so neat however is, that the problem was the “blue line”. The Local calculations. And those differ EXTREMLY from what the server calculates, with EXACT the same code and same input. Driving a right-curve on the client totally differs on the client and the server, the client takes a shorter turn, or longer turn, sometimes the server spinns out, but the client doesnt… This gets nicely visible when I only run the FastForwarding every 3-5 seconds.

Client receives server correction, and calculates where it should be now. This is correct position now, after PING Milliseconds, the Black Dots (Server) align perfectly with the red dots (Fast Forwarded).
Now, the client moves right (right curve), and I would expect them to be SOMEWHAT similar. But often they are heavily different, even on perfectly plane underground.

BlackLines = ServerPos
BlueLiens = LocalPos

Turned of Rewind to demonstrate it better.

My question now is pretty vague, but what can cause this? Is Physx really THAT undeterministic, that even on plain underground with same conditions, a vehicle that only relys on input data and the underground (as far as dependencies go, that are not calculated by fixed mathematical rules, can behave SO differently? If so, I have to say, physx multipler is IMPOSSIBLE. It’s simple impossible, if that is the case. There is no way, to interpolate that correctly, I dont even want to think about driving on unplanar underground then.

I hope someone can tell me that their physics run somewhat similar on server and client…

I mean, I thought maybe, I mess something up with the input and how it is send to the server, but no…
It’s the basic system:

-Player presses key
-Client sends reliable RPC to server
-Both then execute the same function, delayed by PING milliseconds

Nevermind, I made some good progress :smiley:

I have run into a simliar situation. I want my charactor driven by physics, but stucked on how to predict the physical respond on client side. , the video seems work fine, could u share some idea how you archieve it?

Also want to know this, how do you recalculate physics?

Set the position / velocity etc. to what it was at the start of that move, then resimulate for all subsequent moves now that they’ve changed. You just call the ‘simulate’ functionality multiple times for the store delta time for that move, until you catch up to the correct timestamp again. You can also invalidate older moves at that point.

Yes, you have to dig into the FPhysicsSubstepTasks.h / .cpp to do that.

Create an input-history of your local client, save there Velocity / Transform and Input-Data of the player every substep with the local timestep.
You fill out the same structure on the server, let that replicate back to the client, who then calculates based on ping, when this package should have been applied if there would have been no ping. You then initially apply the the Server-Data to your pawn, rewind in the input-history to that time and replay Ping milliseconds via the PhysicsSubstepper by adding additional DeltaTime. Also, replicate input state at this point, if need be.
For example, vehicles don’t accelerate to 100% throttle instantly, but the throttle increases while the W key is being hold. To make the replay as precise as possible, these values also have to be replayed. In fact, you need to recalculate everything your physics relay on.

One major letdown is that this technique also fast-forwards other physics actors. You have to cheat a bit to get around that, but it’s now all working for me. 300ms ping with no lag and no input latency :slight_smile: