Ah interesting, I calculate all the physics locally as part of the movement component (I directly set vel and angular vel at the end), so I can call everything in my component. I like your approach though too… which is probably more reliable with collision etc.
Sounds like you’re getting it nailed though!
I’m proud of you guys! Keep it up!
I expect synced, server-authoritative, client-side-predicting networked gameplay on my desk by tomorrow morning!
This is amazing, thank you, but I might be missing something, because I don’t see any source file in the engine that’s called FPhysicsSubstepTasks :S
EDIT: Never mind, found it, its called PhysSubstepTasks.h / .cpp
What do you mean by the ‘simulate’ functionality? I couldn’t find anything of that kind
Calculate locally? Can you describe this? You mean, you basically run your own physics implementation? Thats a nice approach, if I understood that correct and definitely would bypass the issues with the other physics-actors in the scene.
There are two options for me, either I run a second physx scene and mirror the original one over to the second one and recalculate physics there and transform the results over to the original one, or I just save the transforms and velocitys of all other actors in the game, mark them kinematic before the rewind, and then reapply the data. And thats what I am doing at the moment. It sounds pretty cheap and expensive in terms of processing power, but so far I tested it with 280 physical actors, including 3 destructible meshes and it worked without framedrop.
heh 
We have our deadline next friday, after that I’ll have some time to record some properly edited and documented footage.
Also, I need to structurize the code better. Currently, it’s a huge mess with tons of cryptical variable names. I tend to get messy when implementing complex stuff… Hate that.
After that, when I am really convinced that it is fully working, I can help you guys out a bit 
Yeah that’s pretty much it! I calculate what I want the velocity and angular velocity to be, and set those at the end of the function (I calculate all acceleration etc myself). Since I’m adding to the existing velocities calcualte by the physics engine, it still responds to collision etc. and I don’t have to create a new PhysX scene or anything fancy like that.
Really interested to see your implementation though, sounds like you went deeper than I did. I need to restart my system from scratch soon, it’s been so long since I worked on it I completely forgot where I was with it - and as it never worked 100% I may as well restart anyways…
Yes, I needed something more accurate, because our vehicles rely on a lot of physical forces, like friction, rolling resistance, drag, suspension compression. All that has to be calculated 1:1 again, so I literally just execute the same code again (the CustomPhysics-Substepping function) X times (Ping*2 / FixedTimestamp), but within one visual tick.
It’s not perfect yet, syncing the time is really hard and a problem yet. My corrective interpolation smooths that out a bit, but I still have plenty of stuff to do until I can really call it finished.
I think there are few downsides to this approach. Since the resimulation is done only on clients (unless I’m somehow mistaken) it means following things happen:
Clients exist on server at same time stamp but they start moving there with the delay caused by each ones individual ping value (they actually exist in different time zones but are forced into same time stamp on server). Basically this mean that on server, the player who has lowest ping, will start moving first on servers map and will also reach the finish line first on drag race (just an example). This could be solved on laptime etc measurements but it would still mean clients would see each others at wrong positions. If you don’t compensate the times separately, it would give unfair advantage to the player with lowest ping (head start). This of course isn’t that big issue if you don’t race against time or distance but just want to move rigidbodies around.
Since server doesn’t do resimulation and is authoritative and taken into account that it also runs at one fixed “time zone”, it means that not all inputs will be resimulated if clients ping changes during the gameplay. If ping lags behind, it means that clients inputs may arrive after server has done it’s own simulation (and remember, server is authoritative and doesn’t recalculate anything on this approach). This will result in inaccuracies and means that local simulation needs to be corrected all the time as simulations will not match anymore.
These things may also cause some issues with collisions and what players expect from collision responses.
If however the resimulation was done on server each time new input sequence arrived to the server, you could always have server that runs at same time on each client and you could get more accurate collisions. Also players with low ping wouldn’t appear to others like they got a head start.
For the syncing, only really stable way to sync the time stamps is to force ue4 to run it’s physics on fixed timesteps. This isn’t something that’s difficult to implement and Ori mentioned the place that needs to be modified on some past substepping forum-discussion (I can find the link if you need it). When you run physics simulation at fixed steps, you can always sync 100% the same physics steps on both server and client(s). The biggest downside with running physics at fixed step is that you don’t have physics and frame rendering in sync anymore which can show up as jittering movement (by default ue4 physics always run the last iteration right before tick so they are always in sync). That syncing issue can be solved by simulating additional physics step and interpolating each simulated rigidbody into correct position between those two physics steps (based on how big the time difference is between physics step and next tick). That interpolation will of course be more CPU heavy so it’s possible that you’d only do that for “hero” physics objects, clients rigidbody pawns etc.
Your first statement is absolutely true, but has nothing to do with the csp approach. That’s a completely different topic.
csp is to prevent input latency.
what you talked about is server side ping compensation
Also, my approach uses physics substepping, as I wrote above
I don’t really see how the server-resimulation could work out.
Let’s say, Client A has a ping of 200 milliseconds. Client B has a ping of 20 milliseconds.
At TIME 0, the game starts.
With pre-game syncing, both clients local games start the game at the exact same time and input movement immediately.
The input of Client B would be on the server after 20ms.
The input of Client A after 200ms.
At this moment the server has to recalculate the past 200 milliseconds for Client A’s movement. Okay, now the server has the input for TIME 0 at TIME 200 of Client B. But whops, whats about the input for the 200 ms it should recalculate now (from TIME 0 to TIME 200)? Not there… still on it’s way. That means, to be able to recalculate everything, the server would have to run constantly 200ms in the past, and sync Client B’s input by delaying it 180 milliseconds. Otherwise, they are not in sync and ping can influence the game mechanics, which is a no-go. Now when the server runs 200ms in the past, this means that it replicates for 200ms the locations of locally already past time, and needs additional 20 ms / 200ms to get the resimulated, corrected data to the clients, which will lead for input-latency: Input of the players will not be applied for Client B for 420 milliseconds, who only has 20ms ping.
Correct me if I got this whole concept wrong.
Substepping isn’t running at fully fixed intervals: it’s still synced to your existing framerate in the end. What it does is that it guarantees you’ll have more physics steps in case your framerate drops low (when without substepping you’d be in trouble). This doesn’t however give you fixed way to save same physics tick data on server and client because they may still run substepping at slightly different speeds. With fixed timestep, you’ll feed physx the exact same values on both client and server simulation and it should give you a lot more deterministic results. Ideally, you’d also apply and read your input on these fixed physics steps and also use same compressed input values even at local simulation in case you you compress floats on replication. This would guarantee that both simulations would get same input data.
Resimulating everything after receiving new input package from client would be total overkill. Also obviously you can’t just run the whole physics simulation in the past to make sure the slowest link is getting taken into account. Although depending on game type, this could actually be acceptable if you don’t need to do fast direction changes etc (for example something like boat/ship simulator could work just fine). I don’t have any magic bullets here, but in the nutshell, you’d probably want to only ever resimulate parts that really need to be simulated again for simulation to be accurate.
In this case, you could simulate every client’s rigidbody pawn separately on physics scene from it’s previous update to the current update and only if it happens to collide with some other clients rigidbody in past, then you’d resimulate them both to see the true collision between them. This can be pretty tricky to implement, but once done, it should give correct results and probably with least amount of CPU overhead (this would also help to reduce the workload on clients CPUs).
https://github.com//NetworkedPhysics
I have uploaded my Networked Physics project that I was working on previously. Bear in mind, I haven’t worked on this for about 9-10 months so I have no idea what kind of state it’s really in. I do know, that I’ve just updated it for 4.12 and built it, and everything seems to work about as well as it did before.
Feel free to look (and contribute to) the code, It would be nice to have this simple implementation working. It’s essentially an Unreal version of Glenn Fielder’s networked physics example that I had before it went private.
It’s not set up the best way (movement should be in a component for example), but I’m doing it the quickest way just to get it working first. I’ll worry about organisation later, for now most of the heavy lifting is just done in the Pawn.
Thanks, I will definitely take a look later this week. Also I will investigate 0lentos suggestions / ideas / approaches after our internal deadline this friday. A “bit” too busy until then, hehe
How can you simulate a physics tick on a Physics Scene? I’ve digged into the code and if I’m not wrong, at the start of the game tick, the game goes to the physics scene and calls StartFrame(), and when the end of the game tick comes (or the start of the next one?), the game gets the physic scene’s completion event, a graph event, with GetCompletionEvent(), and checks if its complete, if it is, it calls EndFrame(), and if not, it attaches a delegate to it that will be called when the physics tick ends, and will call EndFrame(), if I’ve understood that right.
I am probably missing the point completely, but what I’ve tried to do is do the same thing, call StartFrame(), and then get the completion event and attach a delegate to it, which will call EndFrame(), and do it a few times to resimulate my physics, but I just get a crash that says the completion event graph event does not exist yet. This is probably because I tried to get it right after starting the frame, and not later on.
I have no idea where to look from here, any help would be greatly appreciated
I and are using substepped physics so they have bit different execution logic on code. Regular physics run usually before each tick but if you use substepping, they run on a separate thread which gives you benefit for doing multiple physics steps automatically even if your framerate drops low.
For that extra simulation question, basically you’d need to change the substepping related code to be able to execute extra physics steps for resimulation etc. has implemented this so he may be able to give you more specific details on that.
For using the substepped custom physics delegate itself (not related to triggering extra physics simulation but rather how to run your own physics code on each substep instead of regular tick) you can take a look at this simplified example I posted on the forums a while ago: The Re-Inventing the Wheel Thread - Community & Industry Discussion - Epic Developer Community Forums . That custom physics delegate is based on this PR: https://github.com/EpicGames/UnrealEngine/pull/585 . It’s not documented on any official docs btw.
For changing the substepped physics to run at fixed timestep, you could check this forum post: Physics Sub-Stepping - Announcements - Epic Developer Community Forums
Someone also wrote an article about unreals physics options: http://www.aclockworkberry.com/unreal-engine-substepping/ although you need to note that it’s bit outdated (as well is that previous post I linked) when talking about “Fixed Frame Rate” setting on project settings. They actually changed the logic in that on 4.11 I think (or 4.10). Currently fixed frame rate setting seems to cap the frame rate into 60 so if your hw could render it faster, game will not run faster anymore. It probably still doesn’t work correctly if you can’t reach the 60 though.
This is cool, thank you very much.
I looked into the substep code, and it made me think, why is the maximum amount of substeps in the settings per game tick is 16? If I have the performance to spare, more substeps will be even better for a physics based game, no? Or its limited to 16 per tick because of some unexpected behavior?
I think the MaxSubsteps is for performance reasons, but can be setted in the Project Settings under “Physics” if I am not wrong. Honestly never played with that variable yet.
You have a PM, SnowCrash5
Well, it’s a failsafe mainly. If you for example run physics at 100 times in a second (so you’ll have max substepping step size at 0.01 s / 10ms), it’ll mean that even when you play your game at 10 fps (to interval is 100 ms for each frame) , you’ll still have only 10 physics substeps. It’s highly unlikely that you need anything more unless you run your physics at really fast (but then you’ll face other issues and it’s likely that your game will get heavily cpu bound). In normal use, you usually have 1-3 substeps between regular ticks.
Why I said it’s a fail safe is because if you don’t limit the max steps, in theory you can get to deadlock situation. For example if you try to run more physics substeps because your frame rate is really low but the reason for frames dropping low is because of your physics calculations, it then just keeps expanding the substep amount which is just making situation worse (and it’ll keep getting out of hand really soon after that).
Hey, quick update… The physx solution came pretty close to the satisfying result, but only close. It wasn’t useable in a release game at any means, and I got really tired of not knowing what to do and not having 100% controll over the physics simulation.
Long story short, I integrated Bullet Physics into UE4 and implemented my same technique for the new physics engine and it works like a charm so far! You have 100% controll about the time and scene very easily in Bullet and resetting and replaying N substeps can be performed within a single tick, and it’s “easy as eating cake”, at least compared to physx, where you have to dig in the engine code, “hack” into the substepping workers and have to wait for obscure physx callbacks and interprete undocumented networking code of UE. Not blaming UE for not documentating those classes, no one barely uses those.
So, advice for all of you: When you look forward to implement custom vehicle physics with competitive multiplayer, use Bullet physics. Got inspired by Rocket League, who also abandoned physx and implemented bullet.