Running GetWorld()->Tick() multiple times per frame to facilitate rollbacks for netplay

Development Programming & Scripting C++
1

I’m trying to implement GGPO netcode in my game.

The way it works is, each peer runs a full copy of the game. Every frame, every peer sends its input state for that frame to every other peer, then advances the current frame locally with the local player’s inputs, without waiting for the remote inputs, using input prediction in the meantime (essentially repeating the old inputs; player 2 will keep walking forward, etc.). When those remote inputs are finally received, GGPO will load the last valid local gamestate (the last frame we had remote inputs from), replace the current local gamestate with that valid one, then use both the previous local inputs and the newly received remote inputs to play back those frames again, in order, so the game state can be caught up to where it’s “supposed” to be.

I have several problems to solve here: namely, using UE4 deterministically, saving the input state, and implementing rollback support, but I’m tackling the last one first, since it’s probably one of the most difficult problems to solve.

I’ve spent the last few days doing a good amount of research around here, and I found that everyone else who has tried to do something similar has ultimately given up and gone with an alternative, less-robust (IMO) solution using some amount of replication.

UWorld::Tick() is called once per frame, every frame. Does anyone have any idea how it and other functions need to be tweaked in order to facilitate sometimes calling that function multiple times in a single frame? Ultimately, I’d have to call it from a custom GameMode class’s Tick function whenever a rollback occurs, but without causing an infinite loop of Tick() calling itself. Right now, I’m iteratively fixing issues as they come up, thanks to the handy assertions all over the engine’s source code.

Edit: Over the years, I’ve gotten a number of PMs seeking help for this problem. It might not be obvious, but I never actually solved it. I got caught in the weeds trying to edit the engine to facilitate determinism, which was definitely not the right way to go about getting it to work. Here are some tips.

Try asking someone who did it. The current last post in this thread is by ganoncl, who actually ended up getting an Unreal Developers Network subscription (costs $XX,XXX, but there is a 90-day trial option if you ask for it) for his team. He got Epic to help them figure out how to get rollback netcode in their UE4 fighting game, and it shipped a year or three ago.

Failing that, you can also try the UDN subscription/trial route to get Epic to help you.

Just try to avoid using Epic’s gameplay code, if you can. That was my main mistake. If you roll your own movement system, etc., and just keep it simple, then you can have an easier time creating a serializable game state.

1 Like
2

Dude you are talking about prediction and correction .ue4’s network does exactly that since start. Hell even ue3 did the same thing .

When ue4 is waiting for something to change it uses the existing values until it receives new one from the server and as soon as it does it corrects the game state as it should be.
"
(essentially repeating the old inputs; player 2 will keep walking forward, etc.)."

Yes that’s what ue4 does every time to give the game a fluent feel. Look at simulated proxies. Sure you’ll need to handle changes manually for things in repnotify but rest is the same.

So before adding in ggpo I recommend you to try ue4’s stock netcode and see the results yourself

3

Using UE4’s built in netcode is not an option for my game.

From the Networking Overview page: “UE4 networking is built around the server/client model.”

My game requires P2P networking for minimal latency. P2P support in UE4 has been marked “wishlist” on the UE4 roadmap for 15 months. GGPO is a proven library, so that’s what I’m gonna use.

4

Couldn’t you just do a ping test and let the lowest latency player be the server? You could even allow the host to drop and one of the peers take over, pretty much exactly like what most console games do. Then you only need to host a lobby somewhere.

5

I appreciate the advice on that alternative, but like I said before, that isn’t an option.

6

Are you able to discuss more details around what you’re trying to do?

In that article you linked, it talks about how Street Fighter could make use of GGPO but it also seems like it has a built in mechanism to invalidate latency problems below a certain ping time.

7

P2P isn’t possible in UE4, and it won’t be, probably for a very long time - if ever. just because it’s on the roadmap doesn’t mean Epic is either working on or even considering it, it’s there for someone to look at later. I have no idea what GGPO stands for, but UE4’s CharacterMovementComponent already does Prediction and Reconciliation so you can use that for reference, and Unreal Tournament wrote their own slightly more optimized system as well. However if your using UE4 characters, you don’t need to do anything at all - it’s already done for you.

Unreals Server/Client model is a much more proven system than GGPO.

You may also want to know that UE4 isn’t deterministic internally, not even slightly, so even if you use the same input and somehow manage to sync up the timestamps, the results will drift over extended periods of time. You absolutely have to do server-authoritative corrections in Unreal, or your connections will go out of sync.

Replaying old moves is easy to understand but very hard to actually program in. I’m doing my own version for physics-based movement based on Glenn Fielders work (wish that dude hung around here!). You don’t tick the actor multiple times (that will cause you all kindzzz of problems), you simply apply the input multiple times in one frame for each out-of-date move, and update the results accordingly. You store a circular buffer of all moves with input, and when you receive a move from the server, you find out which one corresponds to it locally, delete any older moves, then play them back over a single frame using the stored delta time for that move and essentially ‘correct’ the move buffer. You move the collision of the object first, and offset the mesh so that it slowly blends back to the collision and doesn’t ‘snap’ on the clients view. This is roughly how CMC works.

I suggest taking a look at this work by Glenn Fielder, since the premises are basically the same: http://gafferongames.com/networked-physics/introduction-to-networked-physics/
And, I’d also suggest looking at how CMC works and checking out UE4’s documentation: Character Movement Component | Unreal Engine Documentation

Bottom line is, there’s a lot to be said for using the tools the way they are designed to be used. There’s no reason that a fast-paced game can’t used a Server-Authoritative model - which is much safer than P2P and far easier to program. The solidity of the latest Unreal Tournament is a testament to how well these models can work, even in high-latency situations. You don’t want to make your game optimized for a low-latency connection, you want it to work in ALL types of latency situations.

8

I think OP is just blinded by GGPO and never tried using UE4’s networking in first place, also for p2p you might have to find ways across NAT and firewalls

9

P2P has already been done in UE4. Rising Thunder was developed by the creator of GGPO, so I know that what I’m trying to do can work. I just don’t know quite how to finagle the engine into doing what I want, nor do I know exactly how much of the engine I need to avoid in order to do so. If at all possible, I would prefer not to have to either convert Character.cpp and CharacterMovement.cpp

This method sounds promising. Would you please elaborate? It sounds like every tick, I would store, say, the values I’m sending to the MoveForward and MoveRight functions, along with the frame number. Then, when it’s time to do a rollback, in GameMode’s tick function, I would reset each actor to their position on that frame, then iterate over each rolled-back frame for each PlayerController and call their MoveForward and MoveRight functions each frame with the same old data. Do I have that right?

I’ve already made up my mind about this. To me, there is value in using GGPO that is difficult to articulate or justify to most people.

10

I definitely see the point of it if you’re doing a fighting game. Is doing a logic/tiebreaker module on top of the existing UE4 networking a reasonable option? You don’t care about the transport layer, you just care about how the game decides who is authoritative for a particular situation and then how to integrate that back into each client, including anything that may have happened since then.

The P2P aspect of this seems almost unnecessary since what you actually want to do is rationalize how to solve situations then micro-manage them. How UE gets you that data is pretty irrelevant since nothing you’re doing will be managed by prediction anyway.

11

I figure that if Rising Thunder did that, they probably wrote their own complete networking system - or swapped out for something like RakNet (which you could also try, but it costs monies). You could perhaps clone all of the Unreal networking architecture and rewrite chunks of it into your own code but honestly, I have no idea where to begin with that and unless you can find a networking expert for UE it’ll be a bit of a struggle.

My suggestion would be to find a library that already does it, but CharacterMovementComponent is written for Unreals networking system quite specifically (as are a lot of the core engine classes, even Player Controllers). Fair play if you want to try and get GGPO/P2P working, but I personally think that it will be a long uphill battle for a while before you start seeing results and it might kill motivation. Additionally, UE’s networking integrates fairly seamlessly with PS4 / Xbone libraries (plus you get sent a bunch of UE-written libraries for those platforms when you register as a developer for them), so if you ever want to go down the console route it’ll do a lot of the hard work for you.

The Method I mentioned about playing back old moves is already written in CharacterMovementComponent so there’s an example of it there, but yes essentially you store the delta time for a ‘Move’ (Input + Results from Input), get the server to simulate it and send the results back. When the client receives the update, they ‘correct’ the move at that time, then replay all of the moves they made since, so the Client is always ‘Predicting’ what their moves will be, and the server makes minor corrections over time. Unfortunately it’s very implementation-specific so it’s something you’ll just have to dive in and have a pop at, but that website I linked has an example of this working (again though, it’s designed for a Server/Client architecture).

12

I don’t see why you couldn’t just hook in an existing GGPO library then provide event updates to the regular UE networking model, or manage your own replication and game state for things that matter.

Given the context is “fighting game” all the networking parts not to do with actual combat could revert to the normal UE networking methods.

Usually if the word “rewrite” or “build your own” comes up it’s not the right solution except in a few edge cases, but the context is simple enough that you’re basically replicating a handful of events then resolving logical collisions in your own way. I’d immediately revert back to UE networking for everything else like matchmaking and game state events, background animation replication, etc.

Jamsh’s post above sounds like the best idea though. Using the existing system to do it will mean you’ll gain the most benefit from the wheels that have already been invented.

13

Rising Thunder is the debut game for a new version of GGPO called GGPO3. It’s not available to all yet.

14

There is no authority with GGPO. Every peer sends its inputs to every other peer, and that’s it. It’s a type of deterministic lockstep networking; the difference is, the input delay buffer is optional, and can be removed entirely without creating a janky play experience for the player.

Normally, in fighting games, if you played online a lot, you would have to enable some sort of lag simulation option to mimic the input delay caused by the game’s bad netcode, because that delay would alter the timing of your combo inputs relative to the action you see on-screen. GGPO allows the user to not only decide how much or how little input delay they want (in order to mask the “teleportations” rollbacks can sometimes cause for other players’ characters), but also to decide to remove it entirely. That way, the game will feel as if there is no lag at all in terms of the local player’s control over his own character, eliminating the “lag factor” entirely. Suddenly, the advantage provided by mandatory input lag goes out the window.

Rising Thunder uses GGPO 3, which, as far as I know, is not yet available for third-party use. I was given GGPO 2, which should be enough.

I’m already feeling the motivational impact of getting this working. I haven’t made much progress on my game in about a week, largely in part due to the intangibility of the research this particular task requires. I don’t have anything to “show” for the time I’ve spent working on it thus far, and I haven’t written any code at all.

Anyway, it seems like I have about 9000 lines of code to read and understand. I’ll update the thread when I figure out the solution to getting this working. At the very least, once I have this done, I’ll be able to help out anyone else who wants to use GGPO in their UE4 game, depending on how much I need to modify CharacterMovementComponent.

15

I plan on only using GGPO for the “game state,” which would be defined as things like round time, score, position, velocity, and so on. Presentation would be included, but that’s later on down the road.

Avoiding reinventing the wheel as much as I can is the purpose of this thread. :slight_smile:

16

So to get my head around it, you’re using GGPO as a way of resolving two different clients having paradoxical states, as opposed to just letting the server win every conflicting decision. Taking Street Fighter as the example again, context is everything and GGPO would give you a way to apply logic to the decision-making when deciding which move won. Surely you could parse that down to just what’s necessary, like you said above: position, velocity and I guess attack.

So could you add this behaviour as a custom message on top of the UE4 networking? The other side to that coin is you’d need to modify the pawn state yourself and fire off the right hit events. Then let UE handle everything else such as catchup blending.

I’d be interested in how they can remove the delay from resolving an outcome of two clients interacting. My impression from SF has always been that it did need some idle buffering frames in each animation to avoid obvious corrections, but they didn’t have the benefit of skeletal blending. In fact, thinking about it from frames-of-animation the way SF does seems to be the clearest way of visualising it, except like someone above said, you’re applying the logic to hitboxes and the pawn state; the animation is pretty much OK to lag behind and allow UE to blend it towards a constantly moving target.

17

To explain what GGPO does, I’ll walk through a few frames of an online game. Let’s assume the latency is 60 ms, which is about 4 frames. Since the latency is low, the user will probably set their input delay to around 2 frames, in order to mask the rollback effect slightly.

Frame 0: Player 1 walks backwards. Player 2 was holding back before the round started, so GGPO assumes that input remains unchanged, and player 2 walks backwards.
Frames 1-2: P1 is still walking backwards. We haven’t gotten P2’s input yet, so he keeps walking backwards.
Frame 3: P1 continues in the same direction, but we just got P2’s frame 0 input, which was actually forward. GGPO rolls back to frame 0, replays it with the old local input and the newly-received P2 input, saves it as the last caught-up frame, then resimulates each frame after that with the new input predictions (holding forward) before running frame 3.
Frame 4: P1 still holds back, while GGPO just got P2’s frame 1 input of forward. GGPO saves that frame’s game state as the last caught up frame, and doesn’t bother rolling back and resimulating the frames in between because its prediction was correct. Both players, on this frame and the previous one, see and “feel” zero lag effects.

Since there are 4 frames of lag and an input delay of 2 frames, assuming the lag doesn’t get much worse than 4 frames, the worst lag either player will notice, if they can even notice it, is a 2-frame difference on the startup of any action the other player takes. Locally, P1’s character will only have a uniform 2-frame input delay on its moves, which is difficult to notice. If the player doesn’t mind the slightly-off jumpiness of the rollback frames, they can set the input delay to zero and have no lag at all on their control. For fighting games (and similar frame-sensitive genres), rollback netcode is considered to provide the best online competitive experience, and GGPO is the de facto best implementation of that algorithm.

18

Hello I sent you a PM. I’m trying to do the same, the SAME!