Mover & Network Prediction - Fixed Ticking Policy without Fixed Frame Rate

Development Programming & Scripting Multiplayer & Networking
, ,
1

Hello it’s me again :slight_smile:

At first we were using Fixed Ticking on NPP and Fixed Frame Rate, to achieve determinism in the simulation output in both Singleplayer and Multiplayer.

But with the FPS going down it would slow down the whole game, which was inconvenient. So we turned off Fixed Frame Rate and kept NPP on Fixed Ticking.

In single player it worked, we kept the determinism of the simulation without having any slow down of the whole game.

But doing this in multiplayer caused a lot of issues with rollback and desync between Client and Server, that seemed not fixable from what the past few days of me trying to do so told me.

In the end Fixed or Independent, we mainly want to insure determinism in the result of the simulation which currently is not the case in Independent mode.

I’m guessing I’m definitely doing something that I shouldn’t but I fail to see the proper path ahead. What’s the best way to achieve this ?

Thanks !

[Attachment Removed]

2

Steps to Reproduce[Attachment Removed]

3

Hi,

To get a better idea of the problem, I have a couple of questions.

First, is the desync happening to the owning client, the simulated clients, or both? Could you also provide some more detail on the nature of this desync? Are you seeing relatively frequent corrections, or does the character end up in a state where corrections are constantly happening every update?

Could you also provide some more context on what your character is doing when the corrections occur (i.e. is this happening during regular movement, or when switching modes, changing movement base, activating an ability that affects movement, etc.)?

Also, just to double check, are you using “Interpolated” or “Forward Predict” simulated proxy network LOD?

Thanks,

Alex

[Attachment Removed]

4

I think I’ve found the core issue.

First, I was incorrect when I said both pawns are causing corrections. It is only the Rider triggering a rollback.

The code is calling UpdateStatesFromExternalMovement on the Rider, from the Mount’s OnMoverPostSimulationTick function. Since the Mount character’s simulation is ticking first, the Rider’s simulation has not yet advanced its “pending frame”. When reading/writing to the Rider’s pending frame, it’s actually still pointing to the previous frame (so if the Mount character is simulating frame 218, we’re writing to the Rider’s frame 217 data location). The result is that when the server sends us its authoritative snapshot of the Rider’s frame 217, we’re actually comparing it against our locally predicted location of frame 218. So constant corrections will occur while moving.

I think this could work if you changed the flow so that the Rider is always responsible for calling UpdateStatesFromExternalMovement within its own simulation, and/or at a time guaranteed to be after all moving characters are finished.

[Attachment Removed]

5

That’s great it works in Fixed tick mode, but I still think there should be a way to get the mount/rider mechanic working for Independent ticking mode without corrections.

Any insight on the details of which actor is triggering the correction and whether it’s a location difference again?

[Attachment Removed]

6

Hey,

Sorry for the lack of context.

I’m using “Interpolated” simulated proxy network LOD.

The desync happen on the Autonomous Client. Simulated are fine

Regular movement is fine

It can happen rarely happen on Movement Change (Landing Ability Air -> Ground for example).

The worst by far is when my Player Pawn (Which use Mover) is set to follow to the Mount Pawn (which also use Mover), it desync every frame and when I unmount the Player Pawn (stop following the Mount Pawn) I’m in a state of permanent desync.

Which I know is a really touchy setup for the moment because of tick dependency, but the player desync all over the place, not just falling behind. With Fixed Frame Rate it was just falling a bit behind but was stable.

[Attachment Removed]

7

Hi,

Thanks for the additional info! Just to double check a couple more things:

Are you still using the workaround from [this previous [Content removed]

Also related to that case, are you still setting the mount’s input producer to the player’s, rather than having the client possess the mount?

Thanks,

Alex

[Attachment Removed]

8

Yes I’m still using the workaround and haven’t experienced issues with it (or a least until now I guess ?)

And yes I still forward the inputs from the player to the Mount instead of possessing it.

[Attachment Removed]

9

Hi,

Thanks again for answering all my questions!

After discussing this with the dev team, it does seem like possessing the mount pawn may be an approach worth exploring.

Because the client is forward predicting the locally owned character, it will effectively be ahead in the timeline when compared to the mount, due to the latency between the server and client. While the server is simulating the player and mount characters at sim frame X, the client will be forward predicting it’s locally owned character at a sim frame >X, and the mount on the client will be interpolating between some sim frames that are <X.

You did mention that this lag between the character and mount was stable with a fixed frame rate, but having the client possess the mount would allow both characters to be forward predicted on the client, hopefully allowing them to stay more in sync under a variable frame rate.

Thanks,

Alex

[Attachment Removed]

10

Thanks for taking the time to look into it !

It does make sense, I’ll definitely look into it. I’m still quite a new to UE so the concept of possessing is still not my first approach :sweat_smile:

On that note, how do I manage If my player can still do stuff while on the mount ? Like shooting for example. Because from my understanding, by possessing the mount the player will not receive the inputs anymore ? And it will switch to a Simulated Proxy instead so it will not run the simulation or produce inputs.

Or maybe I missed something ?

[Attachment Removed]

11

This is something we haven’t tried with Mover yet, but we shared details about how Fortnite does this in this issue:

[Content removed]

There are differences here since FN is using CMC for movement and GAS abilities for character actions.

Hopefully it gives some ideas to get started. Attaching the rider to the mount is key, possessing the mount, and routing inputs through it. I’d also look at putting the rider into the ‘null’ movement mode or a custom ‘mounted’ mode that does not attempt to move, as well as turning this flag on: bAcceptExternalMovement.

[Attachment Removed]

12

Thanks for your answer and sorry for my late answer, I was OoO

I was somewhat close from what is described in how they did it on FN.

But even with that in mind I still have issues and I tried to remove every other possible sources of issues (disabling Fixed Tick Smoothing, re Enabling Fixed Frame Rate, attaching to actor instead of Skeletal Mesh Component)

I don’t know if I’m doing something wrong but when Attaching my pawn to the Mount Pawn and enabling “bAcceptExternalMovement” it doesn’t seem to follow the mount or by a very small amount (with an Empty Movement Mode).

It seemed that this was called each time with a wrong SyncState even if it was modified in UMoverComponent::CheckForExternalMovement

[Image Removed]So I tried something like at the end of UBasedMovementUtils::UpdateSimpleBasedMovement

where it modify the Pending State. And I would call that in the PostSimulationTick of the MountPawn. The character would actually move and follow properly, only in single player sadly.

In multiplayer the PlayerPawn would desync every time when moving the mount.

I guess there is something in the order of updates or some other thing that changes the Component Transform. But I found it quite hard to isolate where or what can modify the Component Transform between smoothing, rollback, Finalize frame etc

To be more precise on the context:

  • Client Player Pawn use an Ability to Mount the Mount Pawn (which is replicated using a custom Ability System which also use NPP)
  • In this ability Client side, I change the Movement Mode to an Empty Movement mode and Attach my Player Pawn to the Mount Pawn and enable bAcceptExternalMovement.
  • When this ability is launched on the server (from the simulation), it does the same thing as above but also change the ownership and the Remote Role to Autonomous Proxy using the following Helper

[Image Removed]

  • The Mount Pawn then become an Autonomous Proxy on the Client, and the Produce_Input is called and gather the inputs from the Player Pawn which are then using on the Mount Mover Component simulation to move it.
  • And then desync happen every frame

I’m sure I’m missing something but can’t find what it is

[Attachment Removed]

13

I’m not surprised if there’s a bug in Mover that prevents this case from working yet, since internally the only mounted movement we’ve done was in a non-networked game.

My first guess is that somehow the same inputs aren’t being used, or they’re being shifted in time. What’s the nature of the desync you’re seeing? Constantly incorrect transform on the mount character? Does it calm down if you provide no movement input?

If you’re able to share a sample, that could help us pinpoint where things are going wrong.

[Attachment Removed]

14

I had quite some trouble to make a repro sample since it seems that the default Mover shipped with 5.6.1 doesn’t have the “bAcceptExternalMovement”.

So I had to include my local version of Mover which I last updated from the GitHub UE5-main early October

All the changes in the plugin are scoped between //LOCALCHANGE_BEGIN and //LOCALCHANGE_END

It’s a bit messy but it does have the same behavior.

You just have to move towards the Character on the left and Press E when near it to “Mount” (I included a video in the .rar)

You’ll see that when moving you’ll have desync and they’ll stop when not moving. It seems that the transform is constantly incorrect.

I also disabled “Fixed Tick Smoothing” because I think there is something going on there but not sure until the base issue is solved

[Image Removed]

[Attachment Removed]

15

Thanks for the project -- I’ve been able to get it running. The correction pattern is odd, like the client is getting 1 frame ahead of the server.

Both autonomous characters get constant corrections if moving. The client’s predicted locations for frame X are actually identical to the server’s locations for frame X+1. So they get into a kind of rapid thrashing back-and-forth while moving and then end up sync’d once movement comes to a rest.

The same inputs appear to be matching between client and server, as in they are identical for any given simulation frame number, so it feels like the post-sim changes are not being applied equally. Next thing I’d focus on is comparing the server vs client changes in CheckForExternalMovement / UpdateStatesFromExternalMovement for the same sim frame number.

There’s also the possibility of an update order difference. One thing that would test this theory is doing the UpdateStatesFromExternalMovement at a later time when you know that both mount and rider characters have moved. You could enable and schedule another tick function to run in the TG_PostPhysics tick group.

[Attachment Removed]

16

Sorry for the wait, I was working on something else.

One thing that is probably logical but different is that CheckForExternalMovement only trigger modifications on the Client and not on the Server (Except for the first frame where the attachment happens)

UpdateStatesFromExternalMovement changes does appears to not being applied equally on Client and Server but still can’t find why exactly

Update order seems to be identical on Server and Client by logging Owner Role and Owner Name in the SimulationTick

Doing the UpdateStatesFromExternalMovement in a TG_PostPhysics or TG_PostUpdateWork doesn’t appear to solve the issue

[Attachment Removed]

17

I’m taking a deeper pass at this, and see if I can figure out a way to make them behave. Will report back shortly.

[Attachment Removed]

18

Thanks a lot for taking a deeper look at this.

I moved the UpdateStatesFromExternalMovement from PostSimulationTick to PostFinalize. Because I feel like the change should be applied inside the scope of “UNetworkPredictionWorldManager::BeginNewSimulationFrame_Internal”. And also because other tries in TG_PostPhysic where unsuccessful.

It still wasn’t working properly, until I enabled the “Use Fixed Frame Rate” in General Settings. Which I guess is okay but wasn’t really what I was hopping for because it brings me back to the initial topic about Interpolated vs Fixed Ticking policy for NPP😅

[Attachment Removed]

19

Just to be extra clear, I meant “Fixed Frame Rate” from the Engine not the Fixed Tick Policy from NPP which I had enabled in both cases

It is very weird because when testing, sometimes there is no desync and then I redo the exact same test after restarting the PIE and it will desync. Not sure what to think about this :sweat_smile:

The external movement correction seem to stabilize to 12.80 cm which I guess is the max move speed per frame of the character. And both Server an Client are doing the exact same correction on the same frame. (But of course only the when it’s working properly)

In my testing, after Mounting I always wait for the first rollback caused by role change before moving to avoid any potential issues there.

I tried to log every step with Sim number to see what’s happening exactly. And found some weird stuff

The first few frames works nicely, External Movement modification seems to match with the server and everything is fine:

[Image Removed]

Then after frame 419 ,which still produce a similar External Movement correction:

[Image Removed]

There is a rollback: [Image Removed]

The first Simulation frame in the rollback triggers a big external movement correction:

[Image Removed]After this there is some weird behavior where we see a first time a Sim Frame 429 on Client which produce a correction. Then later there is another Sim Frame 429 Client which produce a different correction.

Also the first time there is 2 sim tick in the same frame instead of 1 the second time

[Image Removed]After that the simulations never agree until the movement is stopped.

Since when the few times everything works fine there isn’t any rollback done when the movement is started, my guess is that something weird happens in the rollback.

Maybe it’s not a proper way to prove this guess, but using the command np.SkipReconcile 1, everything works and I don’t see any visual offsets between Client and Server.

I hope this is something that gives you an idea because I’m out of it :sweat_smile:

[Attachment Removed]

20

This is getting pretty deep. A couple things stick out to me:

For the unexplained reason why things might be smooth one run and not the next, it’s probably tick ordering between the mount and rider flipping.

And if disabling reconciliation is fixing the issue, that would align with the case I saw where altering one character’s pending syncstate before that character has simulated for the frame would be changing the wrong frame. Might also be possible they’re just 1 frame off and it’s impossible to see without stepping through in slow motion?

I’m going to try incorporating this mount/rider mechanic into an example I’ve been working on and see if I can get it to work there. I’ll post an update here once it’s completed.

[Attachment Removed]