Making Unreal Engine Deterministic/Rollback ready- Is it even possible? (Problems to tackle)

Hi,
I have read many topics about how this engine is not really good for networked games that need Deterministic behavior across multiple machines, like:

  • RTS games(Replicating thousands of units across machines is very network performance-consuming, and players won’t ever see 100% true state of the game, that’s why most of RTS games use deterministic Lockstep)

  • and fast paced Fighters(With UE’s replication system, before You see the Attack, You already get hit, GAS kinda reduces the impact of that with reconciliation mechanics, but it makes every attack less readable and some hits hit when from perspective of the player they shouldn’t, and effects of attacks aren’t instantly visible, they aren’t synced with animations etc.)

So I Gathered possible problems about determinism that need answering if they’re even possible to solve without rewriting 90% of game code:

  1. Floating Point Determinism

As Machines differ, the deltaTime of ticks with same calculations differ.
As mentioned in Game Engines and Determinism — Duality Robotics, constant timestep for ticking systems is needed for the engine to be deterministic.

Physics substepping wasn’t enough, but Chaos might solve this problem with async tick physics…
…But only for physics. And not completely, as I’m scared.

a) Async physics ticking can guarantee same results on same frame only when machine can perform enough for physics to tick at specified fixed delta, am I right? (it’s not a big problem, but problem worth mentioning)

b) If i’m wrong, there’s still another question. Other systems that calculate transforms(Character Movement for example) at specified framerate are still Hardware and performance-bound.

As floating point math is weird, there’s difference between moving 0.032211 units on one tick and first moving 0.022211 units on first and then 0.01 units on the next one (as article above mentions, it’s 0.032211000000000004).

The way to solve it is to limit fps…
…So it would seem…

There’s the question:
Am i right and if the game doesn’t perform at the fps limit, then we would desync again?

what made me wonder:

If the DeltaTime is bigger than the fixedDeltaTime, will we perform a tick with DeltaTime or FixedDeltaTime and then store the time difference in a timebank to add it to DeltaTime the next frame? (like with physics in Using a fixed physics timestep in Unreal Engine, free the physics approach)
First approach would desync slower machines from simulation, and second one wouldn’t(?correct me if i’m wrong?).

c) Then there are FPU optimisations. Some functions like sqrt and sin are optimised between machines. This can make simulations inconsistent even with everything above fixed!
There’s a way to disable them using other fp models(fp:strict to be exact, as others use machine-specific optimisations)…
…But is there a way to change it in the engine? Without writing it directly in every subsystem?

  1. Tick order of stuff

Tick order of actors in UE is… convoluted.
We know the order of tick groups, but within the tick group?

a) It’s random… or Is it not?
in WTF Is? Tick Prerequisite Actor in Unreal Engine 4 ( UE4 ) - YouTube The ticks are being called randomly before using the nodes in the video, but after I tested this on 4.26?
Technically the tick order of actors I placed was the same…I am baffled

If it’s not random and will perform on every machine the same…forget my question
If it IS random and can’t be predicted in any way for every machine possible…it’s a disaster

As I said, we can order stuff to tick one after another, but with many actors and components on map that need ordered ticking, explicit tick ordering will quickly become a Herculean task.

As we don’t care about which element ticks first, but about if the order of ticks is consistent across machines(because that’s the only thing needed to secure consistency of simulation about that aspect), we would waste SO…MUCH…TIME…

Not to mention in bigger scale projects it’s easy to forget something.

So if the tick order within groups isn’t machine-independent:
b) Isn’t there an option to order everything to tick in the same order every frame across machines? WITHOUT SETTING EVERY ELEMENT MANUALLY?
c) Is it the same with physics? I saw an option with bEnableEnchancedDeterminism and someone on the forum said it orders every physics task so it runs more deterministically…is it True?

  1. Resimulating (Rollback-Specific questions)
    I know that engine has a relatively easy serialization system for actors and objects, but:

a) Serializing Animation Montages, Latent actions and timers - Does the engine support saving states of them and what comes after they are performed?

b) Serializing Animation Blueprint states of meshes- is this possible?

c)Ticking/ resimulating whole world again- is there a Documentation on how to manually call ticks for certain systems and in which order they are called by default?
The only ingredient left after what I said is possibility to “Retick” every element/system needed to actual position in time and amount of frames, reinserting inputs at specific moments.

d) Would c) be Computationally expensive on relatively big games like 4-player RPGs ? I saw other games do this, but they’re usually 2 player fighters and on diferrent engines, so I don’t know how such stuff would affect relatively big games on UE.