I found the setting in the Physic Settings Data Asset. Compare State To Trigger Rewind. So that one is solved. Last issue is the fact that in standalone net mode launching the ball clears ownership but in multiplayer the server never clears the ownership somehow after it launches the ball.
UPDATE 1
I think I found the root cause of the launch in multiplayer. It seems like Scheduled instant movement effects have an issue.
This might also be my design. Basically when the client’s input requests a possession I route that to an instant movement effect the movement effect changes the state. Works fine in single player. In multiplayer it seems the instant effect is never removed so its constantly claiming the ball and when its launched there is a frame where the state is correct then the next frame the player claims the ball through the instant effect
The easy solution is to not route through instant effect and instead let send the winner of the ball possession through the input since I do the query in ProduceInputData anyways (it runs on the game thread allowing me to gather all of the other player’s sync states and see if they’ve made a claim for the ball.
I’m not sure if this will open things up for exploits.
UPDATE 2
A direct effect of turning on compare state to trigger rewind is now the pawn is constantly in resim. Almost every frame is a resim. This is probably what is causing the most issues. I’m guessing code isn’t truly rollback safe. I have to do some more digging.**
UPDATE 3
After more digging, the correction are not coming from my own custom sync state structs. FMoverDefaultSyncState is always out of sync or at least outside of the threshold. Quick solution is to bypass the reconcile there since physics resimulation should already handle positional errors. I do wonder if that will come back to haunt me down the line since I do use the mover sync state quite a bit to determine physics. I wonder if its better to use the controlled particle transforns when doing calculations based on player transforms. I still feel like my code needs to be improved to handle rollback. That way corrections don’t break the game. Right now, I can’t pin point the exact “thing” that is unstable in rollback but I will definitely keep digging tomorrow.
UPDATE 4
Today I found something odd. I have a counter in the sync state that is just counting how many frames have passes and for some reason the client is 3 frames ahead on the counter. The counter is incremented in the fixed tick of the physics thread. This might have something to do with the client dropping frames in order to keep the frame offset but I’m just guess. If I want to count time in this way what is the correct way to do it? My thoughts were it was safe to use the fixed tick to track elapsed frames but any code I have that is dependent on it is now fragile because the frame counter is not the same on client and server.
UPDATE 5
Okay, I’m realizing that command frames are not accurate on the client and server. Which is what I thought was required for deterministic physics. For example I’m sending inputs from the client to the server in the fixed tick. When the client hits frame 345 something in my input triggers an action to happen. That same action doesn’t happen on the server until frame 362. How is this even possible? Looking at the source code the TimeStep is filled out here
void FAsyncCallback::GetCurrentMoverTimeStep(const FAsyncCallbackInput* AsyncInput, FMoverTimeStep& OutMoverTimeStep) const
{
OutMoverTimeStep.BaseSimTimeMs = GetSimTime_Internal() * 1000.0;
OutMoverTimeStep.StepMs = GetDeltaTime_Internal() * 1000.0f;
if (AsyncInput && AsyncInput->PhysicsSolver)
{
OutMoverTimeStep.ServerFrame = AsyncInput->PhysicsSolver->GetCurrentFrame() + AsyncInput->NetworkPhysicsTickOffset;
OutMoverTimeStep.bIsResimulating = AsyncInput->PhysicsSolver->GetEvolution()->IsResimming();
}
}
With the offset applied so my assumption was that the client frame in the time step is the corresponding server frame. Maybe I’m getting NPP & Networked Physics confused in how they work. Another assumption I had was that if the client was handling an input on frame X then the server would also handle that input on the same frame since the client is running ahead of the server and sending inputs from the future. We definitely need some guidance on this subject because all I am making are assumptions.
UPDATE 6
I’ve decided that until I understand why time is not consistent between client and server to bypass and state reconciliation other than the basketball ownership. Unfortunately, I’m not able to keep the simulation consistent when state comparison is on. When its off the game play’s smooth and the playback in CVD is identical almost all the time. I’m not sure why the state is always out of sync when comparing. I still think the time dilation plays a part in it but I can’t be sure and trying to step through the code while it’s multithreaded feels impossible. The debugger is jumping in between worlds and each one is in a different position in the code. I might try to force physics to be single threaded with the console commands but I’m not sure if that will defeat the purpose of async physics. **EDIT Seems line this console variable is set to true by default.