@BlueMan I actually found what the colors meant. Sorry, I searched it before but didn’t find it, now rereading all of this again I found it.
Another issue related with smoothing: I store the transform before setting the server transform and replaying moves. After replay, I set back to the stored transform, calculate the error offset between position at the end of replay and the stored transform position. I then set the position to be the stored transform position + error offset * percentage based on the desync. Is that similar to what you described here: