How to do physics replication? (Conceptual explanation)

Any idea how physics replication in Gang Beasts, Party Animals Is Done/Can be Done? Even on a conceptual level I have no idea how it can be done. The only idea is completely Server Authoritative characters, but that would surely result in a noticable input lag, which neither of the games has. Games such as Human Fall Flat, Wobbly Life, have Server Authoritative characters, and it is of noticeably worse quality than the other games I mentioned.

On a simpler scale, say 2 players are interacting with the same physics object at the same time. How can this be smoothly replicated? One idea I had, was that each player sends to the server the location of the object as they see it, and the server physically lerps the object to the average of all the client sent positions. Sounds bad, and did turn out bad (unstable).

But Gang Beasts and Party Animals do even the harder thing, that players not only can interact with the same physics object, but can smoothly interact with each other.