Getting players to interact physically with an object like that is always tricky. What you probably need is some form of client-side prediction, which Smooth Sync unfortunately does not have built in. You can write your own client-side prediction on top of Smooth Sync, but I don’t expect it will be particularly easy.
If you do some research on client-side prediction the concept itself is not particularly difficult to understand, but implementing it will take some effort.