@xuKyosuke
It sounds like you are moving the player controlled pawn on the server. It looks like it should only be moved on the client in your case.
Does your scenario work when you move the pawn only on the owning client?
Let me know if I seem to be misunderstanding and I can try re-reading. Cheers.