I suppose you could drag out from the character movement component and set “Orient Rotation To Movement” = FALSE while it is being pushed and then set it back to = TRUE when the overlap stops.
EDIT: Since this is multiplayer, the player being pushed also has movement input that they will use to get away or push back, you might want to check if there is some conflict there.