Can you try using PlayerController->SetViewTarget to spectate target pawn, rather than setting location and rotation of spectator pawn
Can you try using PlayerController->SetViewTarget to spectate target pawn, rather than setting location and rotation of spectator pawn