Panning bug

In the video you will see a white box playing.
First I am facing up and turning my head. Observe that the sound is pivoting from one ear to the other in an unnatural way.
Then I turn my head down and down the same thing, the same problem is there.
This is not a problem when you are facing the white box. Panning sounds fine.

How do I fix it?