Maybe it has something to do with the eyes being children to the head, so the head angle is added to the eyes angle. Of course eyes have to be children to the head for their position to be changed accordingly, so maybe you can adjust the angle of the eyes by the angle of the head.
Edit: What happens if the camera is to the right of this person? Where do eyes go then?