After reading over your post it appears that you are encountering gimbal lock. I have provided a visual example below to help you understand gimbal lock.
Additional information on quaternion math and spatial rotation can be found here: https://en.wikipedia.org/wiki/Quaternions_and_spatial_rotation
There are several methods to get around Gimbal Lock, the easiest would be to rotate the actor along the X axis (yaw) as opposed to the Y axis (pitch). Another (significantly more difficult) would be to manually convert the rotator to a quaternion and then back to a rotator, which should allow you to work through Gimbal Lock and continue rotating correctly. I hope this information helps.
Gimbal locked airplane. When the pitch (green) and yaw (magenta) gimbals become aligned, changes to roll (blue) and yaw apply the same rotation to the airplane.
Make it a great day