This does not work at all. Not only does my pitch go crazy when I do this but I even tried just setting the Yaw value to 1.0 and it set it to 90 in game. Why is it doing that and how can I make a stable rotation?
A quaternion rotation is defined differently from Pitch, Roll, Yaw. The real part of the quaternion is the rotation and the imaginary (i, j, k) part is a vector representing the axis of rotation.
To apply an arbitrary rotation Q (about a different axis) to an existing quaternion P (or a 3-vector represented as a quaternion with rotation zero, ie (0, x, y, z)), an operation called conjugation is used. Note that quaternion multiplication is not commutative, so usually PQ =/= QP. With Q^-1 representing the inverse of Q and P’ being the desired result:
P’ = QPQ^-1
I believe that it is best to make a function to apply this rotation; there may be one already in the math quaternion library.