Rotating a vector returns incorrect results

More likely is the values are the same as they were in BP. The BP debugger just shows you less precision than VS’s debugger and does not use exponential notation.