As you said, the key is going to be staying out of Euler rotation. Why are you converting it from a Quat back into a rotational matrix? Once you have your rotation as a Quat it should be all good. SetActorRelativeRotation appears to only use the Quat contained within FRotator, I don’t see it converting things back to Euler angles.
You may also want to sanitize any rotational matrices when they first come in (i.e. make sure all the axis are orthogonal). UE4 has a special matrix for rotational matrices called FRotationMatrix (check RotationMatrix.h) which has some helper methods for constructing a FRotationMatrix and keeping things orthogonal.