Yes, ‘set rotation’, that’s the problem
You might have some joy using ‘combine rotators’. You have to work with the current rotation, and the delta between that and the required rotation. Something like
