Hi,
Hope you’re doing well.
There is an easy fix, simple get your start rotation dynamicly, instead of use it as zero always.
I think this simple change my solve your issue.
Hope that helps - take care!