SetActorRelativeRotation setting as world

Are you calling this every frame? If you do, it is much easier to compute a DesiredRotation and then smootly interp to this rotation every frame.

FMath::RInterTo() should give you decent results.