For me it had to do with the SpringArm default setting for “Lag”
Once I disable the camera lag i no longer had an issue
UE5.5