Hey Ondrej. Thank you so much for your reply. I ended up manually removing GPS exif data from all the images and realigned. Now the issue is gone. Despite having explicitly told RC not to use GPS priors for alignment, it was still doing that.
Solved now.
Thanks again.