Yes, I did eventually 
I found that when I first added my control points, I ended up deleting a lot of them the first time, because the program would state they were too far off (50+ px) - I deleted all of those, which in turn made the model not able to correct the location of the images… So I added the control points again, even if they were claimed to be too far off (they were too far off in the current model because the current model was wrong ofc.) - when I then realigned the model were fixed, and the camera locations are now the correct place.
I also tend to first align with all images location active, then if the model does not end up straight (like rotated/tilted) then I will remove and set unknown on my phone images and let only the drone images set up the placement, and then it tends to get corrected when I realign 
Thanks for the help Götz