Hello!
I followed suit and did some changes to make it work. I got a couple of lessons out it too.
To keep tabs on the rotation angle I did on the lemon each time, I added markers to it with a sharpie so I can make sure that A) I could track the orientation properly each time and B) make sure the program had enough track points from the images.
I also changed the camera I was using due to the lens.
The Nikon D750 I was using has 24 megapixels, and the Sony ZV-1 I used next has 20 megapixels, BUT my Nikon had a 55mm lens tops. The Sony can go up to 70mm.
I mention this the 70mm lens allowed me to get photos that took more of the space of the photo with the asset. So I traded megapixels for pixel space.
I priorized aperture in the photos and I have tried on two lighting setups, studio and flash.
I am glad to report that with those changes (thanks orthan for the insights) I have managed to get all my scans aligned succesfully at first try. I am attaching a gif of one example.
Thanks for all the help, now onwards and upwards!