Hi JOHN,
the randomness in the results prooves that you are on the edge with your image set. Either the colours are the problem (which is only the case if they are close to solid with no texture) or the object of interest is too small in the frame. If it is only for alignment, a well patterned background will help a lot… Also, a rig can be pre-calibrated with an ideal object, which will then leave you “only” with the problem of meshing.
Could you upload two sample images (one for each shooting) here? Screenshots should be enough…