Hello, It also happen a lot to me. I’m almost certain it happen whenever the alignment is to ‘difficult’ for RC. (bundle adjustment is not converging?)
Try to reconstruct small portion of the scan first, or add control points.(Or do both it’s even better)
It always work for me at some point, but it can be painfully slow & long…