Hi amfram,
you can try principles from this article: Tutorial: Banana Effect - What To Do If My Model is Bent
Mostly the option using precalibrated images (images with fixed calibration parameters)
Also, capturing not just in two lines could help and using some control points.