Determining reconstructed 3D point (x,y,z) coordinates using input image (x,y) coordinates

Terve Jukka.

I am sorry, I forgot to mention that the projection of sfm points into cameras must be done in sfm space instead of model space. Thus You need to use the raw sfm point coordinates. To obtain them call ISfmReconstruction->GetStructure(). Each SfmReconstructionPoint contains a WorldPoint X. SfmReconstructionPoint.w is a valid non-zero positive value. The ISfmCameraModel->GetProjection() eats the WorldPoint as it is in its raw non-normalized form, but in the case You would like to draw it on screen in sfm space, do not forget to normalize it (i.e. x = x / w, y = y / w, z = z / w, w = 1.0).

The obtained projection of a 3d point from sfm space into a sfm camera is in relative coordinates, i.e. the longer dimension is in the range [-0.5,+0.5] and the shorter dimension in the range [-0.5*ar,+0.5*ar] where ar is aspect ratio (longer dimension / shorter dimension). The third value is depth which You probably don’t need.

What RC does is that it guesses a 3d model (structure) from 2d images (cameras). This can be, in a more abstract way, represented as a mapping from 2d image pixels to 3d points, which is what You are actually asking for. Simply, I am trying to say that this inverse process is not that easy and straightforward as mapping 3d points to 2d images. What You can try to do is get depth maps calculated from the reconstructed model by CreateModelVisibility() and facilitate them to pair 2d input image pixels with reconstructed 3d model surface points.