Camera Parameters in the output XMP file

[JesseJiang010]

A questions on Metadata (XMP) files in Reality Capture Help Documentation.

Do xcr:Rotation and xcr:Position values in the sample XMP file correspond to the origin of the World Coordinate System (P_w) in Reality Capture?

If not, what do these parameters refer to?

[OndrejTrhan (otrhan)]

Hi Jesse,
those parameters refer to RealityCapture’s XMP coordinate system (https://dev.epicgames.com/community/learning/knowledge-base/mozL/capturing-reality-differences-between-exported-coordinate-systems-in-realitycapture#:~:text=and%20Y%20axis.-,Same%20as%20XMP%20coordinate%20system,-The%20XMP%20coordinate).

[JesseJiang010]

Hi Ondrej, thanks for your quick response.

1. If the parameters refer to the XMP coordinate system, does that mean:

• for specifying the origin of the XMP coordinate system, we set a ground control point at (0, 0, 0)
• then re-align images
  Finally, the images’ rotations and positions turn to refer to the ground control point (0, 0, 0) we have created

?

2. Is it correct to use XMP Coordinate System as the World Coordinate System of the final alignment?

3. If the above is not the case, where to find the origin of the XMP coordinate system. Because I want to convert the position of a specific point in Camera Coordinate System into World Coordinate System
   (Suppose we already got the position in Camera Coordinate System by converting the 2D position in image using Camera Intrinsics).

Thanks always for your kind support.

[OndrejTrhan (otrhan)]

As I sent you, XMP system refers to the system after first alignment. The centre of the system is in the grid centre you can see in 3D view.
For such transformation this could be helpful:https://dev.epicgames.com/community/learning/knowledge-base/vzwB/capturing-reality-realitycapture-xmp-camera-math

[JesseJiang010]

I greatly appreciate your consistent and helpful support. I have a follow-up question regarding the article you provided.

I’ve been attempting to simulate the transformation using the same values mentioned in the article. However, I’ve encountered a discrepancy at the general projection equation stage. According to the article, the output coordinates x and y should be in the range of <-0.5, 0.5>. In my simulation, the y-coordinate falls outside this range.

Here is the code:

import numpy as np

# Step 1: Define the camera rotation matrix (R)
R = np.array([
    [-0.600806990019897, 0.799386597570746, -0.003468193376912],
    [-0.210825371930633, -0.162635135620312, -0.963899618846316],
    [-0.771092486861039, -0.578386445454965, 0.266243303052733]
])

# Step 2: Define the camera position in world space
position = np.array([2111.44219951044, 1607.86624656544, 2302.25896526736])

# Step 3: Calculate the translation vector (t)
t = -np.dot(R, position)

# Step 4: Create the transformation matrix (T)
T = np.hstack((R, t.reshape(3, 1)))

print(f"Camera Rotation R:\\\\n{R}")
print(f"\\\\nTranslation Vector t:\\\\n{t}")
print(f"\\\\nTransformation Matrix T:\\\\n{T}")

# Step 5: Define camera calibration parameters
focal_length_35mm = 82.2539160239028
sensor_width = 36  # Assuming 35mm equivalent
focal = focal_length_35mm * sensor_width / 36
ppU = 0.00621063808526977  # Principal point U-coordinate
ppV = -0.0214264554930412  # Principal point V-coordinate

# Step 6: Create the camera calibration matrix (K)
K = np.array([
    [focal, 0, ppU],
    [0, focal, ppV],
    [0, 0, 1]
])

print(f"\\\\nCamera Calibration Matrix K:\\\\n{K}")

# Step 7: Define a world point (in this case, the world origin)
Pw = np.array([[0], [0], [0], [1]])

# Step 8: Transform the world point to camera coordinates
Pc = np.dot(T, Pw)

# Step 9: Project the point onto the image plane
p_uv = np.dot(K, Pc[:3])

# Step 10: Normalize the projected point
p_uv_normalized = p_uv[:2] / p_uv[2]

print(f"\\\\nNormalized Image Coordinates:\\\\n{p_uv_normalized}")

The output I’m getting is:

Normalized Image Coordinates:
[[-0.36392172]
[123.70203034]]

As you can see, the y-coordinate (123.70203034) is well outside the expected range of <-0.5, 0.5>.

Are there any problems with this code? or Is there anything wrong with the article?

Thank you once again for your time and expertise. I look forward to your response.

[OndrejTrhan (otrhan)]

The mentioned workflow in the article should be correct. It is possible, that the values are not proper, as it was just an example.
Are you using proper scale and image width/height values?

[JesseJiang010]

Thanks always for you prompt response and kind support. I’ve successfully solved the issue of transferring 3D points to 2D coordinates on the image. It’s working perfectly, showing the exact pixel position in Reality Capture!

Now, I have a follow-up question about the reverse transformation—from 2D to 3D.

Is there a way to input a 2D point (in pixel coordinates) to generate its corresponding 3D position while Reality Capture is aligning the images?

(The area where the point is located has low texture, so Reality Capture isn’t generating feature points there.)

[OndrejTrhan (otrhan)]

After alignment you should be able to compute the 3D position of the point using reversing process.

[JesseJiang010]

Thanks! It would be really helpful if we could directly estimate the 3D position of a specific 2D point after alignment. But how exactly do you reverse this process? Are there any resources that explain this?

[OndrejTrhan (otrhan)]

You can do it in RealityCapture, but for such places it could be not so precise. Like, place at least two control points on the images and then you can check the value in pixels and 3D pose of the aligned model.


To reverse the process, I meant to compute it backwards. Like, you know the 2D coordinates, so
(px - (image width / 2)) / scale = m[0]
and so on.

For this we don’t have any documentation, but it is possible to find some way over the internet (I mean the general process).