AR Lens Calibration Issues

Development Cinematics & Media Virtual Production
, ,
1

We’re looking to create a 1 to 1 representation of our live reference cameras within the engine. When going through the lens calibration process detailed here:

https://dev.epicgames.com/documentation/en-us/unreal-engine/camera-lens-calibration-quick-start-for-unreal-engine?application_version=5.6

The calibration results in our virtual checkerboard offset from its physical counterpart.

Current Setup (Hardware & Software)

  • 4 Red Komodo 6K cameras
  • Each fitted with a DCS lens encoder
  • Resolution setting set to 4K 16:9 (which adds a crop factor of 1.58x)
  • Using Optitrack Motive to track the position of the cameras
  • We typically use 3 different types of lens for shoots.
    • 16mm - 35mm
    • 18mm - 80mm
    • 70mm - 200mm

Current Setup (In Engine)

  • 4 CineCamera Actors
  • Each with the Following components:
    • Lens Component
    • 2 Livelink Controller Components
      • One to handle streaming in lens data from the DCS units
      • Another to handle streaming in the Mocap data from Motive
2

Steps to Reproduce

  • Create BlackMagic Media Source asset
    • Configure BlackMagic Media Source asset
    • 1080p 60fps
  • Create Lens File asset (Used 18mm - 80mm lens)
    • Assign BlackMagic Media Source
    • Configure lens file
      • Change sensor dimensions to match our camera
      • Change Image Resolution to match our camera
  • Add cinecamera to the scene
    • Change Filmback and Lens settings within camera component to match our camera
  • Connect to livelink sources via livelink hub
    • Motive
    • DCS
  • Add Lens component to cinecamera
    • Assign Lens file to lens component
  • Add Livelink Component Controller to cinecamera (for Optitrack livelink stream)
    • Create rigid body within Motive
    • Move camera rigid body pivot to the position and orientation of the camera’s sensor within Motive
    • Assign Livelink subject to cine camera
  • Add Livelink Component Controller to cinecamera (for DCS lens encoding system livelink stream)
    • Create lens table for each lens within DCS (18mm-80mm, 16mm-35mm, 70mm-200mm)
    • Calibrate DCS
    • Assign Livelink subject to cine camera
  • Add Checker Board Calibrator to scene
    • Create physical checker board
    • Create rigid body for Checker board in Motive
    • Attach Checker board to parent actor
    • Add livelink component controller to parent actor
    • Assign Livelink subject to parent actor
    • Configure Checkerboard parameters
      • Num Corner Rows
      • Num Corner Cols
      • Square Side Length
  • Within Lens file Configure Lens Distortion settings
    • Calibration Pattern: Checkerboard
    • Calibrator: CameraCalibrationCheckerboard
    • Is Calibrator Tracked: Enabled
    • Is Camera Tracked: Enabled
    • Show Overlay: Enabled
  • Capture multiple samples for the checker board at different angles within the lens file
  • Press calibrate Lens
    • Typically we get a high reprojection error
  • Configure settings within Nodal Offset tab
    • Nodal Offset Algo: Nodal Offset Checkerboard
    • Checkerboard: CameraCalibrationCheckerboard
  • Click Checkerboard within lens file viewport to gather points
  • Click Apply to calibrator
    • At this point the virtual checkerboard and the physical checkerboard are still offset from each other
3

Hey Nateon,

Are you able to share the calibration data, including the images? Those are automatically saved under PROJECT/Saved/CameraCalibration

4

Hey Nateon, thanks for sharing that data! A couple of thoughts I have just from looking at the images and the data(and my apologies that some of these things are not as well documented as they could be):

First, how high was your lens distortion reprojection error? Our solver has the option to try a few different constraints to get a more accurate calibration, but the more constraints it applies, the higher the reprojection error tends to be because we have given the solver less degrees of freedom to solve for. Anything less than ~1.0 pixels might not be too bad, but more than that could be worrisome.

Second, did you feel that the calibrated focal length was accurate? If so, double check that the focal length of the CineCamera is actually the calibrated focal length from the LensFile, and not the calibrated focal length from DCS. Alternatively, if you want to trust the focal length from your DCS calibration, you could try running the distortion solve with the “Fix Focal Length” option checked and see if that improves your result at all.

Let’s see if we can narrow down whether your issue is with your distortion calibration or your nodal offset calibration. The first trick we use to verify our distortion calibration for ourselves is:

  1. Using the Nodal Offset Tool, capture a single image of the checkerboard.
  2. Click “Apply to Calibrator”, rather than “Apply Nodal Offset”. This will move the checkerboard actor in space to try to align it with the camera in its current location.
  3. You may need to temporarily disable the Motive LiveLink component on the checkerboard from evaluating.
  4. If the physical and virtual boards are aligned well, then the distortion is probably good. If the shapes are different (i.e. one is more distorted than the other) then the distortion calibration may need to be done again.
  5. Repeat this “Apply to Calibrator” process a few times with the physical board framed in different parts of the image. Distortion should be more extreme at the edges/corners of the frame, so you can stress test it there.

Now, let’s assume that your distortion calibration was good for a minute. Looking at your nodal offset data, I see that you only ever capture one checkerboard worth of points. Could you try capturing ~4-8 images with the checkerboard in different orientations. Do not move your camera in between captures, just the board. Sometimes having additional data can help the nodal offset calibration.

If those things did not work, or if you think the distortion calibration was not good enough, here are some additional ideas I have based on the data/images you sent:

  1. Looking at your captured images, I can tell that you sometimes move the checkerboard between captures and sometimes you move the camera. While mathematically it should not really matter, we have some optimizations in our solver that work much better if you leave the camera stationary throughout the entire capture process and only move/rotate the checkboard.
  2. From our own experience doing calibrations in our lab, we have found that we get more accurate results when we add a lot of variance to the angle of our checkboard along all 3 major axes (pitch/yaw/roll). You definitely have some rotation between captures, but I recommend trying to capture some images with the board pitched forward/back, rolled more drastically left/right, some with mixed rotations, etc.
    1. Note: Just be careful not to roll it too much or the pattern might capture upside down (and thus tracked position of TopLeft might accidentally be BottomRight, for example).
  3. Given that you have a crop factor of 1.58x, double check that the Filmback and Camera Feed Info measurements are accurate in the calibration tool and on the CineCamera. It probably is, but this has bitten us in the past when we forgot to adjust the filmback on our CineCamera to be the cropped filmback we were shooting at, and instead left it using the full sensor size.
  4. Try setting a focal length estimate in the lens distortion solver settings (You maybe already did, I just didn’t see it mentioned in your detailed steps).

I’ll add one other suggestion from Kevin Cushing:

“The one thing that stands out is the tracked checkerboard. It has to be aligned as carefully as possible in Motive to match the axis placement and rotation expected of the Camera Calibration Checkerboard in UE. If not I have seen high errors.”

Hopefully at least of those suggestions is helpful.

Cheers,

Geoff

5

Hey Nateon,

  1. The focal length guess does not need to be pinpoint accurate, it just gives the solver a starting point for the minimization problem. It can honestly usually figure itself out even if the initial guess is wildly wrong, but starting close to what you think it should be (even what is just printed on the lens) should be good enough to avoid falling into a local minimum.
  2. Yes, the “Solve Nodal Offset” option on the Lens Distortion tab will instruct the solver to do a global solve of all of the camera parameters (intrinsics and extrinsics) at once, thus allowing you to skip the manual nodal offset step afterwards.
    1. This option was added because we found it could give us more accurate calibrations than doing distortion and nodal offset as two separate steps. In our experiments, we found that the focal length and nodal offset could easily compensate for one another and lead to both being inaccurate. Solving for everything simultaneously constrains the system more, so the reprojection error tends to be higher (as mentioned, 1.0 pixels would not be alarming in this scenario), but the overall physical values are more accurate.
    2. This does require the calibrator and camera to both be tracked, and as mentioned before, will perform best if the camera remains stationary throughout the capture process.
  3. Yeah, you are exactly right about those various “Fix” options. They instruct the solver to use the initial values you give it and not to change them during its optimization. I would consider them as “advanced” options for cases where you have values that you believe to be accurate already, perhaps from a previous calibration or from using some other calibration software.

My apologies for the confusion, the button I referred to as “Apply Nodal Offset” is called “Add To Nodal Offset Calibration”. I just forgot what it was called and didn’t double check myself by looking at the UI.

Reprojection Error of 30 - 70 pixels is way too high, so something is definitely wrong. My best guess is that it is still related to the crop factor or some other camera measurement, but I’m really not sure where. Your setup is more advanced than our usual test setup, with multiple things applying a crop.

A couple more questions then to continue ruling things out:

  1. Have you tried any other cameras or lenses, maybe a prime lens without any extra adapters mounted to it, just to see if you can get a good distortion calibration with another setup?
  2. Have you tried doing your distortion calibration without checking the boxes for Is Calibrator Tracked and Is Camera Tracked? I’d be curious to know, if you run the solver using the same data (i.e. import your existing dataset of images) but without those additional constraints, if you get a lower reprojection error. That could potentially point to something being wrong with your taped markers, or something else with the tracking data.

Cheers,

Geoff

6

Here is Calibration Data - Is this enough?

7

Hey Geoff.

We’ve tried your suggestions and still seem unable to get a solid camera calibration.

Here are a few more questions for you after continued testing:

  • How accurate does focal length guess have to be when inputting in a number? The number we would put is from a calculation factoring in a crop factor and an adapter attached to our cameras.
  • Does “Solve Nodal Offset” basically do the nodal offset after calibrating? Would this allow us to skip doing a manual nodal offset? Or is it still the proper workflow to use the Nodal Offset tab for more refined tweaks?
  • We’re still confused about what “Fix Image Center During Optimization” and “Fix Distortion During Optimization” do. From my experiences, it seems like “Fix Focal Length” prevents the vcam from making its own estimated calculation to be similar to the physical camera and keeps the preset ones you assign it. Would this be correct?
  • Would it be possible to have another Zoom meeting, this time with the camera calibration team or the team that designed these tools?

Thanks for all of your help on this, we really appreciate it. Attached is a PDF that details our process from your suggestions.

-Nateon

8

With prime lens I put in the value of the focal length that manufacturer assigns. In my experience I do my first lens calibration at focus distance of about 150cm and calculate the nodal offset at the same time as the distortion and focal length. Will get a better focal length estimate.

At all other focus distances I do not calculate nodal offset as this should not be shifting as you rack focus and it is already applied.

You are doing zooms and would be the same process at each focal length of the zoom as the offset will shift as you change focal length. However it should only change moving toward or away from the camera. May need to throw away any change in X and Y and just use the first calculation for these. If not may find that the image plane shifts up and down as rack focal length. Maybe use middle of the focal length range of the zoom as the start and then extrapolate from there.

Do optical center only once at each focal length of the zoom. Then all other focus distances do not calculate it and check on "Fix Image Center During Optimization. Otherwise may get the same effect as calculating nodal offset at different focus distances where the X and Y of the image seems to swim. If you do rack zoom while shooting then you may see this swimming effect as you may have optical center that is little different at each focal length. IF that is the case may have to throw them all away except for the one calculation of optical center.

The “Fix Distortion During Optimization” would just throw away the distortion calculation and would only get the estimated focal length during this process.

During lens calibration I use a lot of light and try and be on t 8 or greater. The small t stop seems to help. I have 3D printed marker attachement to put on my checkerboard, which is 3D printed on metal. I have sent a quicktime to Andy to forward to you that shows the process I go through to get a good lens cal. In talking with Geoffrey our guess is there is something off in the calculation of sensor with the double crop you are using. Andy is going to set up a zoom with you for later this week when Geoffrey and I are both available.[Image Removed]