What are the model limits of the camera calibration plugin?

Development Pipeline & Plugins
, ,
1

Hey there!

I’m interested in using the Camera Calibration plugin in Unreal Engine for systems that require accurate modeling of lens distortion, particularly for fisheye and ultrawide lenses (with fields-of-view greater than 180 degrees).

It appears that the Camera Calibration plugin natively supports radial distortion models, which map real-world pixels radially to other pixel locations on the image plane, and the intro page for the plugin states

  • “The plugin’s tools and framework are extensible and flexible to support a wide range of lenses and workflows.”
  • “The plugin is extensible and provides the ability to add support for additional models with different parameters.”

However, it’s not clear to me if this includes support for fisheye and other ultrawide (> 180 degree FOV) lenses. The distortion of such lenses is typically modeled by mapping image pixels to unit vectors (rays) and transforming those, rather than by a simple radial transformation. This allows for the correct handling of extreme distortions, including cases where a pixel in the image corresponds to a direction behind the camera.

My questions are:

  1. Does the Camera Calibration plugin support distortion models that map pixels to rays (unit vectors), as required for fisheye and ultrawide lenses?
  2. If not natively supported, are there recommended workflows or extensions for handling such lenses within Unreal Engine?

Thanks!

Connor

2

Steps to Reproduce

3

Hi Connor,

No fisheye model. A challenge with fisheye is that you need multiple renders (eg scene captures or cube), then warp that. If you try to do it with a single render, as you increase the fov the amount of stretching becomes untenable. And the issue with multiple renders is seams and effects like bloom (same nDisplay outer issues). We’re going to work on a special bloom for nDisplay in 5.8 btw.

Offworld has their OWL 360 camera that can do fisheye but I’m not aware of them offering a lens calibration feature. They have their bloom implementation to deal with the multi render issue.

MSG uses multiple scene captures to model their big sky camera and warp into vcam render target. They probably modeled their distortion in that shader.