I am using AirSim as a plugin to UE (for which I’m fairly new) to model a suite of sensors which include a fisheye camera. AirSim uses SceneCaptureComponent2D to through its PIPCamera to achieve this. As far as I can see, the camera projection happens in UpdateSceneCaptureContents function. My idea is to use a custom projection matrix to achieve the fisheye effect in images. I was wondering whether this is the right way to do so. If not, are there any other alternative ways of achieving this (preferably using C++ coding instead of using UE editor)?
I don’t think this is possible by simply modifying the projection matrix. Have a look at OpenCVLensDistortion | Unreal Engine Documentation which basically distorts the linear projection using 6 radial and 2 tangential coefficients. Note however, that this happens after rasterization, leading to under/oversampling issues.
Thank you.
I was under the impression that if I can replace the projection matrix with on of the fisheye projections listed here, this could be achieved. But, by looking at the code, I wasn’t able to figure out a way to implement the theta ( angle in rad between a point in the real world and the optical axis).
I am aware of the the OpenCV Fisheye model. So, then the idea would be to capture the scene using standard perspective projection matrix and then post-process the captured image using the OpenCV distortion model. This would be inefficient, wouldn’t it be?
It is inefficient but a sane approach which works ok for moderate distortions. You have to take into account that under the hood there is a lot of screen space related computation that assumes linear projection. So, if you change the model underneath you will have to disable or reimplement parts of the rendering pipeline. Undersampling can be dealt with by increasing resolution. You can also have a look at the code of Lens Matched Shading which breaks the image plane into linear segments to better match the lens geometry to optimize sampling.
Thank you. I have made some progress using the OpenCV plugin. I will test it first and check the results. Thanks for the link.