Expand description
Camera Calibration and 3D Reconstruction
The functions in this section use a so-called pinhole camera model. The view of a scene
is obtained by projecting a scene’s 3D point into the image plane using a perspective
transformation which forms the corresponding pixel
. Both
and
are
represented in homogeneous coordinates, i.e. as 3D and 2D homogeneous vector respectively. You will
find a brief introduction to projective geometry, homogeneous vectors and homogeneous
transformations at the end of this section’s introduction. For more succinct notation, we often drop
the ‘homogeneous’ and say vector instead of homogeneous vector.
The distortion-free projective transformation given by a pinhole camera model is shown below.
where is a 3D point expressed with respect to the world coordinate system,
is a 2D pixel in the image plane,
is the camera intrinsic matrix,
and
are the rotation and translation that describe the change of coordinates from
world to camera coordinate systems (or camera frame) and
is the projective transformation’s
arbitrary scaling and not part of the camera model.
The camera intrinsic matrix (notation used as in Zhang2000 and also generally notated
as
) projects 3D points given in the camera coordinate system to 2D pixel coordinates, i.e.
The camera intrinsic matrix is composed of the focal lengths
and
, which are
expressed in pixel units, and the principal point
, that is usually close to the
image center:
and thus
The matrix of intrinsic parameters does not depend on the scene viewed. So, once estimated, it can be re-used as long as the focal length is fixed (in case of a zoom lens). Thus, if an image from the camera is scaled by a factor, all of these parameters need to be scaled (multiplied/divided, respectively) by the same factor.
The joint rotation-translation matrix is the matrix product of a projective
transformation and a homogeneous transformation. The 3-by-4 projective transformation maps 3D points
represented in camera coordinates to 2D points in the image plane and represented in normalized
camera coordinates
and
:
The homogeneous transformation is encoded by the extrinsic parameters and
and
represents the change of basis from world coordinate system
to the camera coordinate sytem
. Thus, given the representation of the point
in world coordinates,
, we
obtain
’s representation in the camera coordinate system,
, by
This homogeneous transformation is composed out of , a 3-by-3 rotation matrix, and
, a
3-by-1 translation vector:
and therefore
Combining the projective transformation and the homogeneous transformation, we obtain the projective transformation that maps 3D points in world coordinates into 2D points in the image plane and in normalized camera coordinates:
with and
. Putting the equations for instrincs and extrinsics together, we can write out
as
If , the transformation above is equivalent to the following,
with
The following figure illustrates the pinhole camera model.
Real lenses usually have some distortion, mostly radial distortion, and slight tangential distortion. So, the above model is extended as:
where
with
and
if .
The distortion parameters are the radial coefficients ,
,
,
,
, and
,
and
are the tangential distortion coefficients, and
,
,
, and
,
are the thin prism distortion coefficients. Higher-order coefficients are not considered in OpenCV.
The next figures show two common types of radial distortion: barrel distortion
( monotonically decreasing)
and pincushion distortion (
monotonically increasing).
Radial distortion is always monotonic for real lenses,
and if the estimator produces a non-monotonic result,
this should be considered a calibration failure.
More generally, radial distortion must be monotonic and the distortion function must be bijective.
A failed estimation result may look deceptively good near the image center
but will work poorly in e.g. AR/SFM applications.
The optimization method used in OpenCV camera calibration does not include these constraints as
the framework does not support the required integer programming and polynomial inequalities.
See issue #15992 for additional information.
In some cases, the image sensor may be tilted in order to focus an oblique plane in front of the
camera (Scheimpflug principle). This can be useful for particle image velocimetry (PIV) or
triangulation with a laser fan. The tilt causes a perspective distortion of and
. This distortion can be modeled in the following way, see e.g. Louhichi07.
where
and the matrix is defined by two rotations with angular parameter
and
, respectively,
In the functions below the coefficients are passed or returned as
vector. That is, if the vector contains four elements, it means that . The distortion
coefficients do not depend on the scene viewed. Thus, they also belong to the intrinsic camera
parameters. And they remain the same regardless of the captured image resolution. If, for example, a
camera has been calibrated on images of 320 x 240 resolution, absolutely the same distortion
coefficients can be used for 640 x 480 images from the same camera while
,
,
, and
need to be scaled appropriately.
The functions below use the above model to do the following:
- Project 3D points to the image plane given intrinsic and extrinsic parameters.
- Compute extrinsic parameters given intrinsic parameters, a few 3D points, and their projections.
- Estimate intrinsic and extrinsic camera parameters from several views of a known calibration pattern (every view is described by several 3D-2D point correspondences).
- Estimate the relative position and orientation of the stereo camera “heads” and compute the rectification transformation that makes the camera optical axes parallel.
Homogeneous Coordinates
Homogeneous Coordinates are a system of coordinates that are used in projective geometry. Their use
allows to represent points at infinity by finite coordinates and simplifies formulas when compared
to the cartesian counterparts, e.g. they have the advantage that affine transformations can be
expressed as linear homogeneous transformation.
One obtains the homogeneous vector by appending a 1 along an n-dimensional cartesian
vector
e.g. for a 3D cartesian vector the mapping
is:
For the inverse mapping , one divides all elements of the homogeneous vector
by its last element, e.g. for a 3D homogeneous vector one gets its 2D cartesian counterpart by:
if .
Due to this mapping, all multiples , for
, of a homogeneous point represent
the same point
. An intuitive understanding of this property is that under a projective
transformation, all multiples of
are mapped to the same point. This is the physical
observation one does for pinhole cameras, as all points along a ray through the camera’s pinhole are
projected to the same image point, e.g. all points along the red ray in the image of the pinhole
camera model above would be mapped to the same image coordinate. This property is also the source
for the scale ambiguity s in the equation of the pinhole camera model.
As mentioned, by using homogeneous coordinates we can express any change of basis parameterized by
and
as a linear transformation, e.g. for the change of basis from coordinate system
0 to coordinate system 1 becomes:
Note:
- Many functions in this module take a camera intrinsic matrix as an input parameter. Although all functions assume the same structure of this parameter, they may name it differently. The parameter’s description, however, will be clear in that a camera intrinsic matrix with the structure shown above is required.
- A calibration sample for 3 cameras in a horizontal position can be found at opencv_source_code/samples/cpp/3calibration.cpp
- A calibration sample based on a sequence of images can be found at opencv_source_code/samples/cpp/calibration.cpp
- A calibration sample in order to do 3D reconstruction can be found at opencv_source_code/samples/cpp/build3dmodel.cpp
- A calibration example on stereo calibration can be found at opencv_source_code/samples/cpp/stereo_calib.cpp
- A calibration example on stereo matching can be found at opencv_source_code/samples/cpp/stereo_match.cpp
- (Python) A camera calibration sample can be found at opencv_source_code/samples/python/calibrate.py
Fisheye camera model
Definitions: Let P be a point in 3D of coordinates X in the world reference frame (stored in the matrix X) The coordinate vector of P in the camera reference frame is:
where R is the rotation matrix corresponding to the rotation vector om: R = rodrigues(om); call x, y and z the 3 coordinates of Xc:
The pinhole projection coordinates of P is [a; b] where
Fisheye distortion:
The distorted point coordinates are [x’; y’] where
Finally, conversion into pixel coordinates: The final pixel coordinates vector [u; v] where:
Summary: Generic camera model Kannala2006 with perspective projection and without distortion correction