Home | History | Annotate | Download | only in py_calibration 
      1 Camera Calibration {#tutorial_py_calibration}
      2 ==================
      3 
      4 Goal
      5 ----
      6 
      7 In this section,
      8     -   We will learn about distortions in camera, intrinsic and extrinsic parameters of camera etc.
      9     -   We will learn to find these parameters, undistort images etc.
     10 
     11 Basics
     12 ------
     13 
     14 Today's cheap pinhole cameras introduces a lot of distortion to images. Two major distortions are
     15 radial distortion and tangential distortion.
     16 
     17 Due to radial distortion, straight lines will appear curved. Its effect is more as we move away from
     18 the center of image. For example, one image is shown below, where two edges of a chess board are
     19 marked with red lines. But you can see that border is not a straight line and doesn't match with the
     20 red line. All the expected straight lines are bulged out. Visit [Distortion
     21 (optics)](http://en.wikipedia.org/wiki/Distortion_%28optics%29) for more details.
     22 
     23 ![image](images/calib_radial.jpg)
     24 
     25 This distortion is represented as follows:
     26 
     27 \f[x_{distorted} = x( 1 + k_1 r^2 + k_2 r^4 + k_3 r^6) \\
     28 y_{distorted} = y( 1 + k_1 r^2 + k_2 r^4 + k_3 r^6)\f]
     29 
     30 Similarly, another distortion is the tangential distortion which occurs because image taking lense
     31 is not aligned perfectly parallel to the imaging plane. So some areas in image may look nearer than
     32 expected. It is represented as below:
     33 
     34 \f[x_{distorted} = x + [ 2p_1xy + p_2(r^2+2x^2)] \\
     35 y_{distorted} = y + [ p_1(r^2+ 2y^2)+ 2p_2xy]\f]
     36 
     37 In short, we need to find five parameters, known as distortion coefficients given by:
     38 
     39 \f[Distortion \; coefficients=(k_1 \hspace{10pt} k_2 \hspace{10pt} p_1 \hspace{10pt} p_2 \hspace{10pt} k_3)\f]
     40 
     41 In addition to this, we need to find a few more information, like intrinsic and extrinsic parameters
     42 of a camera. Intrinsic parameters are specific to a camera. It includes information like focal
     43 length (\f$f_x,f_y\f$), optical centers (\f$c_x, c_y\f$) etc. It is also called camera matrix. It depends on
     44 the camera only, so once calculated, it can be stored for future purposes. It is expressed as a 3x3
     45 matrix:
     46 
     47 \f[camera \; matrix = \left [ \begin{matrix}   f_x & 0 & c_x \\  0 & f_y & c_y \\   0 & 0 & 1 \end{matrix} \right ]\f]
     48 
     49 Extrinsic parameters corresponds to rotation and translation vectors which translates a coordinates
     50 of a 3D point to a coordinate system.
     51 
     52 For stereo applications, these distortions need to be corrected first. To find all these parameters,
     53 what we have to do is to provide some sample images of a well defined pattern (eg, chess board). We
     54 find some specific points in it ( square corners in chess board). We know its coordinates in real
     55 world space and we know its coordinates in image. With these data, some mathematical problem is
     56 solved in background to get the distortion coefficients. That is the summary of the whole story. For
     57 better results, we need atleast 10 test patterns.
     58 
     59 Code
     60 ----
     61 
     62 As mentioned above, we need atleast 10 test patterns for camera calibration. OpenCV comes with some
     63 images of chess board (see samples/cpp/left01.jpg -- left14.jpg), so we will utilize it. For sake of
     64 understanding, consider just one image of a chess board. Important input datas needed for camera
     65 calibration is a set of 3D real world points and its corresponding 2D image points. 2D image points
     66 are OK which we can easily find from the image. (These image points are locations where two black
     67 squares touch each other in chess boards)
     68 
     69 What about the 3D points from real world space? Those images are taken from a static camera and
     70 chess boards are placed at different locations and orientations. So we need to know \f$(X,Y,Z)\f$
     71 values. But for simplicity, we can say chess board was kept stationary at XY plane, (so Z=0 always)
     72 and camera was moved accordingly. This consideration helps us to find only X,Y values. Now for X,Y
     73 values, we can simply pass the points as (0,0), (1,0), (2,0), ... which denotes the location of
     74 points. In this case, the results we get will be in the scale of size of chess board square. But if
     75 we know the square size, (say 30 mm), and we can pass the values as (0,0),(30,0),(60,0),..., we get
     76 the results in mm. (In this case, we don't know square size since we didn't take those images, so we
     77 pass in terms of square size).
     78 
     79 3D points are called **object points** and 2D image points are called **image points.**
     80 
     81 ### Setup
     82 
     83 So to find pattern in chess board, we use the function, **cv2.findChessboardCorners()**. We also
     84 need to pass what kind of pattern we are looking, like 8x8 grid, 5x5 grid etc. In this example, we
     85 use 7x6 grid. (Normally a chess board has 8x8 squares and 7x7 internal corners). It returns the
     86 corner points and retval which will be True if pattern is obtained. These corners will be placed in
     87 an order (from left-to-right, top-to-bottom)
     88 
     89 @sa This function may not be able to find the required pattern in all the images. So one good option
     90 is to write the code such that, it starts the camera and check each frame for required pattern. Once
     91 pattern is obtained, find the corners and store it in a list. Also provides some interval before
     92 reading next frame so that we can adjust our chess board in different direction. Continue this
     93 process until required number of good patterns are obtained. Even in the example provided here, we
     94 are not sure out of 14 images given, how many are good. So we read all the images and take the good
     95 ones.
     96 
     97 @sa Instead of chess board, we can use some circular grid, but then use the function
     98 **cv2.findCirclesGrid()** to find the pattern. It is said that less number of images are enough when
     99 using circular grid.
    100 
    101 Once we find the corners, we can increase their accuracy using **cv2.cornerSubPix()**. We can also
    102 draw the pattern using **cv2.drawChessboardCorners()**. All these steps are included in below code:
    103 
    104 @code{.py}
    105 import numpy as np
    106 import cv2
    107 import glob
    108 
    109 # termination criteria
    110 criteria = (cv2.TERM_CRITERIA_EPS + cv2.TERM_CRITERIA_MAX_ITER, 30, 0.001)
    111 
    112 # prepare object points, like (0,0,0), (1,0,0), (2,0,0) ....,(6,5,0)
    113 objp = np.zeros((6*7,3), np.float32)
    114 objp[:,:2] = np.mgrid[0:7,0:6].T.reshape(-1,2)
    115 
    116 # Arrays to store object points and image points from all the images.
    117 objpoints = [] # 3d point in real world space
    118 imgpoints = [] # 2d points in image plane.
    119 
    120 images = glob.glob('*.jpg')
    121 
    122 for fname in images:
    123     img = cv2.imread(fname)
    124     gray = cv2.cvtColor(img,cv2.COLOR_BGR2GRAY)
    125 
    126     # Find the chess board corners
    127     ret, corners = cv2.findChessboardCorners(gray, (7,6),None)
    128 
    129     # If found, add object points, image points (after refining them)
    130     if ret == True:
    131         objpoints.append(objp)
    132 
    133         cv2.cornerSubPix(gray,corners,(11,11),(-1,-1),criteria)
    134         imgpoints.append(corners)
    135 
    136         # Draw and display the corners
    137         cv2.drawChessboardCorners(img, (7,6), corners2,ret)
    138         cv2.imshow('img',img)
    139         cv2.waitKey(500)
    140 
    141 cv2.destroyAllWindows()
    142 @endcode
    143 One image with pattern drawn on it is shown below:
    144 
    145 ![image](images/calib_pattern.jpg)
    146 
    147 ### Calibration
    148 
    149 So now we have our object points and image points we are ready to go for calibration. For that we
    150 use the function, **cv2.calibrateCamera()**. It returns the camera matrix, distortion coefficients,
    151 rotation and translation vectors etc.
    152 @code{.py}
    153 ret, mtx, dist, rvecs, tvecs = cv2.calibrateCamera(objpoints, imgpoints, gray.shape[::-1],None,None)
    154 @endcode
    155 ### Undistortion
    156 
    157 We have got what we were trying. Now we can take an image and undistort it. OpenCV comes with two
    158 methods, we will see both. But before that, we can refine the camera matrix based on a free scaling
    159 parameter using **cv2.getOptimalNewCameraMatrix()**. If the scaling parameter alpha=0, it returns
    160 undistorted image with minimum unwanted pixels. So it may even remove some pixels at image corners.
    161 If alpha=1, all pixels are retained with some extra black images. It also returns an image ROI which
    162 can be used to crop the result.
    163 
    164 So we take a new image (left12.jpg in this case. That is the first image in this chapter)
    165 @code{.py}
    166 img = cv2.imread('left12.jpg')
    167 h,  w = img.shape[:2]
    168 newcameramtx, roi=cv2.getOptimalNewCameraMatrix(mtx,dist,(w,h),1,(w,h))
    169 @endcode
    170 #### 1. Using **cv2.undistort()**
    171 
    172 This is the shortest path. Just call the function and use ROI obtained above to crop the result.
    173 @code{.py}
    174 # undistort
    175 dst = cv2.undistort(img, mtx, dist, None, newcameramtx)
    176 
    177 # crop the image
    178 x,y,w,h = roi
    179 dst = dst[y:y+h, x:x+w]
    180 cv2.imwrite('calibresult.png',dst)
    181 @endcode
    182 #### 2. Using **remapping**
    183 
    184 This is curved path. First find a mapping function from distorted image to undistorted image. Then
    185 use the remap function.
    186 @code{.py}
    187 # undistort
    188 mapx,mapy = cv2.initUndistortRectifyMap(mtx,dist,None,newcameramtx,(w,h),5)
    189 dst = cv2.remap(img,mapx,mapy,cv2.INTER_LINEAR)
    190 
    191 # crop the image
    192 x,y,w,h = roi
    193 dst = dst[y:y+h, x:x+w]
    194 cv2.imwrite('calibresult.png',dst)
    195 @endcode
    196 Both the methods give the same result. See the result below:
    197 
    198 ![image](images/calib_result.jpg)
    199 
    200 You can see in the result that all the edges are straight.
    201 
    202 Now you can store the camera matrix and distortion coefficients using write functions in Numpy
    203 (np.savez, np.savetxt etc) for future uses.
    204 
    205 Re-projection Error
    206 -------------------
    207 
    208 Re-projection error gives a good estimation of just how exact is the found parameters. This should
    209 be as close to zero as possible. Given the intrinsic, distortion, rotation and translation matrices,
    210 we first transform the object point to image point using **cv2.projectPoints()**. Then we calculate
    211 the absolute norm between what we got with our transformation and the corner finding algorithm. To
    212 find the average error we calculate the arithmetical mean of the errors calculate for all the
    213 calibration images.
    214 @code{.py}
    215 mean_error = 0
    216 for i in xrange(len(objpoints)):
    217     imgpoints2, _ = cv2.projectPoints(objpoints[i], rvecs[i], tvecs[i], mtx, dist)
    218     error = cv2.norm(imgpoints[i],imgpoints2, cv2.NORM_L2)/len(imgpoints2)
    219     tot_error += error
    220 
    221 print "total error: ", mean_error/len(objpoints)
    222 @endcode
    223 Additional Resources
    224 --------------------
    225 
    226 Exercises
    227 ---------
    228 
    229 -#  Try camera calibration with circular grid.
    230