Master Mathematical Analysis and Applications Course M1 - S2 Computer vision Image formation and Camera Calibration Week 9-10 Mohammed Hachama [email protected] http://sites.google.com/site/hachamam/ University of Khemis Miliana -March 2020-

Image formation Plan 1. Image formation and Camera Calibration Computer vision -Image formation and Camera Calibration- (Week 9-10) (2/9) M. Hachama ([email protected])

Image formation Image formation Camera calibration Image formation Computer vision -Image formation and Camera Calibration- (Week 9-10) (3/9) M. Hachama ([email protected])

Image formation Image formation Camera calibration Image Formation • How to form an image • Place a piece of film in front of an object → Do we get a reasonable image ? Computer vision -Image formation and Camera Calibration- (Week 9-10) (4/9) M. Hachama ([email protected])

Image formation Image formation Camera calibration Image Formation • How to form an image • Add a barrier to block off most of the rays • This reduces blurring • The opening is known as the aperture Computer vision -Image formation and Camera Calibration- (Week 9-10) (4/9) M. Hachama ([email protected])

Image formation Image formation Camera calibration Image Formation • Effects of the Aperture Size • In an ideal pinhole, only one ray of light reaches each point on the film the image can be very dim • Making aperture bigger makes the image blurry Computer vision -Image formation and Camera Calibration- (Week 9-10) (4/9) M. Hachama ([email protected])

Image formation Image formation Camera calibration Image Formation • Effects of the Aperture Size • In an ideal pinhole, only one ray of light reaches each point on the film the image can be very dim • Making aperture bigger makes the image blurry Computer vision -Image formation and Camera Calibration- (Week 9-10) (4/9) M. Hachama ([email protected])

Image formation Image formation Camera calibration Image Formation • Effects of the Aperture Size • Why not make the aperture as small as possible ? • Less light gets through (must increase the exposure) • Diffraction effects... Computer vision -Image formation and Camera Calibration- (Week 9-10) (4/9) M. Hachama ([email protected])

Image formation Image formation Camera calibration Image Formation • Image formation using a converging lens • A lens focuses light onto the film • Rays passing through the Optical Center are not deviated Computer vision -Image formation and Camera Calibration- (Week 9-10) (4/9) M. Hachama ([email protected])

Image formation Image formation Camera calibration Image Formation • Image formation using a converging lens • All rays parallel to the Optical Axis converge at the Focal Point. Computer vision -Image formation and Camera Calibration- (Week 9-10) (4/9) M. Hachama ([email protected])

Image formation Image formation Camera calibration Image Formation • In focus • There is a specific distance from the lens, at which world points are “in focus” in the image • Other points project to a “blur circle” in the image Computer vision -Image formation and Camera Calibration- (Week 9-10) (4/9) M. Hachama ([email protected])

Image formation Image formation Camera calibration Image Formation • Projective Geometry • What is preserved : Straight lines are still straight Computer vision -Image formation and Camera Calibration- (Week 9-10) (4/9) M. Hachama ([email protected])

Image formation Image formation Camera calibration Image Formation • Projective Geometry • What is lost ? : Length, Angles Computer vision -Image formation and Camera Calibration- (Week 9-10) (4/9) M. Hachama ([email protected])

Image formation Image formation Camera calibration Image Formation • Vanishing points and lines • Parallel lines in the world intersect in the image at a “vanishing point” Computer vision -Image formation and Camera Calibration- (Week 9-10) (4/9) M. Hachama ([email protected])

Image formation Image formation Camera calibration Image Formation • Vanishing points and lines • Parallel lines in the world intersect in the image at a “vanishing point” Computer vision -Image formation and Camera Calibration- (Week 9-10) (4/9) M. Hachama ([email protected])

Image formation Image formation Camera calibration Image Formation • Vanishing points and lines • Parallel planes in the world intersect in the image at a “vanishing line” Computer vision -Image formation and Camera Calibration- (Week 9-10) (4/9) M. Hachama ([email protected])

Image formation Image formation Camera calibration Camera parameters Focus and depth of field • Depth of field (DOF) : the distance between the nearest and farthest objects in a scene that appear acceptably sharp. • Although a lens can precisely focus at only one distance at a time, the decrease in sharpness is gradual on each side of the focused distance. Computer vision -Image formation and Camera Calibration- (Week 9-10) (5/9) M. Hachama ([email protected])

Image formation Image formation Camera calibration Camera parameters Focus and depth of field • How does the aperture affect the depth of field ? • A smaller aperture increases the depth of field but reduces the amount of light into the camera Computer vision -Image formation and Camera Calibration- (Week 9-10) (5/9) M. Hachama ([email protected])

Image formation Image formation Camera calibration Camera parameters Field of view • Angular measure of portion of 3d space seen by the camera • Smaller FOV = larger Focal Length Computer vision -Image formation and Camera Calibration- (Week 9-10) (5/9) M. Hachama ([email protected])

Image formation Image formation Camera calibration From World to Pixel coordinates Perspective camera model • Camera and world coordinates Computer vision -Image formation and Camera Calibration- (Week 9-10) (6/9) M. Hachama ([email protected])

Image formation Image formation Camera calibration From World to Pixel coordinates Perspective camera model • Camera and world coordinates Computer vision -Image formation and Camera Calibration- (Week 9-10) (6/9) M. Hachama ([email protected])

Image formation Image formation Camera calibration From World to Pixel coordinates Perspective camera model • From the Camera frame to the image plane • 1. Convert world point Pw to camera point Pc (Extrinsic params. [R|T]) • 2. Convert Pc to image-plane coordinates (x, y) (Intrinsic params. f , ...) • 3. Convert Pc to (discretised) pixel coordinates (u, v) Computer vision -Image formation and Camera Calibration- (Week 9-10) (6/9) M. Hachama ([email protected])

Image formation Image formation Camera calibration From World to Pixel coordinates Perspective camera model • 1. Convert world point Pw to camera point Pc    Xc Yc Zc    =    r11 r12 r13 r21 r22 r23 r31 r32 r33    R    Xw Yw Zw    +    t1 t2 t3    T    Xc Yc Zc    =    r11 r12 r13 |t1 r21 r22 r23 |t2 r31 r32 r33 |t3         Xw Yw Zw 1      = [R|T]      Xw Yw Zw 1      Computer vision -Image formation and Camera Calibration- (Week 9-10) (6/9) M. Hachama ([email protected])

Image formation Image formation Camera calibration From World to Pixel coordinates Perspective camera model • 2. Convert Pc to image-plane coordinates (x, y) • Pc = (Xc , 0, Zc )T projects to p = (x, y) onto the image plane • From similar triangles : x f = Xc Zc =⇒ x = fXc Zc • In the general case : y f = Yc Zc =⇒ y = fYc Zc Computer vision -Image formation and Camera Calibration- (Week 9-10) (6/9) M. Hachama ([email protected])

Image formation Image formation Camera calibration From World to Pixel coordinates Perspective camera model • 3. Convert Pc to (discretised) pixel coordinates (u, v) • The pixel coords of the camera optical center : O = (u0 , v0 ) • Scale factors ku , kv for the pixel-size in both dimensions u = u0 + ku x =⇒ u = u0 + ku fXc ZC , v = v0 + kv fYc ZC Computer vision -Image formation and Camera Calibration- (Week 9-10) (6/9) M. Hachama ([email protected])

Image formation Image formation Camera calibration From World to Pixel coordinates Perspective camera model • Homogeneous coordinates :    λu λv λ    =    ku f 0 u0 0 kv f v0 0 0 1       Xc Yc Zc    =    αu 0 u0 0 αv v0 0 0 1    K    Xc Yc Zc    • αu and αv are the Focal length in pixels and K is the “Calibration matrix” or “Matrix of Intrinsic Parameters”.    λu λv λ    = K[R|T]      Xw Yw Zw 1      Computer vision -Image formation and Camera Calibration- (Week 9-10) (6/9) M. Hachama ([email protected])

Image formation Image formation Camera calibration From World to Pixel coordinates Perspective camera model • Lens distortion : from the undistorted coordinates (u, v) to the observable distorted coordinates (ud , vd ) • Simple quadratic model (works for most lenses) : ud vd = (1 + k1 r2) u − u0 v − v0 + u0 v0 where r2 = (u − u0 )2 + (v − v0 )2. Computer vision -Image formation and Camera Calibration- (Week 9-10) (6/9) M. Hachama ([email protected])

Image formation Image formation Camera calibration Camera calibration Computer vision -Image formation and Camera Calibration- (Week 9-10) (7/9) M. Hachama ([email protected])

Image formation Image formation Camera calibration Direct Linear Transform (DLT) From 3D objects • Compute K, R, and T that satisfy the perspective projection equation 1 ˜ p =    ˜ u ˜ v ˜ w    = λ    u v 1    = K[R|T]      Xw Yw Zw 1      = M      Xw Yw Zw 1      =⇒    ˜ u ˜ v ˜ w    =    mT 1 mT 2 mT 3         Xw Yw Zw 1      = M      Xw Yw Zw 1      where mT i is the i-th row of M. 1. we suppose that radial distortions are already corrected Computer vision -Image formation and Camera Calibration- (Week 9-10) (8/9) M. Hachama ([email protected])

Image formation Image formation Camera calibration Direct Linear Transform (DLT) From 3D objects • For n points, we can stack all these equations into a big matrix : QM = 0 Computer vision -Image formation and Camera Calibration- (Week 9-10) (8/9) M. Hachama ([email protected])

Image formation Image formation Camera calibration Direct Linear Transform (DLT) From 3D objects • For n points, we can stack all these equations into a big matrix : QM = 0 • Minimal solution • Q(2n×12) should have rank 11 to have a unique (up to a scale) non-trivial solution M • Each 3D-to-2D point correspondence provides 2 independent equations • Thus, 5+ 2 point correspondences are needed (in practice 6 point correspondences !) Computer vision -Image formation and Camera Calibration- (Week 9-10) (8/9) M. Hachama ([email protected])

Image formation Image formation Camera calibration Direct Linear Transform (DLT) From 3D objects • For n points, we can stack all these equations into a big matrix : QM = 0 • Over-determined solution • n ≥ 6 points • A solution is to minimize QM 2 subject to the constraint M 2 = 1. =⇒ Singular Value Decomposition (SVD) : The solution is the eigenvector corresponding to the smallest eigenvalue of the matrix QT Q. Computer vision -Image formation and Camera Calibration- (Week 9-10) (8/9) M. Hachama ([email protected])

Image formation Image formation Camera calibration Direct Linear Transform (DLT) From 3D objects • For n points, we can stack all these equations into a big matrix : QM = 0 • Once we have the M matrix, we can recover the intrinsic and extrinsic parameters M = K(R|T) • We enforce the constraint that R is orthogonal (RT R = I), we can use the QR factorization Computer vision -Image formation and Camera Calibration- (Week 9-10) (8/9) M. Hachama ([email protected])

Image formation Image formation Camera calibration Direct Linear Transform (DLT) From planar grids Direct Linear Transform (DLT) From planar grids 2 • Compute K, R, and T that satisfy the perspective projection equation 1 • Use of a planar grid (e.g., a chessboard) and a few images of this shown at different orientations • We can find relations between points without knowing their 3D coordinates. 1. we suppose that radial distortions are already corrected 2. Zhang (1999) Computer vision -Image formation and Camera Calibration- (Week 9-10) (9/9) M. Hachama ([email protected])

Image formation Image formation Camera calibration Direct Linear Transform (DLT) From planar grids • Since the points lie on a plane, we have Zw = 0    λu λv λ    =    αu 0 u0 0 αv v0 0 0 1       r11 r12 r13 |t1 r21 r22 r23 |t2 r31 r32 r33 |t3         Xw Yw 0 1      =⇒    λu λv λ    =    αu 0 u0 0 αv v0 0 0 1       r11 r12 |t1 r21 r22 |t2 r31 r32 |t3    H    Xw Yw 1    H is called Homography. Computer vision -Image formation and Camera Calibration- (Week 9-10) (9/9) M. Hachama ([email protected])

Image formation Image formation Camera calibration Direct Linear Transform (DLT) From planar grids • For n points         X1 w Y 1 w 1 0 0 0 −u1 X1 w −u1 Y 1 w −u1 0 0 0 X1 w Y 1 w 1 −v1 X1 w −v1 Y 1 w −v1 . . . . . . . . . Xn w Y n w 1 0 0 0 −un Xn w −un Y n w −un 0 0 0 Xn w Y n w 1 −vn Xn w −vn Y n w −vn            h1 h2 h3    =          0 0 . . . 0 0          =⇒ Q.H = 0 Computer vision -Image formation and Camera Calibration- (Week 9-10) (9/9) M. Hachama ([email protected])

Image formation Image formation Camera calibration Direct Linear Transform (DLT) From planar grids • Minimal solution • Q2n×9 should have rank 8 to have a unique (up to a scale) non-trivial solution H • Each point correspondence provides 2 independent equations • Thus, a minimum of 4 non-collinear points is required • Over-determined solution : n ≥ 4 points • Singular Value Decomposition (SVD) • Solving for K, R and T : H decomposition    h11 h12 h13 h21 h22 h23 h31 h32 h33    =    αu 0 u0 0 αv v0 0 0 1       r11 r12 t1 r21 r22 t2 r31 r32 t3    Computer vision -Image formation and Camera Calibration- (Week 9-10) (9/9) M. Hachama ([email protected])

Image formation Image formation Camera calibration Direct Linear Transform (DLT) From planar grids • Plane-based self calibration : Zhang’s method • 1. Images I1, ...IM are taken under different views. • 2. From each i, estimate the associated homography Hi . Thus, the intrinsic and extrinsic parameters are computed. • 3. Refining all parameters (Global optimization) E = M i=1 N j=1 (ui,j − T(Pi,j W , Ri , Ti , αu , αv , u0 , v0 ))2 Computer vision -Image formation and Camera Calibration- (Week 9-10) (9/9) M. Hachama ([email protected])