Master Mathematical Analysis and Applications Course M1 - S2 Computer vision Motion estimation Week 8 Mohammed Hachama [email protected] http://hachama.github.io/home/ University of Khemis Miliana -2020-

Motion Plan 1. Motion estimation Computer vision -Motion estimation- (Week 8) (2/10) M. Hachama ([email protected])

Motion Optical flow Lucas-Kanade method Horn and Schunck Global models Optical flow Computer vision -Motion estimation- (Week 8) (3/10) M. Hachama ([email protected])

Motion Optical flow Lucas-Kanade method Horn and Schunck Global models Optical flow • A video is a sequence of frames captured over time • Now our image data is a function of space (x, y) and time (t) Computer vision -Motion estimation- (Week 8) (4/10) M. Hachama ([email protected])

Motion Optical flow Lucas-Kanade method Horn and Schunck Global models Optical flow Motion Computer vision -Motion estimation- (Week 8) (4/10) M. Hachama ([email protected])

Motion Optical flow Lucas-Kanade method Horn and Schunck Global models Optical flow Applications • 3D shape reconstruction • Object segmentation • Learning and tracking of dynamical models • Event and activity recognition Computer vision -Motion estimation- (Week 8) (4/10) M. Hachama ([email protected])

Motion Optical flow Lucas-Kanade method Horn and Schunck Global models Optical flow • Optical flow is the apparent motion of brightness patterns in the image • The motion field is the projection of the 3D scene motion into the image • Apparent motion can be caused by lighting changes without any actual motion • Ideally, optical flow would be the same as the motion field Computer vision -Motion estimation- (Week 8) (4/10) M. Hachama ([email protected])

Motion Optical flow Lucas-Kanade method Horn and Schunck Global models Optical flow • Given two subsequent frames, estimate the apparent motion field u(x,y) and v(x,y) between them • Key assumptions • Brightness constancy : projection of the same point looks the same in every frame • Small motion : points do not move very far • Spatial coherence : points move like their neighbors Computer vision -Motion estimation- (Week 8) (4/10) M. Hachama ([email protected])

Motion Optical flow Lucas-Kanade method Horn and Schunck Global models Optical flow • Brightness Constancy Equation : I(x, y, t − 1) = I(x + u(x, y), y + v(x, y), t) • Linearizing the right side using Taylor expansion : I(x, y, t − 1) ≈ I(x, y, t) + Ix u(x, y) + Iy v(x, y) • Hence, Ix u + Iy v + It ≈ 0 • One equation, two unknowns per pixel • The component of the flow perpendicular to the gradient (i.e., parallel to the edge) is unknown ∇.(u, v) + It = 0 Computer vision -Motion estimation- (Week 8) (4/10) M. Hachama ([email protected])

Motion Optical flow Lucas-Kanade method Horn and Schunck Global models Optical flow Computer vision -Motion estimation- (Week 8) (4/10) M. Hachama ([email protected])

Motion Optical flow Lucas-Kanade method Horn and Schunck Global models Lucas-Kanade method Computer vision -Motion estimation- (Week 8) (5/10) M. Hachama ([email protected])

Motion Optical flow Lucas-Kanade method Horn and Schunck Global models Lucas-Kanade method Solving the aperture problem • How to get more equations for a pixel ? • Spatial coherence constraint : pretend the pixel’s neighbors have the same (u, v) • E.g., if we use a 5 × 5 window, that gives us 25 equations per pixel ∇I(xi )[u, v] + It (xi ) = 0    Ix (x1 ) Iy (x1 ) . . . Ix (xn ) Iy (xn )    u v = −    It (x1 ) . . . It (xn )    1 1. B. Lucas and T. Kanade. An iterative image registration technique with an application to stereo vision. 1981 Computer vision -Motion estimation- (Week 8) (6/10) M. Hachama ([email protected])

Motion Optical flow Lucas-Kanade method Horn and Schunck Global models Lucas-Kanade method • Linear least squares problem (over-constrained) : n×2 A 2×1 d = n×1 b • Solution given by (AT A)d = AT b Ix Ix Ix Iy Ix Iy Iy Iy u v = − Ix It Iy It • AT A should be invertible • AT A should not be too small due to noise : eigenvalues λ1 and λ2 of AT A should not be too small • AT A should be well-conditioned : λ1/λ2 should not be too large (λ2 = larger eigenvalue). Computer vision -Motion estimation- (Week 8) (6/10) M. Hachama ([email protected])

Motion Optical flow Lucas-Kanade method Horn and Schunck Global models Lucas-Kanade method Computer vision -Motion estimation- (Week 8) (6/10) M. Hachama ([email protected])

Motion Optical flow Lucas-Kanade method Horn and Schunck Global models Lucas-Kanade method Computer vision -Motion estimation- (Week 8) (6/10) M. Hachama ([email protected])

Motion Optical flow Lucas-Kanade method Horn and Schunck Global models Lucas-Kanade method Pyramids for large motion Computer vision -Motion estimation- (Week 8) (6/10) M. Hachama ([email protected])

Motion Optical flow Lucas-Kanade method Horn and Schunck Global models Lucas-Kanade method Pyramids for large motion Computer vision -Motion estimation- (Week 8) (6/10) M. Hachama ([email protected])

Motion Optical flow Lucas-Kanade method Horn and Schunck Global models Horn and Schunck Computer vision -Motion estimation- (Week 8) (7/10) M. Hachama ([email protected])

Motion Optical flow Lucas-Kanade method Horn and Schunck Global models Horn and Schunck The flow is formulated as a global energy function which is should be minimized : (Ix u + Iy v + It )2 + λ( ∇u 2 + ∇v 2) dx dy (Ix u + Iy v + It )2 + λ(u2 x + u2 y + v2 x + v2 y ) dx dy (Ix u + Iy v + It ) Ix + λ (∆u) = 0 (Ix u + Iy v + It ) Iy + λ (∆v) = 0 Computer vision -Motion estimation- (Week 8) (8/10) M. Hachama ([email protected])

Motion Optical flow Lucas-Kanade method Horn and Schunck Global models Global models Computer vision -Motion estimation- (Week 8) (9/10) M. Hachama ([email protected])

Motion Optical flow Lucas-Kanade method Horn and Schunck Global models Global (parametric) models Translational alignment • Data : Template image I0 , Reference image I1 . • Sum of squared differences (SSD) function ESSD (u) = i [I1 (xi + u) − I0 (xi )]2 = i e2 i , where u = (u, v) is the displacement • Robust error metrics ER (u) = i ρ([I1 (xi + u) − I0 (xi )) = i ρ(ei ), Computer vision -Motion estimation- (Week 8) (10/10) M. Hachama ([email protected])

Motion Optical flow Lucas-Kanade method Horn and Schunck Global models Global (parametric) models Translational alignment • Bias and gain (exposure differences) • Linear (affine) intensity variation model I1 (x + u) = (1 + α)I0 (x) + β, where β is the bias and α is the gain • Least squares formulation ESSD (u) = i [I1 (xi + u) − (1 + α)I0 (xi ) − β]2. Computer vision -Motion estimation- (Week 8) (10/10) M. Hachama ([email protected])

Motion Optical flow Lucas-Kanade method Horn and Schunck Global models Global (parametric) models Translational alignment • Correlation : maximize the product (or cross-correlation) of the two aligned images ECC (u) = i I0 (xi )I1 (xi + u) • Normalized cross-correlation is more commonly used ENCC (u) = i [I0 (xi ) − ¯ I0 )] [I1 (xi + u) − ¯ I1 ] i [I0 (xi ) − ¯ I0 )]2 [I1 (xi + u) − ¯ I1 ]2 where ¯ I0 = 1 N i I0 (xi ), ¯ I1 = 1 N i I1 (xi + u) Computer vision -Motion estimation- (Week 8) (10/10) M. Hachama ([email protected])

Motion Optical flow Lucas-Kanade method Horn and Schunck Global models Global (parametric) models Translational alignment : Optimization • Full search over some range of shifts, using either integer or sub-pixel steps (use of a hierarchical approach). • Gradient descent : Incremental refinement. Computer vision -Motion estimation- (Week 8) (10/10) M. Hachama ([email protected])

Motion Optical flow Lucas-Kanade method Horn and Schunck Global models Global (parametric) models Translational alignment : Optimization • Fourier-based alignment : (when the search range corresponds to a significant fraction of the larger image, as is the case in image stitching) F{I1 (x + u)} = F{I1 (x)}e−ju.ω = I1 (ω)e−ju.ω F{ECC (u) = F i I0 (xi )I1 (xi + u) = I0 (ω)I∗ 1 (ω), where I∗ 1 (ω) is the complex conjugate of I1 (ω). Fourier-based convolution is often used to accelerate the computation of image correlations. Computer vision -Motion estimation- (Week 8) (10/10) M. Hachama ([email protected])