Computer vision Motion estimation (Week 5) NHSM - 4th year - Spring 2025 - Prof. Mohammed Hachama hachamam@gmail.com http://hachama.github.io/home/

Outline Motion estimation Optical flow Lucas-Kanade method Horn and Schunck Global models NHSM - 4th year: Computer vision - Motion (Week 5) - M. Hachama (hachamam@gmail.com) 2/9

Motion NHSM - 4th year: Computer vision - Motion (Week 5) - M. Hachama (hachamam@gmail.com) 3/9

Outline Motion estimation Optical flow Lucas-Kanade method Horn and Schunck Global models NHSM - 4th year: Computer vision - Motion (Week 5) - M. Hachama (hachamam@gmail.com) 4/9

Optical flow Video = Sequence • A video is a sequence of frames captured over time • Now our image data is a function of space (x, y) and time (t) NHSM - 4th year: Computer vision - Motion (Week 5) - M. Hachama (hachamam@gmail.com) 5/9

Optical flow Motion NHSM - 4th year: Computer vision - Motion (Week 5) - M. Hachama (hachamam@gmail.com) 5/9

Optical flow Applications • 3D shape reconstruction • Object segmentation • Learning and tracking of dynamical models • Event and activity recognition NHSM - 4th year: Computer vision - Motion (Week 5) - M. Hachama (hachamam@gmail.com) 5/9

Optical flow Optical flow • Optical flow is the apparent motion of brightness patterns • The motion field is the projection of the 3D motion into the image • Apparent motion can be caused by lighting changes without any actual motion • Ideally, optical flow would be the same as the motion field NHSM - 4th year: Computer vision - Motion (Week 5) - M. Hachama (hachamam@gmail.com) 5/9

Optical flow Optical flow: Key assumptions • Given two subsequent frames, estimate the apparent motion field u(x,y) and v(x,y) between them NHSM - 4th year: Computer vision - Motion (Week 5) - M. Hachama (hachamam@gmail.com) 5/9

Optical flow Optical flow: Key assumptions • Key assumptions • 1. Brightness constancy: projection of the same point looks the same in every frame • 2. Small motion: points do not move very far • 3. Spatial coherence: points move like their neighbors NHSM - 4th year: Computer vision - Motion (Week 5) - M. Hachama (hachamam@gmail.com) 5/9

Optical flow Optical flow: fundamental equation • 1. Brightness Constancy Equation I(x, y, t − 1) = I(x + u(x, y), y + v(x, y), t) • 2. Small motion: Linearizing the right side using Taylor expansion I(x, y, t − 1) ≈ I(x, y, t) + Ix u(x, y) + Iy v(x, y) 0 ≈ It + Ix u + Iy v. • One equation, two unknowns per pixel • The component of the flow perpendicular to the gradient (i.e., parallel to the edge) is unknown (aperture problem) ∇I.(u, v) + It = 0 NHSM - 4th year: Computer vision - Motion (Week 5) - M. Hachama (hachamam@gmail.com) 5/9

Outline Motion estimation Optical flow Lucas-Kanade method Horn and Schunck Global models NHSM - 4th year: Computer vision - Motion (Week 5) - M. Hachama (hachamam@gmail.com) 6/9

Lucas-Kanade method 1 Solving the aperture problem • How to get more equations for a pixel? • 3. Spatial coherence constraint: neighbors have the same (u, v) • E.g., if we use a 5 × 5 window, that gives us 25 equations per pixel ∇I(xi )[u, v] + It (xi ) = 0    Ix (x1 ) Iy (x1 ) . . . Ix (xn ) Iy (xn )    u v = −    It (x1 ) . . . It (xn )    1B. Lucas and T. Kanade. An iterative image registration technique with an application to stereo vision. 1981 NHSM - 4th year: Computer vision - Motion (Week 5) - M. Hachama (hachamam@gmail.com) 7/9

Lucas-Kanade method 1 Solving the aperture problem • Linear least squares problem (over-constrained): n×2 A 2×1 d = n×1 b • Solution given by (AT A)d = AT b Ix Ix Ix Iy Ix Iy Iy Iy u v = − Ix It Iy It 1B. Lucas and T. Kanade. An iterative image registration technique with an application to stereo vision. 1981 NHSM - 4th year: Computer vision - Motion (Week 5) - M. Hachama (hachamam@gmail.com) 7/9

Lucas-Kanade method 1 Dealing with large motions 1B. Lucas and T. Kanade. An iterative image registration technique with an application to stereo vision. 1981 NHSM - 4th year: Computer vision - Motion (Week 5) - M. Hachama (hachamam@gmail.com) 7/9

Lucas-Kanade method 1 Dealing with large motions 1B. Lucas and T. Kanade. An iterative image registration technique with an application to stereo vision. 1981 NHSM - 4th year: Computer vision - Motion (Week 5) - M. Hachama (hachamam@gmail.com) 7/9

Lucas-Kanade method 1 Pyramids for large motion 1B. Lucas and T. Kanade. An iterative image registration technique with an application to stereo vision. 1981 NHSM - 4th year: Computer vision - Motion (Week 5) - M. Hachama (hachamam@gmail.com) 7/9

Lucas-Kanade method 1 Dealing with large motions 1B. Lucas and T. Kanade. An iterative image registration technique with an application to stereo vision. 1981 NHSM - 4th year: Computer vision - Motion (Week 5) - M. Hachama (hachamam@gmail.com) 7/9

Lucas-Kanade method 1 Dealing with large motions 1B. Lucas and T. Kanade. An iterative image registration technique with an application to stereo vision. 1981 NHSM - 4th year: Computer vision - Motion (Week 5) - M. Hachama (hachamam@gmail.com) 7/9

Outline Motion estimation Optical flow Lucas-Kanade method Horn and Schunck Global models NHSM - 4th year: Computer vision - Motion (Week 5) - M. Hachama (hachamam@gmail.com) 8/9

Horn and Schunck An energy approach The flow is formulated as a global energy function which is should be minimized: min u,v (Ix u + Iy v + It )2 + λ(∥∇u∥2+∥∇v∥2) dx dy min u,v (Ix u + Iy v + It )2 + λ(u2 x + u2 y + v2 x + v2 y ) dx dy • Euler-Lagrange (Ix u + Iy v + It ) Ix + λ (∆u) = 0 (Ix u + Iy v + It ) Iy + λ (∆v) = 0 • Discrete version: exercise. NHSM - 4th year: Computer vision - Motion (Week 5) - M. Hachama (hachamam@gmail.com) 9/9

Outline Motion estimation Optical flow Lucas-Kanade method Horn and Schunck Global models NHSM - 4th year: Computer vision - Motion (Week 5) - M. Hachama (hachamam@gmail.com) 10/9

Global (parametric) models Translational alignment • Data: Template image I0 , Reference image I1 . • Sum of squared differences (SSD) function ESSD (u) = i [I1 (xi + u) − I0 (xi )]2 = i e2 i , where u = (u, v) is the displacement • Robust error metrics ER (u) = i ρ([I1 (xi + u) − I0 (xi )) = i ρ(ei ), NHSM - 4th year: Computer vision - Motion (Week 5) - M. Hachama (hachamam@gmail.com) 11/9

Global (parametric) models Translational alignment • Bias and gain (exposure differences) • Linear (affine) intensity variation model I1 (x + u) = (1 + α)I0 (x) + β, where β is the bias and α is the gain • Least squares formulation ESSD (u) = i [I1 (xi + u) − (1 + α)I0 (xi ) − β]2. NHSM - 4th year: Computer vision - Motion (Week 5) - M. Hachama (hachamam@gmail.com) 11/9

Global (parametric) models Translational alignment • Correlation: maximize the product (or cross-correlation) of the two aligned images ECC (u) = i I0 (xi )I1 (xi + u) • Normalized cross-correlation is more commonly used ENCC (u) = i [I0 (xi ) − ¯ I0 )] [I1 (xi + u) − ¯ I1 ] i [I0 (xi ) − ¯ I0 )]2 [I1 (xi + u) − ¯ I1 ]2 where ¯ I0 = 1 N i I0 (xi ), ¯ I1 = 1 N i I1 (xi + u) NHSM - 4th year: Computer vision - Motion (Week 5) - M. Hachama (hachamam@gmail.com) 11/9

Global (parametric) models Translational alignment • Full search over some range of shifts, using either integer or sub-pixel steps (use of a hierarchical approach). • Gradient descent: Incremental refinement. NHSM - 4th year: Computer vision - Motion (Week 5) - M. Hachama (hachamam@gmail.com) 11/9

Global (parametric) models Translational alignment • Fourier-based alignment: (when the search range corresponds to a significant fraction of the larger image, as is the case in image stitching) F{I1 (x + u)} = F{I1 (x)}e−ju.ω = I1 (ω)e−ju.ω F{ECC (u) = F i I0 (xi )I1 (xi + u) = I0 (ω)I∗ 1 (ω), where I∗ 1 (ω) is the complex conjugate of I1 (ω). Fourier-based convolution is often used to accelerate the computation of image correlations. NHSM - 4th year: Computer vision - Motion (Week 5) - M. Hachama (hachamam@gmail.com) 11/9