CV_3_Keypoints - Speaker Deck

Slide 1

Slide 1 text

Computer vision Key-points detection and description (Week 3-4) NHSM - 4th year - Spring 2025 - Prof. Mohammed Hachama hachamam@gmail.com http://hachama.github.io/home/

Slide 2

Slide 2 text

Outline Keypoints detection, description and matching Keypoints Corners Blobs detection LoG detector Harris-Laplace Scale-Invariant Feature Transform (SIFT) Geometric transformations NHSM - 4th year: Computer vision - Keypoints (Week 3-4) - M. Hachama (hachamam@gmail.com) 2/20

Slide 3

Slide 3 text

Keypoints detection, description and matching NHSM - 4th year: Computer vision - Keypoints (Week 3-4) - M. Hachama (hachamam@gmail.com) 3/20

Slide 4

Slide 4 text

Slide 5

Slide 5 text

Key-points Template matching failures • Different scales, orientations, perspectives • Different lighting conditions • Natural variability • Partial occlusion • Motion, Blur NHSM - 4th year: Computer vision - Keypoints (Week 3-4) - M. Hachama (hachamam@gmail.com) 5/20

Slide 6

Slide 6 text

Key-points What is a good Keypoint? NHSM - 4th year: Computer vision - Keypoints (Week 3-4) - M. Hachama (hachamam@gmail.com) 6/20

Slide 7

Slide 7 text

Key-points What is a good Keypoint? • Repeatability and stability • The same keypoint can be found in several images despite geometric and photometric transformations. These points usually don’t keep moving around in the image. This helps tracking. • Saliency (Uniquely identifiable) • Each keypoint is distinctive • Compactness and efficiency • Many fewer keypoints than image pixels • Locality • A keypoint occupies a relatively small area of the image; robust to clutter and occlusion NHSM - 4th year: Computer vision - Keypoints (Week 3-4) - M. Hachama (hachamam@gmail.com) 6/20

Slide 8

Slide 8 text

Key-points Applications: Panorama stitching NHSM - 4th year: Computer vision - Keypoints (Week 3-4) - M. Hachama (hachamam@gmail.com) 6/20

Slide 9

Slide 9 text

Key-points Applications: Panorama stitching • Step 1: extract keypoints NHSM - 4th year: Computer vision - Keypoints (Week 3-4) - M. Hachama (hachamam@gmail.com) 6/20

Slide 10

Slide 10 text

Key-points Applications: Panorama stitching • Step 1: extract keypoints • Step 2: match keypoint features NHSM - 4th year: Computer vision - Keypoints (Week 3-4) - M. Hachama (hachamam@gmail.com) 6/20

Slide 11

Slide 11 text

Key-points Applications: Panorama stitching • Step 1: extract keypoints • Step 2: match keypoint features • Step 3: align images NHSM - 4th year: Computer vision - Keypoints (Week 3-4) - M. Hachama (hachamam@gmail.com) 6/20

Slide 12

Slide 12 text

Key-points Applications: Matching NHSM - 4th year: Computer vision - Keypoints (Week 3-4) - M. Hachama (hachamam@gmail.com) 6/20

Slide 13

Slide 13 text

Key-points Applications: Matching NHSM - 4th year: Computer vision - Keypoints (Week 3-4) - M. Hachama (hachamam@gmail.com) 6/20

Slide 14

Slide 14 text

Key-points Applications: Matching NHSM - 4th year: Computer vision - Keypoints (Week 3-4) - M. Hachama (hachamam@gmail.com) 6/20

Slide 15

Slide 15 text

Key-points Applications: 3D reconstruction (SFM) NHSM - 4th year: Computer vision - Keypoints (Week 3-4) - M. Hachama (hachamam@gmail.com) 6/20

Slide 16

Slide 16 text

Key-points Applications: Calibration (estimating “fundamental matrix”) NHSM - 4th year: Computer vision - Keypoints (Week 3-4) - M. Hachama (hachamam@gmail.com) 6/20

Slide 17

Slide 17 text

Key-points Applications • Image alignment • Motion tracking • Robot navigation • Indexing and database retrieval • Object recognition • 3D reconstruction NHSM - 4th year: Computer vision - Keypoints (Week 3-4) - M. Hachama (hachamam@gmail.com) 6/20

Slide 18

Slide 18 text

Slide 19

Slide 19 text

Corner Detection NHSM - 4th year: Computer vision - Keypoints (Week 3-4) - M. Hachama (hachamam@gmail.com) 8/20

Slide 20

Slide 20 text

Harris Corner Detector Keypoints: Corners NHSM - 4th year: Computer vision - Keypoints (Week 3-4) - M. Hachama (hachamam@gmail.com) 9/20

Slide 21

Slide 21 text

Harris Corner Detector Keypoints: Corners detection • Based on boundary extraction • First step edge detection • Curvature analysis of edges • Based on brightness of images • Usually image derivatives • Popular technique: detector NHSM - 4th year: Computer vision - Keypoints (Week 3-4) - M. Hachama (hachamam@gmail.com) 9/20

Slide 22

Slide 22 text

Harris Corner Detector Basic Idea • Recognize the point by looking through a small window • Shifting a window in any direction should give a large change in intensity NHSM - 4th year: Computer vision - Keypoints (Week 3-4) - M. Hachama (hachamam@gmail.com) 9/20

Slide 23

Slide 23 text

Harris Corner Detector Mathematics • Change in appearance of window W for the shift [u, v]: E(u, v) = (x,y)∈W [I(x + u, y + v) − I(x, y)]2 • First-order Taylor approximation I(x + u, y + v) ≈ I(x, y) + Ix (x, y)u + Iy (x, y)v • Let’s plug this into E(u, v) E(u, v) ≈ (x,y)∈W [I(x, y) + Ix (x, y)u + Iy (x, y)v − I(x, y)]2 ≈ (x,y)∈W I2 x u2 + 2Ix Iy uv + I2 y v2 NHSM - 4th year: Computer vision - Keypoints (Week 3-4) - M. Hachama (hachamam@gmail.com) 9/20

Slide 24

Slide 24 text

Harris Corner Detector Mathematics • Change in appearance of window W for the shift [u, v]: E(u, v) ≈ u v x,y I2 x x,y Ix Iy x,y Ix Iy x,y I2 y u v • The surface E(u,v) is locally approximated by a quadratic form. • In which directions does it have the smallest/greatest change? NHSM - 4th year: Computer vision - Keypoints (Week 3-4) - M. Hachama (hachamam@gmail.com) 9/20

Slide 25

Slide 25 text

Harris Corner Detector Harris region • Matrix and eigenvalues: M = x,y I2 x x,y Ix Iy x,y Ix Iy x,y I2 y • Analysis of λ1 and λ2 , the eigenvalues of M NHSM - 4th year: Computer vision - Keypoints (Week 3-4) - M. Hachama (hachamam@gmail.com) 9/20

Slide 26

Slide 26 text

Harris Corner Detector Criteria • Harris and Stephens (1988): R = det(M) − k(trace(M))2 • |R| is small: λ1 and λ2 are small → the region is flat. • R < 0: λ1 >> λ2 or vice versa → the region is edge. • R is large: λ1 and λ2 are large and λ1 ∼ λ2 , → corner. • k is a sensitivity parameter, usually chosen in [0.04, 0.06]. • Kanade and Tomasi (1994): threshold R = min(λ1, λ2 ) • Experimentally, this score criteria was much better. • Nobel (1998): R = det(M) det(M) + ϵ NHSM - 4th year: Computer vision - Keypoints (Week 3-4) - M. Hachama (hachamam@gmail.com) 9/20

Slide 27

Slide 27 text

Harris Corner Detector Algorithm • Compute horizontal and vertical derivatives of image Ix and Iy • Compute three images corresponding to three terms in matrix M. Convolve these three images with a Gaussian filter. • Harris uses a Gaussian weighting instead: E(u, v) = (x,y)∈W w(x, y) [I(x + u, y + v) − I(x, y)]2 • Compute R(a, b) for every pixel (a, b) • Find local maxima by thresholding R and applying non maximum suppression. NHSM - 4th year: Computer vision - Keypoints (Week 3-4) - M. Hachama (hachamam@gmail.com) 9/20

Slide 28

Slide 28 text

Harris Corner Detector Example NHSM - 4th year: Computer vision - Keypoints (Week 3-4) - M. Hachama (hachamam@gmail.com) 9/20

Slide 29

Slide 29 text

Harris Corner Detector Example NHSM - 4th year: Computer vision - Keypoints (Week 3-4) - M. Hachama (hachamam@gmail.com) 9/20

Slide 30

Slide 30 text

Harris Corner Detector Example NHSM - 4th year: Computer vision - Keypoints (Week 3-4) - M. Hachama (hachamam@gmail.com) 9/20

Slide 31

Slide 31 text

Harris Corner Detector Example NHSM - 4th year: Computer vision - Keypoints (Week 3-4) - M. Hachama (hachamam@gmail.com) 9/20

Slide 32

Slide 32 text

Harris Corner Detector Invariance and covariance • Invariance: when the image is transformed, corner locations doe not change. • covariance: when the image is transformed, corner locations get transformed with the same function. NHSM - 4th year: Computer vision - Keypoints (Week 3-4) - M. Hachama (hachamam@gmail.com) 9/20

Slide 33

Slide 33 text

Harris Corner Detector Invariance and covariance • Corner location is covariant w.r.t. translation NHSM - 4th year: Computer vision - Keypoints (Week 3-4) - M. Hachama (hachamam@gmail.com) 9/20

Slide 34

Slide 34 text

Harris Corner Detector Invariance and covariance • Corner location is covariant w.r.t. rotation NHSM - 4th year: Computer vision - Keypoints (Week 3-4) - M. Hachama (hachamam@gmail.com) 9/20

Slide 35

Slide 35 text

Harris Corner Detector Invariance and covariance • Corner location is not covariant to scaling! NHSM - 4th year: Computer vision - Keypoints (Week 3-4) - M. Hachama (hachamam@gmail.com) 9/20

Slide 36

Slide 36 text

Harris Corner Detector Invariance and covariance • Affine intensity change: J = aI + b • Only derivatives are used: invariance to intensity shift J = I + b • Intensity scaling: J = aI Corner location is not covariant to scaling! NHSM - 4th year: Computer vision - Keypoints (Week 3-4) - M. Hachama (hachamam@gmail.com) 9/20

Slide 37

Slide 37 text

Slide 38

Slide 38 text

Blob detection • A Blob is a group of connected pixels in an image that share some common property (E.g grayscale value). NHSM - 4th year: Computer vision - Keypoints (Week 3-4) - M. Hachama (hachamam@gmail.com) 11/20

Slide 39

Slide 39 text

Blob detection • A simple geometric technique 1. Thresholding: Convert the source images to several binary images by thresholding the source image with thresholds: minThreshold : thresholdStep : maxThreshold 2. Grouping: Extract connected components from every binary image and calculate their centers. 3. Merging: Blobs whose centers are located closer than minDistBetweenBlobs are merged. 4. Center and Radius Calculation: The centers and radii of the new merged blobs are computed and returned. NHSM - 4th year: Computer vision - Keypoints (Week 3-4) - M. Hachama (hachamam@gmail.com) 11/20

Slide 40

Slide 40 text

Blob detection • A simple geometric technique: examples NHSM - 4th year: Computer vision - Keypoints (Week 3-4) - M. Hachama (hachamam@gmail.com) 11/20

Slide 41

Slide 41 text

Blob detection • Blobs are bright on dark or dark on bright regions in an image. • Blob = superposition of two ripples NHSM - 4th year: Computer vision - Keypoints (Week 3-4) - M. Hachama (hachamam@gmail.com) 12/20

Slide 42

Slide 42 text

Blob detection • Convolution of a blob signal with a Laplacian filter σ = 1 • Matching between the blob size and the Laplacian variance σ2. • characteristic scale - the scale that produces peak filter response NHSM - 4th year: Computer vision - Keypoints (Week 3-4) - M. Hachama (hachamam@gmail.com) 12/20

Slide 43

Slide 43 text

Blob detection • ”Scale normalized”-Laplacian of the Gaussian (LoG) L(x, y; σ) = σ2∆gσ ∗ f (x, y), gσ (x, y) = 1 2πσ2 e− x2+y2 2σ2 • σ2 is a scale normalization; without, lim σ→+∞ L(x, y; σ) = 0 • Strong positive responses for dark blobs (maximum response for σ = r/ √ 2). • Strong negative responses for bright blobs of similar size. • For a given blob, choose the right scale for the right blob size. NHSM - 4th year: Computer vision - Keypoints (Week 3-4) - M. Hachama (hachamam@gmail.com) 12/20

Slide 44

Slide 44 text

Blob detection Effect of Scale normalization NHSM - 4th year: Computer vision - Keypoints (Week 3-4) - M. Hachama (hachamam@gmail.com) 12/20

Slide 45

Slide 45 text

Blob detection • Characteristic scale - the scale that produces peak filter response NHSM - 4th year: Computer vision - Keypoints (Week 3-4) - M. Hachama (hachamam@gmail.com) 12/20

Slide 46

Slide 46 text

Blob detection • Blobs with different scales NHSM - 4th year: Computer vision - Keypoints (Week 3-4) - M. Hachama (hachamam@gmail.com) 12/20

Slide 47

Slide 47 text

Blob detection • Blobs with different scales NHSM - 4th year: Computer vision - Keypoints (Week 3-4) - M. Hachama (hachamam@gmail.com) 12/20

Slide 48

Slide 48 text

Blob detection • Optimal scale NHSM - 4th year: Computer vision - Keypoints (Week 3-4) - M. Hachama (hachamam@gmail.com) 12/20

Slide 49

Slide 49 text

Blob detection • Optimal scale NHSM - 4th year: Computer vision - Keypoints (Week 3-4) - M. Hachama (hachamam@gmail.com) 12/20

Slide 50

Slide 50 text

Slide 51

Slide 51 text

Blob detection: LoG detector1 • Convolve the input with scale-normalized LoG at several scales. • Find local maxima of squared Laplacian response in scale-space • Select (x, y, σ) that maximizes NormalizedLoG(x, y, σ). 1T. Lindeberg. Feature detection with automatic scale selection. IJCV, 1998. NHSM - 4th year: Computer vision - Keypoints (Week 3-4) - M. Hachama (hachamam@gmail.com) 14/20

Slide 52

Slide 52 text

Blob detection: LoG detector1 • Scale-space extrema are used to detect blobs of different sizes. • Laplacian of Gaussian (LoG) highlights circular structures by detecting rapid intensity changes. 1T. Lindeberg. Feature detection with automatic scale selection. IJCV, 1998. NHSM - 4th year: Computer vision - Keypoints (Week 3-4) - M. Hachama (hachamam@gmail.com) 14/20

Slide 53

Slide 53 text

Slide 54

Slide 54 text

Harris-Laplace Detector 2 • Harris-Laplace Detector: a feature detection method that combines: • Harris corner detection for spatial localization. • Laplacian of Gaussian (LoG) or Difference of Gaussians (DoG) for scale selection. → scale-invariant Harris detector. 2K.Mikolajczyk, C.Shmitd ”Indexing Based on Scale invariant Interest Points”, 2001. NHSM - 4th year: Computer vision - Keypoints (Week 3-4) - M. Hachama (hachamam@gmail.com) 16/20

Slide 55

Slide 55 text

Harris-Laplace Detector 2 • Multiscale Harris corner detection: σn = snσ0 . Local maxima in (x, y) 2K.Mikolajczyk, C.Shmitd ”Indexing Based on Scale invariant Interest Points”, 2001. NHSM - 4th year: Computer vision - Keypoints (Week 3-4) - M. Hachama (hachamam@gmail.com) 16/20

Slide 56

Slide 56 text

Harris-Laplace Detector 2 • Multiscale Harris corner detection: σn = snσ0 . • Select the scale where the LoG response is maximum. • Accept a keypoints only if its scale is a local maximum across scales. • Remove weak keypoints using a threshold on Harris and LoG. 2K.Mikolajczyk, C.Shmitd ”Indexing Based on Scale invariant Interest Points”, 2001. NHSM - 4th year: Computer vision - Keypoints (Week 3-4) - M. Hachama (hachamam@gmail.com) 16/20

Slide 57

Slide 57 text

Harris-Laplace Detector 2 • Scale selection based on Laplacian 2K.Mikolajczyk, C.Shmitd ”Indexing Based on Scale invariant Interest Points”, 2001. NHSM - 4th year: Computer vision - Keypoints (Week 3-4) - M. Hachama (hachamam@gmail.com) 16/20

Slide 58

Slide 58 text

Slide 59

Slide 59 text

SIFT Key Characteristics of SIFT • Scale Invariance: Detects features at multiple scales, allowing recognition regardless of image size. • Rotation Invariance: Features remain consistent even if the image is rotated. • Robust to Illumination Changes: Uses gradient-based descriptors, making it resistant to lighting variations. • Distinctive Features: Can match features across different images with high accuracy. NHSM - 4th year: Computer vision - Keypoints (Week 3-4) - M. Hachama (hachamam@gmail.com) 18/20

Slide 60

Slide 60 text

SIFT Basic idea • In the Scale-space, select (x, y, σ) that maximizes DoG(x, y, σ) (approx. of approximation of LoG) in all three dimensions. • Use of an efficient implementation. • Keypoints description: Definition of a feature vector for each keypoint. NHSM - 4th year: Computer vision - Keypoints (Week 3-4) - M. Hachama (hachamam@gmail.com) 18/20

Slide 61

Slide 61 text

SIFT Algorithm 1. The scale space. 2. LoG approximations: DoG. 3. Finding keypoints. 4. Getting rid of low contrast keypoints. 5. Keypoint orientations. 6. Generating a feature. NHSM - 4th year: Computer vision - Keypoints (Week 3-4) - M. Hachama (hachamam@gmail.com) 18/20

Slide 62

Slide 62 text

SIFT Algorithm • 1. Scale space: repeating generating blurred half-sized images. • Images of the same octave have same size. • Each octave has 5 images obtained by increasing the blur. NHSM - 4th year: Computer vision - Keypoints (Week 3-4) - M. Hachama (hachamam@gmail.com) 18/20

Slide 63

Slide 63 text

SIFT Algorithm • 1. Scale space: repeating generating blurred half-sized images. • Image size reduction • Reason 1: when σ increases, the kernel size needed for precise calculations increases, and so do the calculation time. We reduce the image size instead of increasing σ. • Reason 2: After the convolution with the Gauss kernel, the higher frequencies in the image spectrum are almost erased - there is no gain in keeping a higher resolution NHSM - 4th year: Computer vision - Keypoints (Week 3-4) - M. Hachama (hachamam@gmail.com) 18/20

Slide 64

Slide 64 text

SIFT Algorithm • 2. LoG approximations of the Laplacian with a difference of Gaussians DoG = G(x, y, kσ) − G(x, y, σ) ≈ (k − 1)σ2∆G(x, y) • Typical values: σ = 1.6, k = √ 2. • DoG already incorporate the σ2 normalization. NHSM - 4th year: Computer vision - Keypoints (Week 3-4) - M. Hachama (hachamam@gmail.com) 18/20

Slide 65

Slide 65 text

SIFT Algorithm • 2. LoG approximations of the Laplacian with a difference of Gaussians NHSM - 4th year: Computer vision - Keypoints (Week 3-4) - M. Hachama (hachamam@gmail.com) 18/20

Slide 66

Slide 66 text

SIFT Algorithm • 3. Finding keypoints: For each octave, iterate over the pixels. X is marked as a ”key point” if its DoG is the greatest or least of all 26 neighbors. NHSM - 4th year: Computer vision - Keypoints (Week 3-4) - M. Hachama (hachamam@gmail.com) 18/20

Slide 67

Slide 67 text

SIFT Algorithm • 4. Getting rid of low contrast keypoints • Removing low contrast features in the DoG image. • Removing edges: use the Harris corner detector. NHSM - 4th year: Computer vision - Keypoints (Week 3-4) - M. Hachama (hachamam@gmail.com) 18/20

Slide 68

Slide 68 text

SIFT Algorithm • 5. Keypoint orientations • To assign a unique orientation to circular image windows: • Create histogram of local gradient directions in the patch • Assign canonical orientation at peak of smoothed histogram • Any later calculations are done relative to this orientation. This ensures rotation invariance. NHSM - 4th year: Computer vision - Keypoints (Week 3-4) - M. Hachama (hachamam@gmail.com) 18/20

Slide 69

Slide 69 text

SIFT Algorithm • 6. Generating a feature vector for the keypoint. • A 16x16 window around the keypoint. This 16x16 window is broken into sixteen 4x4 windows. NHSM - 4th year: Computer vision - Keypoints (Week 3-4) - M. Hachama (hachamam@gmail.com) 18/20

Slide 70

Slide 70 text

SIFT Algorithm • 6. Generating a feature vector for the keypoint. • Within each 4x4 window, gradient magnitudes and orientations are calculated. These orientations are put into an 8 bin histogram. NHSM - 4th year: Computer vision - Keypoints (Week 3-4) - M. Hachama (hachamam@gmail.com) 18/20

Slide 71

Slide 71 text

SIFT Algorithm • 6. Generating a feature vector for the keypoint. • Form weighted histogram (8 bin) for 4x4 regions • Weight by magnitude and spatial Gaussian • Concatenate 16 histograms in one long vector of 128 dimensions. NHSM - 4th year: Computer vision - Keypoints (Week 3-4) - M. Hachama (hachamam@gmail.com) 18/20

Slide 72

Slide 72 text

Geometric transformations NHSM - 4th year: Computer vision - Keypoints (Week 3-4) - M. Hachama (hachamam@gmail.com) 19/20