Upgrade to Pro — share decks privately, control downloads, hide ads and more …

CV_3_Keypoints

Mohammed Hachama
March 01, 2025
79

 CV_3_Keypoints

Mohammed Hachama

March 01, 2025
Tweet

Transcript

  1. Computer vision Key-points detection and description (Week 3-4) NHSM -

    4th year - Spring 2025 - Prof. Mohammed Hachama [email protected] http://hachama.github.io/home/
  2. Outline Keypoints detection, description and matching Keypoints Corners Blobs detection

    LoG detector Harris-Laplace Scale-Invariant Feature Transform (SIFT) Geometric transformations NHSM - 4th year: Computer vision - Keypoints (Week 3-4) - M. Hachama ([email protected]) 2/20
  3. Outline Keypoints detection, description and matching Keypoints Corners Blobs detection

    LoG detector Harris-Laplace Scale-Invariant Feature Transform (SIFT) Geometric transformations NHSM - 4th year: Computer vision - Keypoints (Week 3-4) - M. Hachama ([email protected]) 4/20
  4. Key-points Template matching failures • Different scales, orientations, perspectives •

    Different lighting conditions • Natural variability • Partial occlusion • Motion, Blur NHSM - 4th year: Computer vision - Keypoints (Week 3-4) - M. Hachama ([email protected]) 5/20
  5. Key-points What is a good Keypoint? NHSM - 4th year:

    Computer vision - Keypoints (Week 3-4) - M. Hachama ([email protected]) 6/20
  6. Key-points What is a good Keypoint? • Repeatability and stability

    • The same keypoint can be found in several images despite geometric and photometric transformations. These points usually don’t keep moving around in the image. This helps tracking. • Saliency (Uniquely identifiable) • Each keypoint is distinctive • Compactness and efficiency • Many fewer keypoints than image pixels • Locality • A keypoint occupies a relatively small area of the image; robust to clutter and occlusion NHSM - 4th year: Computer vision - Keypoints (Week 3-4) - M. Hachama ([email protected]) 6/20
  7. Key-points Applications: Panorama stitching • Step 1: extract keypoints NHSM

    - 4th year: Computer vision - Keypoints (Week 3-4) - M. Hachama ([email protected]) 6/20
  8. Key-points Applications: Panorama stitching • Step 1: extract keypoints •

    Step 2: match keypoint features NHSM - 4th year: Computer vision - Keypoints (Week 3-4) - M. Hachama ([email protected]) 6/20
  9. Key-points Applications: Panorama stitching • Step 1: extract keypoints •

    Step 2: match keypoint features • Step 3: align images NHSM - 4th year: Computer vision - Keypoints (Week 3-4) - M. Hachama ([email protected]) 6/20
  10. Key-points Applications • Image alignment • Motion tracking • Robot

    navigation • Indexing and database retrieval • Object recognition • 3D reconstruction NHSM - 4th year: Computer vision - Keypoints (Week 3-4) - M. Hachama ([email protected]) 6/20
  11. Outline Keypoints detection, description and matching Keypoints Corners Blobs detection

    LoG detector Harris-Laplace Scale-Invariant Feature Transform (SIFT) Geometric transformations NHSM - 4th year: Computer vision - Keypoints (Week 3-4) - M. Hachama ([email protected]) 7/20
  12. Harris Corner Detector Keypoints: Corners detection • Based on boundary

    extraction • First step edge detection • Curvature analysis of edges • Based on brightness of images • Usually image derivatives • Popular technique: detector NHSM - 4th year: Computer vision - Keypoints (Week 3-4) - M. Hachama ([email protected]) 9/20
  13. Harris Corner Detector Basic Idea • Recognize the point by

    looking through a small window • Shifting a window in any direction should give a large change in intensity NHSM - 4th year: Computer vision - Keypoints (Week 3-4) - M. Hachama ([email protected]) 9/20
  14. Harris Corner Detector Mathematics • Change in appearance of window

    W for the shift [u, v]: E(u, v) = (x,y)∈W [I(x + u, y + v) − I(x, y)]2 • First-order Taylor approximation I(x + u, y + v) ≈ I(x, y) + Ix (x, y)u + Iy (x, y)v • Let’s plug this into E(u, v) E(u, v) ≈ (x,y)∈W [I(x, y) + Ix (x, y)u + Iy (x, y)v − I(x, y)]2 ≈ (x,y)∈W I2 x u2 + 2Ix Iy uv + I2 y v2 NHSM - 4th year: Computer vision - Keypoints (Week 3-4) - M. Hachama ([email protected]) 9/20
  15. Harris Corner Detector Mathematics • Change in appearance of window

    W for the shift [u, v]: E(u, v) ≈ u v x,y I2 x x,y Ix Iy x,y Ix Iy x,y I2 y u v • The surface E(u,v) is locally approximated by a quadratic form. • In which directions does it have the smallest/greatest change? NHSM - 4th year: Computer vision - Keypoints (Week 3-4) - M. Hachama ([email protected]) 9/20
  16. Harris Corner Detector Harris region • Matrix and eigenvalues: M

    = x,y I2 x x,y Ix Iy x,y Ix Iy x,y I2 y • Analysis of λ1 and λ2 , the eigenvalues of M NHSM - 4th year: Computer vision - Keypoints (Week 3-4) - M. Hachama ([email protected]) 9/20
  17. Harris Corner Detector Criteria • Harris and Stephens (1988): R

    = det(M) − k(trace(M))2 • |R| is small: λ1 and λ2 are small → the region is flat. • R < 0: λ1 >> λ2 or vice versa → the region is edge. • R is large: λ1 and λ2 are large and λ1 ∼ λ2 , → corner. • k is a sensitivity parameter, usually chosen in [0.04, 0.06]. • Kanade and Tomasi (1994): threshold R = min(λ1, λ2 ) • Experimentally, this score criteria was much better. • Nobel (1998): R = det(M) det(M) + ϵ NHSM - 4th year: Computer vision - Keypoints (Week 3-4) - M. Hachama ([email protected]) 9/20
  18. Harris Corner Detector Algorithm • Compute horizontal and vertical derivatives

    of image Ix and Iy • Compute three images corresponding to three terms in matrix M. Convolve these three images with a Gaussian filter. • Harris uses a Gaussian weighting instead: E(u, v) = (x,y)∈W w(x, y) [I(x + u, y + v) − I(x, y)]2 • Compute R(a, b) for every pixel (a, b) • Find local maxima by thresholding R and applying non maximum suppression. NHSM - 4th year: Computer vision - Keypoints (Week 3-4) - M. Hachama ([email protected]) 9/20
  19. Harris Corner Detector Invariance and covariance • Invariance: when the

    image is transformed, corner locations doe not change. • covariance: when the image is transformed, corner locations get transformed with the same function. NHSM - 4th year: Computer vision - Keypoints (Week 3-4) - M. Hachama ([email protected]) 9/20
  20. Harris Corner Detector Invariance and covariance • Corner location is

    covariant w.r.t. translation NHSM - 4th year: Computer vision - Keypoints (Week 3-4) - M. Hachama ([email protected]) 9/20
  21. Harris Corner Detector Invariance and covariance • Corner location is

    covariant w.r.t. rotation NHSM - 4th year: Computer vision - Keypoints (Week 3-4) - M. Hachama ([email protected]) 9/20
  22. Harris Corner Detector Invariance and covariance • Corner location is

    not covariant to scaling! NHSM - 4th year: Computer vision - Keypoints (Week 3-4) - M. Hachama ([email protected]) 9/20
  23. Harris Corner Detector Invariance and covariance • Affine intensity change:

    J = aI + b • Only derivatives are used: invariance to intensity shift J = I + b • Intensity scaling: J = aI Corner location is not covariant to scaling! NHSM - 4th year: Computer vision - Keypoints (Week 3-4) - M. Hachama ([email protected]) 9/20
  24. Outline Keypoints detection, description and matching Keypoints Corners Blobs detection

    LoG detector Harris-Laplace Scale-Invariant Feature Transform (SIFT) Geometric transformations NHSM - 4th year: Computer vision - Keypoints (Week 3-4) - M. Hachama ([email protected]) 10/20
  25. Blob detection • A Blob is a group of connected

    pixels in an image that share some common property (E.g grayscale value). NHSM - 4th year: Computer vision - Keypoints (Week 3-4) - M. Hachama ([email protected]) 11/20
  26. Blob detection • A simple geometric technique 1. Thresholding: Convert

    the source images to several binary images by thresholding the source image with thresholds: minThreshold : thresholdStep : maxThreshold 2. Grouping: Extract connected components from every binary image and calculate their centers. 3. Merging: Blobs whose centers are located closer than minDistBetweenBlobs are merged. 4. Center and Radius Calculation: The centers and radii of the new merged blobs are computed and returned. NHSM - 4th year: Computer vision - Keypoints (Week 3-4) - M. Hachama ([email protected]) 11/20
  27. Blob detection • A simple geometric technique: examples NHSM -

    4th year: Computer vision - Keypoints (Week 3-4) - M. Hachama ([email protected]) 11/20
  28. Blob detection • Blobs are bright on dark or dark

    on bright regions in an image. • Blob = superposition of two ripples NHSM - 4th year: Computer vision - Keypoints (Week 3-4) - M. Hachama ([email protected]) 12/20
  29. Blob detection • Convolution of a blob signal with a

    Laplacian filter σ = 1 • Matching between the blob size and the Laplacian variance σ2. • characteristic scale - the scale that produces peak filter response NHSM - 4th year: Computer vision - Keypoints (Week 3-4) - M. Hachama ([email protected]) 12/20
  30. Blob detection • ”Scale normalized”-Laplacian of the Gaussian (LoG) L(x,

    y; σ) = σ2∆gσ ∗ f (x, y), gσ (x, y) = 1 2πσ2 e− x2+y2 2σ2 • σ2 is a scale normalization; without, lim σ→+∞ L(x, y; σ) = 0 • Strong positive responses for dark blobs (maximum response for σ = r/ √ 2). • Strong negative responses for bright blobs of similar size. • For a given blob, choose the right scale for the right blob size. NHSM - 4th year: Computer vision - Keypoints (Week 3-4) - M. Hachama ([email protected]) 12/20
  31. Blob detection Effect of Scale normalization NHSM - 4th year:

    Computer vision - Keypoints (Week 3-4) - M. Hachama ([email protected]) 12/20
  32. Blob detection • Characteristic scale - the scale that produces

    peak filter response NHSM - 4th year: Computer vision - Keypoints (Week 3-4) - M. Hachama ([email protected]) 12/20
  33. Blob detection • Blobs with different scales NHSM - 4th

    year: Computer vision - Keypoints (Week 3-4) - M. Hachama ([email protected]) 12/20
  34. Blob detection • Blobs with different scales NHSM - 4th

    year: Computer vision - Keypoints (Week 3-4) - M. Hachama ([email protected]) 12/20
  35. Blob detection • Optimal scale NHSM - 4th year: Computer

    vision - Keypoints (Week 3-4) - M. Hachama ([email protected]) 12/20
  36. Blob detection • Optimal scale NHSM - 4th year: Computer

    vision - Keypoints (Week 3-4) - M. Hachama ([email protected]) 12/20
  37. Outline Keypoints detection, description and matching Keypoints Corners Blobs detection

    LoG detector Harris-Laplace Scale-Invariant Feature Transform (SIFT) Geometric transformations NHSM - 4th year: Computer vision - Keypoints (Week 3-4) - M. Hachama ([email protected]) 13/20
  38. Blob detection: LoG detector1 • Convolve the input with scale-normalized

    LoG at several scales. • Find local maxima of squared Laplacian response in scale-space • Select (x, y, σ) that maximizes NormalizedLoG(x, y, σ). 1T. Lindeberg. Feature detection with automatic scale selection. IJCV, 1998. NHSM - 4th year: Computer vision - Keypoints (Week 3-4) - M. Hachama ([email protected]) 14/20
  39. Blob detection: LoG detector1 • Scale-space extrema are used to

    detect blobs of different sizes. • Laplacian of Gaussian (LoG) highlights circular structures by detecting rapid intensity changes. 1T. Lindeberg. Feature detection with automatic scale selection. IJCV, 1998. NHSM - 4th year: Computer vision - Keypoints (Week 3-4) - M. Hachama ([email protected]) 14/20
  40. Outline Keypoints detection, description and matching Keypoints Corners Blobs detection

    LoG detector Harris-Laplace Scale-Invariant Feature Transform (SIFT) Geometric transformations NHSM - 4th year: Computer vision - Keypoints (Week 3-4) - M. Hachama ([email protected]) 15/20
  41. Harris-Laplace Detector 2 • Harris-Laplace Detector: a feature detection method

    that combines: • Harris corner detection for spatial localization. • Laplacian of Gaussian (LoG) or Difference of Gaussians (DoG) for scale selection. → scale-invariant Harris detector. 2K.Mikolajczyk, C.Shmitd ”Indexing Based on Scale invariant Interest Points”, 2001. NHSM - 4th year: Computer vision - Keypoints (Week 3-4) - M. Hachama ([email protected]) 16/20
  42. Harris-Laplace Detector 2 • Multiscale Harris corner detection: σn =

    snσ0 . Local maxima in (x, y) 2K.Mikolajczyk, C.Shmitd ”Indexing Based on Scale invariant Interest Points”, 2001. NHSM - 4th year: Computer vision - Keypoints (Week 3-4) - M. Hachama ([email protected]) 16/20
  43. Harris-Laplace Detector 2 • Multiscale Harris corner detection: σn =

    snσ0 . • Select the scale where the LoG response is maximum. • Accept a keypoints only if its scale is a local maximum across scales. • Remove weak keypoints using a threshold on Harris and LoG. 2K.Mikolajczyk, C.Shmitd ”Indexing Based on Scale invariant Interest Points”, 2001. NHSM - 4th year: Computer vision - Keypoints (Week 3-4) - M. Hachama ([email protected]) 16/20
  44. Harris-Laplace Detector 2 • Scale selection based on Laplacian 2K.Mikolajczyk,

    C.Shmitd ”Indexing Based on Scale invariant Interest Points”, 2001. NHSM - 4th year: Computer vision - Keypoints (Week 3-4) - M. Hachama ([email protected]) 16/20
  45. Outline Keypoints detection, description and matching Keypoints Corners Blobs detection

    LoG detector Harris-Laplace Scale-Invariant Feature Transform (SIFT) Geometric transformations NHSM - 4th year: Computer vision - Keypoints (Week 3-4) - M. Hachama ([email protected]) 17/20
  46. SIFT Key Characteristics of SIFT • Scale Invariance: Detects features

    at multiple scales, allowing recognition regardless of image size. • Rotation Invariance: Features remain consistent even if the image is rotated. • Robust to Illumination Changes: Uses gradient-based descriptors, making it resistant to lighting variations. • Distinctive Features: Can match features across different images with high accuracy. NHSM - 4th year: Computer vision - Keypoints (Week 3-4) - M. Hachama ([email protected]) 18/20
  47. SIFT Basic idea • In the Scale-space, select (x, y,

    σ) that maximizes DoG(x, y, σ) (approx. of approximation of LoG) in all three dimensions. • Use of an efficient implementation. • Keypoints description: Definition of a feature vector for each keypoint. NHSM - 4th year: Computer vision - Keypoints (Week 3-4) - M. Hachama ([email protected]) 18/20
  48. SIFT Algorithm 1. The scale space. 2. LoG approximations: DoG.

    3. Finding keypoints. 4. Getting rid of low contrast keypoints. 5. Keypoint orientations. 6. Generating a feature. NHSM - 4th year: Computer vision - Keypoints (Week 3-4) - M. Hachama ([email protected]) 18/20
  49. SIFT Algorithm • 1. Scale space: repeating generating blurred half-sized

    images. • Images of the same octave have same size. • Each octave has 5 images obtained by increasing the blur. NHSM - 4th year: Computer vision - Keypoints (Week 3-4) - M. Hachama ([email protected]) 18/20
  50. SIFT Algorithm • 1. Scale space: repeating generating blurred half-sized

    images. • Image size reduction • Reason 1: when σ increases, the kernel size needed for precise calculations increases, and so do the calculation time. We reduce the image size instead of increasing σ. • Reason 2: After the convolution with the Gauss kernel, the higher frequencies in the image spectrum are almost erased - there is no gain in keeping a higher resolution NHSM - 4th year: Computer vision - Keypoints (Week 3-4) - M. Hachama ([email protected]) 18/20
  51. SIFT Algorithm • 2. LoG approximations of the Laplacian with

    a difference of Gaussians DoG = G(x, y, kσ) − G(x, y, σ) ≈ (k − 1)σ2∆G(x, y) • Typical values: σ = 1.6, k = √ 2. • DoG already incorporate the σ2 normalization. NHSM - 4th year: Computer vision - Keypoints (Week 3-4) - M. Hachama ([email protected]) 18/20
  52. SIFT Algorithm • 2. LoG approximations of the Laplacian with

    a difference of Gaussians NHSM - 4th year: Computer vision - Keypoints (Week 3-4) - M. Hachama ([email protected]) 18/20
  53. SIFT Algorithm • 3. Finding keypoints: For each octave, iterate

    over the pixels. X is marked as a ”key point” if its DoG is the greatest or least of all 26 neighbors. NHSM - 4th year: Computer vision - Keypoints (Week 3-4) - M. Hachama ([email protected]) 18/20
  54. SIFT Algorithm • 4. Getting rid of low contrast keypoints

    • Removing low contrast features in the DoG image. • Removing edges: use the Harris corner detector. NHSM - 4th year: Computer vision - Keypoints (Week 3-4) - M. Hachama ([email protected]) 18/20
  55. SIFT Algorithm • 5. Keypoint orientations • To assign a

    unique orientation to circular image windows: • Create histogram of local gradient directions in the patch • Assign canonical orientation at peak of smoothed histogram • Any later calculations are done relative to this orientation. This ensures rotation invariance. NHSM - 4th year: Computer vision - Keypoints (Week 3-4) - M. Hachama ([email protected]) 18/20
  56. SIFT Algorithm • 6. Generating a feature vector for the

    keypoint. • A 16x16 window around the keypoint. This 16x16 window is broken into sixteen 4x4 windows. NHSM - 4th year: Computer vision - Keypoints (Week 3-4) - M. Hachama ([email protected]) 18/20
  57. SIFT Algorithm • 6. Generating a feature vector for the

    keypoint. • Within each 4x4 window, gradient magnitudes and orientations are calculated. These orientations are put into an 8 bin histogram. NHSM - 4th year: Computer vision - Keypoints (Week 3-4) - M. Hachama ([email protected]) 18/20
  58. SIFT Algorithm • 6. Generating a feature vector for the

    keypoint. • Form weighted histogram (8 bin) for 4x4 regions • Weight by magnitude and spatial Gaussian • Concatenate 16 histograms in one long vector of 128 dimensions. NHSM - 4th year: Computer vision - Keypoints (Week 3-4) - M. Hachama ([email protected]) 18/20
  59. Geometric transformations Transformation estimation • Exact matches: Linear systems •

    Approximate matches: Linear systems (LMS) • RANSAC NHSM - 4th year: Computer vision - Keypoints (Week 3-4) - M. Hachama ([email protected]) 20/20
  60. Geometric transformations Transformation estimation • Exact matches: Linear systems •

    Approximate matches: Linear systems (LMS) • RANSAC NHSM - 4th year: Computer vision - Keypoints (Week 3-4) - M. Hachama ([email protected]) 20/20