Efficient Image Search and Identification: the Making of WINE-O.AI

Slides for my presentation at SciPy 2017 in Austin, TX. GitHub repo for this presentation: https://github.com/mlgill/scipy_2017_wine-o.ai

Transcript

1. Efficient Image Search and Identification: The Making of WINE-O.AI Michelle L. Gill, Ph.D. Senior Data Scientist, Metis @modernscientist SciPy 2017 link.mlgill.co/scipy2017
2. Metis Data Science Training • Data Science Bootcamp • 12 Week, In-Person • New York, San Francisco, Chicago, Seattle • Corporate Training • Python for Data Science • Machine Learning • Natural Language Processing • Spark • Evening Professional Development Courses • Explore Data Science Online Training thisismetis.com
3. Motivation for WINE-O.AI • Facilitate discovery and exploration of new wine • Open source, computer vision project to share with the community
4. Ingredients of WINE-O.AI Recommendation System Web Application Wine Label Identification Wine Label Identification Web Application
5. Ingredients of WINE-O.AI Recommendation System Web Application Wine Label Identification Wine Label Identification Web Application Content Based Image Retrieval
6. Content Based Image Retrieval Image Database Extracted Features
7. Content Based Image Retrieval Image Database Query Image Extracted Features Extracted Features
8. Content Based Image Retrieval Image Database Query Image Extracted Features Extracted Features Image Comparison Candidate Images
9. Content Based Image Retrieval Image Database Query Image Extracted Features Extracted Features Image Comparison Candidate Images Database Match
10. Image Comparison Challenges Query Image Database Image
11. Image Comparison Challenges • Must be robust to differences in size, rotation, occlusion, and illumination • And search must remain fast! Query Image Database Image
12. Web Application Image Selection with RANSAC Creation of Bag of Visual Words Scraped ~50,000 Wine Labels Identification of Image Features with SIFT 2 3 4 1 Content Based Image Retrieval in WINE-O.AI
13. Web Application Image Selection with RANSAC Creation of Bag of Visual Words Scraped ~50,000 Wine Labels Identification of Image Features with SIFT 2 3 4 1 2 3 4 Content Based Image Retrieval in WINE-O.AI
14. Web Application Image Selection with RANSAC Creation of Bag of Visual Words Scraped ~50,000 Wine Labels Identification of Image Features with SIFT 2 3 4 1 2 Content Based Image Retrieval in WINE-O.AI
15. SIFT Feature Detection • Scale Invariant Feature Transformation (SIFT) • Blur images using a Gaussian function of increasing width (σ) Lowe, D., IJCV, 2004 Gaussian Width (σ)
16. SIFT Feature Detection • Align images in 3D scale-space (x, y, σ) • Subtract adjacent images • Local extrema evaluated as potential features x Gaussian Blurred Images Gaussian kernel size (σ) y
17. SIFT Feature Detection • Align images in 3D scale-space (x, y, σ) • Subtract adjacent images • Local extrema evaluated as potential features x Gaussian Blurred Images Difference of Gaussians Gaussian kernel size (σ) y –
18. – – SIFT Feature Detection • Align images in 3D scale-space (x, y, σ) • Subtract adjacent images • Local extrema evaluated as potential features x Gaussian Blurred Images Difference of Gaussians Gaussian kernel size (σ) y –
19. – – SIFT Feature Detection • Align images in 3D scale-space (x, y, σ) • Subtract adjacent images • Local extrema evaluated as potential features x Gaussian Blurred Images Difference of Gaussians Gaussian kernel size (σ) y –
20. SIFT Feature Descriptor • Descriptor calculated in 16 regions around key point • Changes in intensity calculated and binned • Produces 128 dimension descriptor for each key point
21. SIFT Feature Descriptor • Descriptor calculated in 16 regions around key point • Changes in intensity calculated and binned • Produces 128 dimension descriptor for each key point
22. SIFT Feature Descriptor • Descriptor calculated in 16 regions around key point • Changes in intensity calculated and binned • Produces 128 dimension descriptor for each key point
23. SIFT Feature Descriptor • Descriptor calculated in 16 regions around key point • Changes in intensity calculated and binned • Produces 128 dimension descriptor for each key point
24. SIFT Feature Descriptor • Descriptor calculated in 16 regions around key point • Changes in intensity calculated and binned • Produces 128 dimension descriptor for each key point
25. SIFT Feature Descriptor • Descriptor calculated in 16 regions around key point • Changes in intensity calculated and binned • Produces 128 dimension descriptor for each key point
26. # image = array of grayscale, resized image pixels # Detect features detector = cv2.FeatureDetector_create('SIFT') keypoints = detector.detect(image) # Get feature descriptors descriptor = cv2.DescriptorExtractor_create('SIFT') keypoints, features = descriptor.compute(image, keypoints) # RootSIFT uses L1 norm (absolute value) descriptors /= np.sqrt(descriptors.sum()) SIFT in Practice Arandjelovic, R. and Zisserman, A., IEEE CCVPR, 2012 PyImageSearch RootSIFT Discussion OpenCV2 code has been streamlined for presentation
27. # image = array of grayscale, resized image pixels # Detect features detector = cv2.FeatureDetector_create('SIFT') keypoints = detector.detect(image) # Get feature descriptors descriptor = cv2.DescriptorExtractor_create('SIFT') keypoints, features = descriptor.compute(image, keypoints) # RootSIFT uses L1 norm (absolute value) descriptors /= np.sqrt(descriptors.sum()) SIFT in Practice Arandjelovic, R. and Zisserman, A., IEEE CCVPR, 2012 PyImageSearch RootSIFT Discussion OpenCV2 code has been streamlined for presentation Load and Process Image
28. # image = array of grayscale, resized image pixels # Detect features detector = cv2.FeatureDetector_create('SIFT') keypoints = detector.detect(image) # Get feature descriptors descriptor = cv2.DescriptorExtractor_create('SIFT') keypoints, features = descriptor.compute(image, keypoints) # RootSIFT uses L1 norm (absolute value) descriptors /= np.sqrt(descriptors.sum()) SIFT in Practice Arandjelovic, R. and Zisserman, A., IEEE CCVPR, 2012 PyImageSearch RootSIFT Discussion OpenCV2 code has been streamlined for presentation Load and Process Image Feature Detection
29. # image = array of grayscale, resized image pixels # Detect features detector = cv2.FeatureDetector_create('SIFT') keypoints = detector.detect(image) # Get feature descriptors descriptor = cv2.DescriptorExtractor_create('SIFT') keypoints, features = descriptor.compute(image, keypoints) # RootSIFT uses L1 norm (absolute value) descriptors /= np.sqrt(descriptors.sum()) SIFT in Practice Arandjelovic, R. and Zisserman, A., IEEE CCVPR, 2012 PyImageSearch RootSIFT Discussion OpenCV2 code has been streamlined for presentation Load and Process Image Feature Detection Get Feature Descriptors
30. # image = array of grayscale, resized image pixels # Detect features detector = cv2.FeatureDetector_create('SIFT') keypoints = detector.detect(image) # Get feature descriptors descriptor = cv2.DescriptorExtractor_create('SIFT') keypoints, features = descriptor.compute(image, keypoints) # RootSIFT uses L1 norm (absolute value) descriptors /= np.sqrt(descriptors.sum()) SIFT in Practice Arandjelovic, R. and Zisserman, A., IEEE CCVPR, 2012 PyImageSearch RootSIFT Discussion OpenCV2 code has been streamlined for presentation Load and Process Image Feature Detection Get Feature Descriptors Convert to RootSIFT
31. # image = array of grayscale, resized image pixels # Detect features detector = cv2.FeatureDetector_create('SIFT') keypoints = detector.detect(image) # Get feature descriptors descriptor = cv2.DescriptorExtractor_create('SIFT') keypoints, features = descriptor.compute(image, keypoints) # RootSIFT uses L1 norm (absolute value) descriptors /= np.sqrt(descriptors.sum()) SIFT in Practice Arandjelovic, R. and Zisserman, A., IEEE CCVPR, 2012 PyImageSearch RootSIFT Discussion OpenCV2 code has been streamlined for presentation Load and Process Image Feature Detection Get Feature Descriptors Convert to RootSIFT
32. Detection of Wine Label Features • Image sizes: 100 x 100 to 1000 x 1200 • Features from high resolution images did not encode well
33. Detection of Wine Label Features • Image sizes: 100 x 100 to 1000 x 1200 • Features from high resolution images did not encode well
34. Creation of Bag of Visual Words Scraped ~50,000 Wine Labels Detection of Image Features with SIFT 2 3 1 3 Wine Label Recognition: Bag of Visual Words Web Application Image Selection with RANSAC 4
35. • K-means clustering on combined features from data set • Map features for each label to nearest cluster • Creates a histogram "fingerprint" for each label Bag of Visual Words Wine Label Cluster Histogram 0 1 2 3 4
36. Bag of Visual Words Wine Label Cluster Histogram 0 2 3 1 Wine Labels Cluster ID 0 1 2 3 4
37. Bag of Visual Words Wine Label Cluster Histogram Inverted Index 0 2 3 1 Wine Labels Cluster ID 0 1 2 3 4
38. Bag of Visual Words Wine Label Cluster Histogram Inverted Index 0 2 3 1 Wine Labels Cluster ID 0 1 2 3 4
39. Bag of Visual Words Wine Label Cluster Histogram Inverted Index 0 2 3 1 Wine Labels Cluster ID 0 1 2 3 4
40. Creation of Bag of Visual Words Scraped ~50,000 Wine Labels Identification of Image Features with SIFT 2 3 1 Wine Label Selection Web Application Image Selection with RANSAC 4 4
41. • Random sample consensus (RANSAC) used to choose best candidate image • Random subsets of data fit to model • Robust to outliers but very slow Image Retrieval with RANSAC
42. • Random sample consensus (RANSAC) used to choose best candidate image • Random subsets of data fit to model • Robust to outliers but very slow Image Retrieval with RANSAC
43. • Random sample consensus (RANSAC) used to choose best candidate image • Random subsets of data fit to model • Robust to outliers but very slow Image Retrieval with RANSAC
44. # keypointsA, featuresA from query image # keypointsB, featuresB from candidate image # Pair similar features from each image d_matcher = cv2.DescriptorMatcher_create('BruteForce') matches = d_matcher.knnMatch(featuresB, featuresA, 2) # Select matched keypoints match_kpsA = keypointsA[matches[0]] match_kpsB = keypointsB[matches[1]] # Run RANSAC to calculate transformation matrix matrix, status = cv2.findHomography(match_kpsA, match_kpsB, cv2.RANSAC, 4.0) Running RANSAC OpenCV2 code has been streamlined for presentation
45. # keypointsA, featuresA from query image # keypointsB, featuresB from candidate image # Pair similar features from each image d_matcher = cv2.DescriptorMatcher_create('BruteForce') matches = d_matcher.knnMatch(featuresB, featuresA, 2) # Select matched keypoints match_kpsA = keypointsA[matches[0]] match_kpsB = keypointsB[matches[1]] # Run RANSAC to calculate transformation matrix matrix, status = cv2.findHomography(match_kpsA, match_kpsB, cv2.RANSAC, 4.0) Running RANSAC OpenCV2 code has been streamlined for presentation Features from Comparison Images
46. # keypointsA, featuresA from query image # keypointsB, featuresB from candidate image # Pair similar features from each image d_matcher = cv2.DescriptorMatcher_create('BruteForce') matches = d_matcher.knnMatch(featuresB, featuresA, 2) # Select matched keypoints match_kpsA = keypointsA[matches[0]] match_kpsB = keypointsB[matches[1]] # Run RANSAC to calculate transformation matrix matrix, status = cv2.findHomography(match_kpsA, match_kpsB, cv2.RANSAC, 4.0) Running RANSAC OpenCV2 code has been streamlined for presentation Features from Comparison Images Brute Force Feature Matching
47. # keypointsA, featuresA from query image # keypointsB, featuresB from candidate image # Pair similar features from each image d_matcher = cv2.DescriptorMatcher_create('BruteForce') matches = d_matcher.knnMatch(featuresB, featuresA, 2) # Select matched keypoints match_kpsA = keypointsA[matches[0]] match_kpsB = keypointsB[matches[1]] # Run RANSAC to calculate transformation matrix matrix, status = cv2.findHomography(match_kpsA, match_kpsB, cv2.RANSAC, 4.0) Running RANSAC OpenCV2 code has been streamlined for presentation Features from Comparison Images Brute Force Feature Matching Find Projection
48. # keypointsA, featuresA from query image # keypointsB, featuresB from candidate image # Pair similar features from each image d_matcher = cv2.DescriptorMatcher_create('BruteForce') matches = d_matcher.knnMatch(featuresB, featuresA, 2) # Select matched keypoints match_kpsA = keypointsA[matches[0]] match_kpsB = keypointsB[matches[1]] # Run RANSAC to calculate transformation matrix matrix, status = cv2.findHomography(match_kpsA, match_kpsB, cv2.RANSAC, 4.0) Running RANSAC OpenCV2 code has been streamlined for presentation Features from Comparison Images Brute Force Feature Matching Find Projection
49. # Run SIFT & calculate histogram on query image keypointsA, featuresA = run_SIFT(image) histogramA = map_features_to_clusters(featuresA) # Get candidate images with similar histograms candidate_images = get_similar_images(histogramA) for cimage in candidate_images: # Load keypointsB, featuresB for cimage # Run RANSAC on candidate image score = get_ransac_matches(keypointsA, featuresA, keypointsB, featuresB) # Matched image has best score Putting It All Together OpenCV2 code has been streamlined for presentation
50. # Run SIFT & calculate histogram on query image keypointsA, featuresA = run_SIFT(image) histogramA = map_features_to_clusters(featuresA) # Get candidate images with similar histograms candidate_images = get_similar_images(histogramA) for cimage in candidate_images: # Load keypointsB, featuresB for cimage # Run RANSAC on candidate image score = get_ransac_matches(keypointsA, featuresA, keypointsB, featuresB) # Matched image has best score Putting It All Together OpenCV2 code has been streamlined for presentation Calculate Histogram from SIFT Features
51. # Run SIFT & calculate histogram on query image keypointsA, featuresA = run_SIFT(image) histogramA = map_features_to_clusters(featuresA) # Get candidate images with similar histograms candidate_images = get_similar_images(histogramA) for cimage in candidate_images: # Load keypointsB, featuresB for cimage # Run RANSAC on candidate image score = get_ransac_matches(keypointsA, featuresA, keypointsB, featuresB) # Matched image has best score Putting It All Together OpenCV2 code has been streamlined for presentation Calculate Histogram from SIFT Features Identify Candidate Images
52. # Run SIFT & calculate histogram on query image keypointsA, featuresA = run_SIFT(image) histogramA = map_features_to_clusters(featuresA) # Get candidate images with similar histograms candidate_images = get_similar_images(histogramA) for cimage in candidate_images: # Load keypointsB, featuresB for cimage # Run RANSAC on candidate image score = get_ransac_matches(keypointsA, featuresA, keypointsB, featuresB) # Matched image has best score Putting It All Together OpenCV2 code has been streamlined for presentation Calculate Histogram from SIFT Features Identify Candidate Images Use RANSAC to Choose Image Match
53. # Run SIFT & calculate histogram on query image keypointsA, featuresA = run_SIFT(image) histogramA = map_features_to_clusters(featuresA) # Get candidate images with similar histograms candidate_images = get_similar_images(histogramA) for cimage in candidate_images: # Load keypointsB, featuresB for cimage # Run RANSAC on candidate image score = get_ransac_matches(keypointsA, featuresA, keypointsB, featuresB) # Matched image has best score Putting It All Together OpenCV2 code has been streamlined for presentation Calculate Histogram from SIFT Features Identify Candidate Images Use RANSAC to Choose Image Match
54. WINE-O.AI Demonstration Candidate Image Database Image
55. WINE-O.AI Demonstration
56. • Open source wine application • Educational use • GitHub repo: mlgill/wine-o.ai • Website: wine-o.ai Future of WINE-O.AI WINE-O.AI: Imbibe Intelligently
57. ! michelle@thisismetis.com " @modernscientist # themodernscientist $ mlgill Thank You