Pet Lens - Visual pet search in Flutter

In this presentation I go through our journey of creating an Image Visual Search app on Android using Flutter.
Made for "GDG MAD March 2019"

Transcript

1. Pet Lens Visual pet search in Flutter
2. $ whoami Name: Jawad Shaikh Title: Machine learning Engineer at difference-engine.ai Interests: Machine Learning, Web dev and Security @0xchr00t on twitter
3. Why am i here? - We wanted to explore ML on Client side - I got lazy, and couldn’t complete this app in time
4. Creating the Model
5. What is ML? - The art of creating algorithms that learn to perform tasks without explicitly being programmed to do so - In simple terms - “They are functions that map input to output. But the function body is not defined by us.” - Bunch of different algorithms come under this, we’ve used “Neural Networks”
6. What is DL? If is c i l a n
7. What is DL? If is c i l a n The h De p a n
8. Let’s state our Problem We have a collection of pet images. Given a new pet’s image, we want to search and retrieve similar looking pets from our collection.
9. Can’t we just do Classification or Object Detection? - We want to search for similar looking pets. Whether they belong to the same breed or not, does not matter. So ...
10. Ummm... Is there another way? How should we approach this problem?
12. We can use “Similarity Search” techniques - The idea is to train a model to identify similarity between 2 given images
13. Current landscape
14. Current landscape - Siamese Networks and Variants - Networks trained using Triplet Loss - Prototypical Networks
15. What’s a Siamese Network
16. What’s a Siamese Network SeResnet18 (Imagenet) SeResnet18 (Imagenet) Global Max Pooling Embedding (1, 1024) Distance Metric (1, 256) Similarity Score: 1 -> Same Whale 0 -> Different Whales
17. Ughh, wut?
18. Let’s take a simpler approach
19. What are CNNs? - Neural nets with Convolutional layers - Convolutional layers specialize in extracting features from images
20. This is what a convolution operation looks like
21. What does a CNN look like? This is one convolution filter. We have many such filters at each layer Output of the final covolution block We convert covolution output into an array of numbers called embeddings These embeddings are used by fully connected layers to produce final outputs
22. Let’s talk about the embedding layer These layers extract features from images These layers use previously extracted features to classify We assume that these embeddings represent our input image
23. Let’s talk about the embedding layer - So, the embedding layer output is a vector representation of our input - If these outputs represent our input in vector space, we can infer that -- “ Outputs generated by images with similar features, should lie close to each other in euclidean space. And those generated by images with different features, should be far apart. ”
24. How did we extract these embeddings - Created a classification model (NasNet architecture) to classify between 37 breeds NasNet (Imagenet) . . 512 . . . 37 .
25. How did we extract these embeddings - Created a classification model (NasNet architecture) to classify between 37 breeds - Sliced model uptil the embedding layer NasNet (Imagenet) . . 512 . . . 37 .
26. How did we extract these embeddings - Created a classification model (NasNet architecture) to classify between 37 breeds - Sliced model uptil the embedding layer - Ran model on entire dataset NasNet (Imagenet) . . 512 . .
27. We have the embeddings, now what? These are the steps we follow for a query image: - Calculate embedding vector for query image (let’s call it ‘Q’) - From our pre-calculated embedding vectors, search for those that are at minimum distance from Q (let’s call them set ‘R’) - Retrieve images that correspond to vectors in set ‘R’
28. We have the embeddings, now what? The question becomes “How do we efficiently find nearest embedding vectors?”
29. We used Annoy by Spotify It’s an approximate nearest neighbour search algorithm for Python
30. Building Annoy index
31. Query in python?
32. Results Query
33. Results Query
35. Flutter
36. What is it? - Cross Platform native app dev toolkit - Heavily inspired from React - Made on Dart
37. Why should you care? - Cross platform - Solid development experience (Hot Reloading!!) And if you’re really willing to put on your tinfoil hats And Flutter currently seems to be the only way to develop for it
38. Core ideas - Everything is a Widget - Widgets can have a local State - This state should define what gets displayed - Widgets should only modify there states in a setState callback
39. Let’s build something
40. ML in Flutter
42. Some motivations... - Infrastructure Cost
43. Some motivations... - Infrastructure Cost - Infrastructure Management
44. Some motivations... - Infrastructure Cost - Infrastructure Management - Network Latency
45. Some motivations... - Infrastructure Cost - Infrastructure Management - Network Latency - User Privacy
46. Our Options - mlkit - Unofficial MLKit - firebase_ml_vision - Official MLKit - tflite - Wrapper around Android’s tflite - our_monkey_patched_tflite - To extract raw network outputs Available at https://github.com/difference-engine/flutter_tflite
47. What about Annoy? - No Dart port available
48. What about Annoy? - No Dart port available - We have a Java port though!
49. What about Annoy? - No Dart port available - We have a Java port though! - And we can execute custom Java code with Flutter
50. Let’s talk about our app
51. What should it look like?
52. The Landing Screen What are the UI components? - Two custom buttons with some text and an icon - Possibly a repeating background pattern
53. The Landing Screen Wrapped in Container to apply background image SafeArea to make sure alignment happens w.r.t to app screen function that returns the button widget background image
54. The Landing Screen 2 Lines of text and 1 Icon
55. The Landing Screen (Result)
56. We need to pick Images
57. We need to pick Images
58. We need to pick Images Wrapped in GestureDetector to capture taps
59. We need to pick Images add in pubspec.yaml Using image_picker package Navigating to IntermediateScreen
60. We need to pick Images (Results)
61. We need to pick Images (Results)
62. Intermediate Screen What are the UI components? - A nice image display - A search button slightly above screen bottom
63. Intermediate Screen using PhotoView package to display selected image displaying search icon and text handling search button click
64. Intermediate Screen (Result)
65. The Search operation Here, we perform following steps - We pass our image through the model and calculate the 512d embedding array - We then pass this array to the Java code via MethodChannel, where we calculate and return similar image ids using annoy-java
66. The Search operation running tflite model on image registering MethodChannel with a unique name passing data to java code via the defined MethodChannel
67. The Search operation
68. The Search operation (Results) We get similar image ids in dart land!
69. Results Screen What are the UI components? - A nice image display - A bottom drawer that slides up to display similar pets and some metadata
70. Results Screen - The code turned out to be slightly messy ¯\_(ツ)_/¯ - So directly gonna show results, feel free to check out the code at https://github.com/difference-engine/pet_lens
71. Results Screen (Result)
72. Results Screen (Result)
73. Demo
74. What’s next ... - Flutter is still relatively new, but it's growing - Doing ML on android is definitely possible
75. Roll Credits ... Prathamesh Sarang - @prat0s Soumendra Dhanee - @dataBiryani Danish Shah - @da9ish
76. Thank you for listening!