Image ATM - Image Classification for Everyone

These are the slides for my talk at ODSC India 2019: https://confengine.com/odsc-india-2019/proposal/9350/image-atm-image-classification-for-everyone

Abstract:

At idealo.de we store and display millions of images. Our gallery contains pictures of all sorts. You’ll find there vacuum cleaners, bike helmets as well as hotel rooms. Working with huge volume of images brings some challenges: How to organize the galleries? What exactly is in there? Do we actually need all of it?

To tackle these problems you first need to label all the pictures. In 2018 our Data Science team completed four projects in the area of image classification. In 2019 there were many more to come. Therefore, we decided to automate this process by creating a software we called Image ATM (Automated Tagging Machine). With the help of transfer learning, Image ATM enables the user to train a Deep Learning model without knowledge or experience in the area of Machine Learning. All you need is data and spare couple of minutes!

In this talk we will discuss the state-of-art technologies available for image classification and present Image ATM in the context of these technologies. We will then give a crash course of our product where we will guide you through different ways of using it - in shell, on Jupyter Notebook and on the Cloud. We will also talk about our roadmap for Image ATM.

Transcript

1. Dat Tran - Head of AI @ Axel Springer AI (@datitran) 9 august 2019 ~ india ~ Image-ATM Image Classification for Everyone
2. echo $(whoami)
3. echo $(whoami)
4. https://ai.axelspringer.com/
5. echo $(whoami)
6. echo $(whoami)
7. echo $(whoami)
8. echo $(whoami) https://www.volkswagen-newsroom.com/de/pressemitteilungen/volkswagen-konzern-it-setzt-bei-digitalisierung-auf-neue-recruiting-und-arbeitsmethoden-542
9. echo $(whoami) Face2Face Hydroplaning Prediction Image Quality Assessment
10. echo $(whoami)
11. Agenda • Motivation • Image Classification Problem • Image-ATM • Conclusion
12. Motivation
13. That’s us
14. Europe‘s leading digital publisher appr. 16.000 employees worldwide over 250 brands in over 40 countries appr. 70% of the revenue is digital 49,4% of the revenue is from international business The soul and spirit of the company Axel Springer is journalism The Axel Springer Family In a nutshell
15. A family of many strong brands
16. idealo.de 16
17. perfect shopping experience 330 million offers Business Model
18. 330 million offers perfect shopping experience A lot of manual work Business Model
19. Product Gallery
21. Perfect Product Gallery
23. idealo.de - Hotel Price Comparision ~ 2.306.658 accommodations ~ 308.519.299 images ~ ~ 133 images per accommodation hotel.idealo.de
24. Importance of Photography for Hotels “.. after price, photography is the most important factor for travelers and prospects scanning OTA sites..” “.. Photography plays a role of 60% in the decision to book with a particular hotel ..” “.. study published today by TripAdvisor, it would seem like photos have the greatest impact driving engagement from travelers researching on hotel and B&B pages ..”
27. 1 2 3 4 5 6 7 8 9 10 11 12 13
28. Current Image Placement - Bedroom Position: 19 Position: 1
29. Current Image Placement - Reception Position: 17 Position: 3
30. Footer 26 April 2019 Beautiful images should appear earlier in the gallery
31. Image Quality Assessment • NVIDIA Developer Blog: https://devblogs.nvidia.com/deep- learning-hotel-aesthetics-photos/ • GitHub: https://github.com/idealo/image-q uality-assessment
32. 1 2 3 4 5 6 7 8 9 10 11 12 13
33. Ensure different areas get depicted
34. 1 2 3 4 5 6 7 8 Bedroom Bathroom Restaurant Facade Fitness Studio Kitchen
35. Ad-hoc: Winter/non-winter classification ~ Ad-hoc request to label 300k hotel images for travel department ~ Final solution: train neural network and predict 300k images
36. Problem Statement Many tagging problems • 2000 product categories • Many classes with a lot of hotel images; classes can change and ad-hoc requests such as winter/non-winter • Also a lot of image tagging problems within the Axel Springer group
37. Problem Statement Requirements • Needed a tool to do fast experimentation • Easy to use also for non-machine learners/data scientists • Good documentation • Explainable AI
38. Image Classification
39. What is Image Classification? • Image classification is the task of assigning an input image one label from a fixed set of categories: http://cs231n.github.io/classification/ • Supervised learning problem
40. Typical Example • MNIST • Fashion-MNIST • ImageNet • CIFAR-10 • Cats vs. dogs • etc...
41. How to solve it? • SVMs • Feed-forward neural networks • CNNs • CapsuleNet • SVMs • Feed-forward neural networks • CNNs • CapsuleNet
42. Transfer Learning
43. Transfer Learning 1. Use pre-trained CNN that was trained on millions of images (e.g. MobileNet or VGG16) 2. Replace top layers so that the output fits with classification task 3. Train existing and new layer weights
44. Libraries Any many more...
45. Example tf.keras
46. Example tf.keras Use ImageDataGenerator class to • Load images • Some preprocessing, target size, train/validation split etc..
47. Using the Keras Functional API • Transfer Learning with MobileNet • Input shape matches with target size; rgb • Add dropout and also dense layer with two classes Example tf.keras
48. Compile model and train it • Define loss and metrics • Define number of epochs, validation data • And many more... Example tf.keras
49. Many things can go wrong with tf.keras You can solve image classification in many ways • Sequential API vs. Functional API vs. Subclassing API • Train/test/validation split via ImageDataGenerator or scikit-learn’s model_selection • Choice of models, optimizer, loss function, metrics, number of epochs etc… • Cloud training (AWS, Google Cloud, Azure, your own GPU cluster)
50. Explainable AI Source: https://medium.com/@raeidsaqur/explainable-machine-learning-5-must-read-papers-95660d9f0c72
51. Ethics & Biases
52. Interpretability of ML models
53. Interpretability of ML models PASCAL VOC Challenge source: Wojciech Samek, “Interpretable and Trustworthy Machine Learning” Why is this a train? Why is this a horse? Why is this a boot?
54. Interpretability of ML models PASCAL VOC Challenge source: Wojciech Samek, “Interpretable and Trustworthy Machine Learning”
55. CNNs can be explainable • Attribution techniques • Visualization techniques https://github.com/idealo/cnn-exposed
56. Attribution techniques Grad-Cam Further methods are e.g. saliency maps, LRP etc...
57. Visualization techniques source: Zhou et. al., “Learning Deep Features for Discriminative Localization”
58. Image-ATM
59. Labeling Input Processing Modeling Output Deployment Highly manual Problem revisited
60. Initial idea Input Processing Modeling Output
61. Initial idea Input Processing Modeling Output
62. What is Image-ATM?
63. Installation
64. Usage • Train with CLI • Train without CLI ◦ Google Colab
66. Conclusion
67. Summary ~ Image-ATM reduced our training workflow to a couple of mins from sometimes a few hours (max 1-2 hrs) ~ Abstraction generates standardized workflow and enabled non-machine learners to do image classification ~ Library is extensively used now at idealo.de ~ Glasses, washing machines, cameras, smart phones etc. ~ Many more use cases...
68. Sneaker Gallery
69. Sneaker Gallery
70. Roadmap ~ Upgrade to TensorFlow 2.0 ~ Add AutoML capabilities (at the moment ENAS takes 8hrs on MNIST for a good model whereas transfer learning takes 20mins) ~ More interpretable AI techniques (SHAP etc..) ~ PDF report output ~ More image augmentation ~ Semi-supervised learning ~ Image Deduplication (coming soon)
71. Team Christopher Lennan Senior Data Scientist idealo.de Malgorzata Adamczyk Machine Learning Engineer Axel Springer AI Gunar Maiwald Data Scientist idealo.de Dat Tran Head of AI Axel Springer AI
72. Contributors Check out our repo and contribute: https://github.com/idealo/imageatm
73. Url: www.dat-tran.com Twitter: @datitran Questions?