Juan De Dios Santos Rivera - DETECTING PIKACHU ON ANDROID (AND MORE!) USING TENSORFLOW OBJECT DETECTION

TensorFlow has a component named TensorFlow Object Detection, whose purpose is to train a system capable of recognizing objects in a frame. Usually, this technology is used to detect real-life objects, until I took on the challenge of building a detection system to detect everybody's favorite Pokemon, Pikachu. What is object detection, and how I built my model? That's what this talk is about.

Transcript

1. DETECTING PIKACHU ON ANDROID (AND MORE!) USING TENSORFLOW OBJECT DETECTION PyData Amsterdam 2018 May 26, 2018 Juan De Dios Santos Rivera (@jdiossantos) Software Engineer Big Data @ LOVOO Pokemon Data Master
2. PyData Amsterdam 2018 May 26, 2018 OPENING Juan De Dios Santos Rivera (@jdiossantos) Software Engineer Big Data @ LOVOO Pokemon Data Master
3. PyData Amsterdam 2018 May 26, 2018 Juan De Dios Santos Rivera (@jdiossantos) Software Engineer Big Data @ LOVOO Pokemon Data Master Pikachu OPENING
4. PyData Amsterdam 2018 May 26, 2018 Juan De Dios Santos Rivera (@jdiossantos) Software Engineer Big Data @ LOVOO Pokemon Data Master ? ? ? ? ? ? ? ? ? ? ? ? OPENING
5. PyData Amsterdam 2018 May 26, 2018 Juan De Dios Santos Rivera (@jdiossantos) Software Engineer Big Data @ LOVOO Pokemon Data Master I wanna teach computers what a Pikachu is. OPENING
6. PyData Amsterdam 2018 May 26, 2018 Juan De Dios Santos Rivera (@jdiossantos) Software Engineer Big Data @ LOVOO Pokemon Data Master OPENING
7. PyData Amsterdam 2018 May 26, 2018 AGENDA Juan De Dios Santos Rivera (@jdiossantos) Software Engineer Big Data @ LOVOO Pokemon Data Master • OPENING • OBJECT DETECTION • TENSORFLOW OBJECT DETECTION API • BUILDING YOUR CUSTOM OBJECT DETECTION MODEL • DETECTING PIKACHU! • RESULTS • END :(
8. PyData Amsterdam 2018 May 26, 2018 OBJECT DETECTION Juan De Dios Santos Rivera (@jdiossantos) Software Engineer Big Data @ LOVOO Pokemon Data Master Process of recognizing instances of objects in images or videos
9. PyData Amsterdam 2018 May 26, 2018 OBJECT DETECTION Juan De Dios Santos Rivera (@jdiossantos) Software Engineer Big Data @ LOVOO Pokemon Data Master What’s the difference between object detection and image classification?
10. PyData Amsterdam 2018 May 26, 2018 OBJECT DETECTION Juan De Dios Santos Rivera (@jdiossantos) Software Engineer Big Data @ LOVOO Pokemon Data Master Classification Object Detection
11. PyData Amsterdam 2018 May 26, 2018 OBJECT DETECTION Juan De Dios Santos Rivera (@jdiossantos) Software Engineer Big Data @ LOVOO Pokemon Data Master In image classification the goal is to, well, classify the image into one of several categories. In detection, the goal is more granular. Here, we want to recognise a possible list of objects in the image, and for example, count the instances of them, or know where exactly it is.
12. PyData Amsterdam 2018 May 26, 2018 TENSORFLOW OBJECT DETECTION API Juan De Dios Santos Rivera (@jdiossantos) Software Engineer Big Data @ LOVOO Pokemon Data Master • Tensorflow’s answer to the detection problem • Trades accuracy for speed and memory consumption… • …so, it works on mobile! • Library includes several out-of-the-box detection architectures and feature extractors
13. BUILDING A CUSTOM OBJECT DETECTION MODEL PyData Amsterdam 2018 May 26, 2018 Juan De Dios Santos Rivera (@jdiossantos) Software Engineer Big Data @ LOVOO Pokemon Data Master
14. PyData Amsterdam 2018 May 26, 2018 INSTALLATION Juan De Dios Santos Rivera (@jdiossantos) Software Engineer Big Data @ LOVOO Pokemon Data Master Besides having TensorFlow installed, you need to clone the models repository from https:/ /github.com/ tensorflow/models
15. PyData Amsterdam 2018 May 26, 2018 CREATING DATASET Juan De Dios Santos Rivera (@jdiossantos) Software Engineer Big Data @ LOVOO Pokemon Data Master
16. PyData Amsterdam 2018 May 26, 2018 LABELING DATASET Juan De Dios Santos Rivera (@jdiossantos) Software Engineer Big Data @ LOVOO Pokemon Data Master Done using RectLabel (https:/ /rectlabel.com/)
17. PyData Amsterdam 2018 May 26, 2018 GENERATE TFRECORD Juan De Dios Santos Rivera (@jdiossantos) Software Engineer Big Data @ LOVOO Pokemon Data Master The labelled images, and the XML produced during the labelling process, must be converted to TensorFlow’s format, TFRecord* *script used for doing this is on my GitHub
18. PyData Amsterdam 2018 May 26, 2018 LABEL MAP Juan De Dios Santos Rivera (@jdiossantos) Software Engineer Big Data @ LOVOO Pokemon Data Master
19. PyData Amsterdam 2018 May 26, 2018 SO FAR… Juan De Dios Santos Rivera (@jdiossantos) Software Engineer Big Data @ LOVOO Pokemon Data Master INSTALLATION CREATION/ LABELLING DATASET TFRECORD LABEL MAP
20. PyData Amsterdam 2018 May 26, 2018 TRAINING PIPELINE Juan De Dios Santos Rivera (@jdiossantos) Software Engineer Big Data @ LOVOO Pokemon Data Master https:/ /github.com/tensorflow/models/ blob/master/research/object_detection/ g3doc/configuring_jobs.md
21. PyData Amsterdam 2018 May 26, 2018 Juan De Dios Santos Rivera (@jdiossantos) Software Engineer Big Data @ LOVOO Pokemon Data Master TRAINING
22. PyData Amsterdam 2018 May 26, 2018 TRAINING PIPELINE Juan De Dios Santos Rivera (@jdiossantos) Software Engineer Big Data @ LOVOO Pokemon Data Master …before continuing, I’d like to say a word or two about the architecture I used.
23. PyData Amsterdam 2018 May 26, 2018 SINGLE SHOT DETECTION (SSD) Juan De Dios Santos Rivera (@jdiossantos) Software Engineer Big Data @ LOVOO Pokemon Data Master SSD is the architecture I used. It is called single shot because it predicts the class of the detected object, and its location during the same shot. The opposite is an architecture that requires a second component, the proposal generator, to predict the position of the box.
24. PyData Amsterdam 2018 May 26, 2018 Juan De Dios Santos Rivera (@jdiossantos) Software Engineer Big Data @ LOVOO Pokemon Data Master “Speed/accuracy trade-offs for modern convolutional object detectors” https:/ /arxiv.org/pdf/1611.10012.pdf SINGLE SHOT DETECTION (SSD)
25. PyData Amsterdam 2018 May 26, 2018 Juan De Dios Santos Rivera (@jdiossantos) Software Engineer Big Data @ LOVOO Pokemon Data Master EVALUATION USING TENSORBOARD The library provides an evaluation script to assess the model during and after training. From all the metrics it provides, the one I used were the total loss and PASCAL VOC mAP (mean Average Precision). The results can be observed in TensorBoard.
26. PyData Amsterdam 2018 May 26, 2018 Juan De Dios Santos Rivera (@jdiossantos) Software Engineer Big Data @ LOVOO Pokemon Data Master EVALUATION USING TENSORBOARD
27. PyData Amsterdam 2018 May 26, 2018 Juan De Dios Santos Rivera (@jdiossantos) Software Engineer Big Data @ LOVOO Pokemon Data Master EVALUATION USING TENSORBOARD
28. PyData Amsterdam 2018 May 26, 2018 Juan De Dios Santos Rivera (@jdiossantos) Software Engineer Big Data @ LOVOO Pokemon Data Master EVALUATION USING TENSORBOARD
29. PyData Amsterdam 2018 May 26, 2018 Juan De Dios Santos Rivera (@jdiossantos) Software Engineer Big Data @ LOVOO Pokemon Data Master FREEZING THE MODEL Training is over! And we have our model. Now we need to freeze it (convert it into a TensorFlow Graph Proto). Fortunately, once again, the library provides a script to do this.
30. PyData Amsterdam 2018 May 26, 2018 Juan De Dios Santos Rivera (@jdiossantos) Software Engineer Big Data @ LOVOO Pokemon Data Master FREEZING THE MODEL
31. PyData Amsterdam 2018 May 26, 2018 Juan De Dios Santos Rivera (@jdiossantos) Software Engineer Big Data @ LOVOO Pokemon Data Master FREEZING THE MODEL At the end, you will have a file named “frozen_inference_graph.pb”. This is what we will use for performing the detections
32. PyData Amsterdam 2018 May 26, 2018 SO FAR… Juan De Dios Santos Rivera (@jdiossantos) Software Engineer Big Data @ LOVOO Pokemon Data Master TRAINING PIPELINE SSD MODEL
 EVALUATION FREEZING MODEL
33. USING THE MODEL PyData Amsterdam 2018 May 26, 2018 Juan De Dios Santos Rivera (@jdiossantos) Software Engineer Big Data @ LOVOO Pokemon Data Master
34. PyData Amsterdam 2018 May 26, 2018 Juan De Dios Santos Rivera (@jdiossantos) Software Engineer Big Data @ LOVOO Pokemon Data Master DETECTING PIKACHU IN IMAGES Again, the library provides a notebook with most of the code needed to perform the detections. The only thing we have to change are related to the location of the model, the number of classes and the threshold.
35. PyData Amsterdam 2018 May 26, 2018 Juan De Dios Santos Rivera (@jdiossantos) Software Engineer Big Data @ LOVOO Pokemon Data Master DETECTING PIKACHU IN IMAGES
36. PyData Amsterdam 2018 May 26, 2018 Juan De Dios Santos Rivera (@jdiossantos) Software Engineer Big Data @ LOVOO Pokemon Data Master DETECTING PIKACHU IN IMAGES
37. PyData Amsterdam 2018 May 26, 2018 Juan De Dios Santos Rivera (@jdiossantos) Software Engineer Big Data @ LOVOO Pokemon Data Master https:/ /github.com/tensorflow/tensorflow/tree/master/ tensorflow/examples/android DETECTING PIKACHU ON ANDROID
38. PyData Amsterdam 2018 May 26, 2018 Juan De Dios Santos Rivera (@jdiossantos) Software Engineer Big Data @ LOVOO Pokemon Data Master GET THE CODE Also, we are going to need Android Studio*. *all the following steps are done on Android Studio
39. PyData Amsterdam 2018 May 26, 2018 Juan De Dios Santos Rivera (@jdiossantos) Software Engineer Big Data @ LOVOO Pokemon Data Master IMPORT THE PROJECT Import a new project, and select the directory “Android” from the code we just cloned
40. PyData Amsterdam 2018 May 26, 2018 Juan De Dios Santos Rivera (@jdiossantos) Software Engineer Big Data @ LOVOO Pokemon Data Master Our goal here is to install the demo app, as it is, but with our own detection model. …BUT BEFORE THAT! IMPORT THE PROJECT
41. PyData Amsterdam 2018 May 26, 2018 Juan De Dios Santos Rivera (@jdiossantos) Software Engineer Big Data @ LOVOO Pokemon Data Master BUILDING THE PROJECT We need to build and compile the project with its dependencies using the Gradle build file. Make sure the variable ‘nativeBuildSystem’ is not set to ‘none’, otherwise, the dependencies won’t be compiled. Instead, you can use any of these options: Bazel, Makefile or CMake (this one worked for me).
42. PyData Amsterdam 2018 May 26, 2018 Juan De Dios Santos Rivera (@jdiossantos) Software Engineer Big Data @ LOVOO Pokemon Data Master BUILDING THE PROJECT Now, just click the green “play” button, select your Android device from the list of devices (connect it before, please :P), cross your fingers, and good luck.
43. PyData Amsterdam 2018 May 26, 2018 Juan De Dios Santos Rivera (@jdiossantos) Software Engineer Big Data @ LOVOO Pokemon Data Master BUILDING THE PROJECT If you are lucky, you will have the app running on your phone. However, I was not lucky.
44. PyData Amsterdam 2018 May 26, 2018 Juan De Dios Santos Rivera (@jdiossantos) Software Engineer Big Data @ LOVOO Pokemon Data Master BUILDING THE PROJECT Errors errors errors errors errors…
45. PyData Amsterdam 2018 May 26, 2018 Juan De Dios Santos Rivera (@jdiossantos) Software Engineer Big Data @ LOVOO Pokemon Data Master BUILDING THE PROJECT My solution -> somewhere deep (really deep) in the code* used for freezing the model, there is an argument named `optimize_graph` which by default it is set to True. I changed to False. * the repo changed recently.
46. PyData Amsterdam 2018 May 26, 2018 Juan De Dios Santos Rivera (@jdiossantos) Software Engineer Big Data @ LOVOO Pokemon Data Master DETECTING PIKACHU ON ANDROID
47. PyData Amsterdam 2018 May 26, 2018 Juan De Dios Santos Rivera (@jdiossantos) Software Engineer Big Data @ LOVOO Pokemon Data Master IMPROVEMENTS I was not entirely happy with the initial results. I felt like I needed more training images. Moreover, the detections on Android were quite off; cases where Pikachu was in a non- frontal position were not detected. Also, in many cases where two Pikachus were close to each other, the model detected them as one.
48. PyData Amsterdam 2018 May 26, 2018 Juan De Dios Santos Rivera (@jdiossantos) Software Engineer Big Data @ LOVOO Pokemon Data Master IMPROVEMENTS So, more images were added as the training set. This time I tried to look for a more diverse set. Then, I extended the training for another 5000 epochs. Lastly, the detection threshold from the Android app was increased to 0.85.
49. PyData Amsterdam 2018 May 26, 2018 Juan De Dios Santos Rivera (@jdiossantos) Software Engineer Big Data @ LOVOO Pokemon Data Master DETECTING PIKACHU IN VIDEOS
50. PyData Amsterdam 2018 May 26, 2018 Juan De Dios Santos Rivera (@jdiossantos) Software Engineer Big Data @ LOVOO Pokemon Data Master DETECTION IN VIDEOS https:/ /youtu.be/p_vrAFvpRUY https:/ /youtu.be/hFNaYGaca4E
51. PyData Amsterdam 2018 May 26, 2018 Juan De Dios Santos Rivera (@jdiossantos) Software Engineer Big Data @ LOVOO Pokemon Data Master RESULTS The main issues were several false negatives on cases where Pikachu was in a non-frontal position. On the Android side, a big issue is that it takes quite some seconds to get some detections. Also, the whole system in overall seems weaker that the non-Android detection counterpart.
52. PyData Amsterdam 2018 May 26, 2018 Juan De Dios Santos Rivera (@jdiossantos) Software Engineer Big Data @ LOVOO Pokemon Data Master CONCLUSION Overall, I am happy and pleased with the results. While no perfect, the system works, and it is enough to impress your friends and other Pokemon freaks. Besides this, it have been interesting to learn and play around this field of Deep Learning. Also, I am here speaking to all of you today :) which is kinda cool.
53. PyData Amsterdam 2018 May 26, 2018 Juan De Dios Santos Rivera (@jdiossantos) Software Engineer Big Data @ LOVOO Pokemon Data Master FUTURE WORK Keep improving the training set with non-frontal Pikachu images. Maybe, add a second class so it can detect another Pokemon (Charmander, maybe?).
54. PyData Amsterdam 2018 May 26, 2018 Juan De Dios Santos Rivera (@jdiossantos) Software Engineer Big Data @ LOVOO Pokemon Data Master THANKS • Twitter: @jdiossantos • Medium: medium.com/@jdiossantos/ • GitHub: https:/ /github.com/juandes/pikachu-detection • Credits: • “Unhappy Smiley Laptop” icon by Till Teenck from The Noun Project