Heat the Neurons of Your Smartphone with Deep Learning

[AndroidMakers 2017] TensorFlow on Android, the project Magritte is an educative application that allows people to learn a language with deep learning on-device!

Video demo here: https://www.youtube.com/watch?v=FCzrWMmrbYQ
Video for the talk: https://www.youtube.com/watch?v=znjGEW6pESQ

Transcript

1. Heat the Neurons of Your Smartphone with Deep Learning Qian Jin @bonbonking Yoann Benoit @yoannbenoit AndroidMakers Paris | 10th April 2017
2. On-Device Intelligence
3. 3 Image Credit: Google Research Blog
5. “In my 34 years in the semiconductor industry, I have witnessed the advertised death of Moore’s Law no less than four times. As we progress from 14 nanometer technology to 10 nanometer and plan for 7 nanometer and 5 nanometer and even beyond, our plans are proof that Moore’s Law is alive and well.” 5
6. 6 Ref: https://www.qualcomm.com/news/snapdragon/2017/01/09/tensorflow-machine-learning-now-optimized-snapdragon-835-and-hexagon-682
7. 7 Ref: https://9to5google.com/2017/01/10/qualcomm-snapdragon-835-machine-learning-tensorflow/
8. The ultimate goal of the on-device intelligence is to improve mobile devices’ ability to understand the world. 8
9. Magritte Ceci n’est pas une pomme.
11. #datamobile Chat History of the Slack channel 11
13. Build TensorFlow Android Example With Bazel 13
15. Android Developer Deep Learning Noob
16. NEURONS NEURONS EVERYWHERE 16
17. WE CAN RECOGNIZE ALL THE THINGS! 17
18. I THOUGHT THERE WERE MODELS FOR EVERYTHING... 18
19. Neural Networks in a Nutshell
20. Here’s a Neural Network 20
21. Prediction on an image - Inference 21
22. Prediction on an image - Inference 22
23. Prediction on an image - Inference Apple: 0.98 Banana: 0.02 23
24. Training a model
26. Training a model - Back Propagation 26
27. Training a model - Back Propagation Apple: 0.34 Banana: 0.66 27
28. Training a model - Back Propagation Apple: 0.34 Banana: 0.66 Prediction error 28
29. Training a model - Back Propagation Apple: 0.34 Banana: 0.66 Prediction error 29
30. Training a model - Back Propagation Apple: 0.34 Banana: 0.66 Prediction error 30
31. Training a model - Back Propagation Apple: 0.27 Banana: 0.73 31
32. Transfer Learning
33. Deep Convolutional Neural Network 33
34. Transfer Learning • Use a pre-trained Deep Neural Network • Keep all operations but the last one • Re-train only the last operation to specialize your network to your classes Keep all weights identical except these ones 34
35. Save the model • 2 things to save • Execution graph • Weights for each operation • 2 outputs • Model as protobuf file • Labels in text file 35 model.pb label.txt
36. java.lang.UnsupportedOperationException: Op BatchNormWithGlobalNormalization is not available in GraphDef version 21. 36
37. Unsupported Operation • Only keep the operations dedicated to the inference step • Remove decoding, training, loss and evaluation operations 37
38. Data Scientist Android Development Noob
39. CLICK 7 TIMES ON BUILD NUMBER 39
40. Build Standalone App
41. Standalone App • Use nightly build • Library .so • Java API jar android { //… sourceSets { main { jniLibs.srcDirs = ['libs'] } } } 41
42. App size ~80MB 42
43. Reducing model size
44. WHO CARES? MODEL SIZE 44
45. Model Size All weights are stored as they are (64-bit floats) => 80MB 45
46. 80 MB => 20 MB 46 Weights quantization 6.372638493746383 => 6.4
47. Architecture Underneath
48. Android SDK (Java) Android NDK (C++) Classifier Implementation TensorFlow JNI wrapper Image (Bitmap) Trained Model top_results Classifications + Confidence input_tensor 1 2 3 4 Camera Preview Ref: https://jalammar.github.io/Supercharging-android-apps-using-tensorflow/ Overlay Display 48
49. Image Sampling Get Image from Camera Preview Crop the center square Resize Sample Image 49
50. Converts YUV420 to ARGB8888 public static native void convertYUV420ToARGB8888( byte[] y, byte[] u, byte[] v, int[] output, int width, int height, int yRowStride, int uvRowStride, int uvPixelStride, boolean halfSize ); 50
51. Create Input Tensor From RGB values // Preprocess the image data from 0-255 int to normalized float based // on the provided parameters. bitmap.getPixels(intValues, 0, bitmap.getWidth(), 0, 0, bitmap.getWidth(), bitmap.getHeight()); for (int i = 0; i < intValues.length; ++i) { final int val = intValues[i]; floatValues[i * 3 + 0] = (((val >> 16) & 0xFF) - imageMean) / imageStd; floatValues[i * 3 + 1] = (((val >> 8) & 0xFF) - imageMean) / imageStd; floatValues[i * 3 + 2] = ((val & 0xFF) - imageMean) / imageStd; } inferenceInterface.feed(inputName, floatValues, 1, inputSize, inputSize, 3); 51
52. Adding new models
53. Adding a new model 53 2 * 20 MB = 40 MB
54. Model Stacking 54 • Start from previous model to keep all specific operations in the graph • Specify all operations to keep when optimizing for inference graph_util.convert_variables_to_constants(sess, graph.as_graph_def(), [“final_result_fruits”, “final_result_vegetables”]
55. Demo Time
57. What’s next? 57
58. Next: Pulling Model From the Cloud 58 Ref: https://www.youtube.com/watch?v=EnFyneRScQ8
59. Next: Federate Learning Collaborative Machine Learning without Centralized Training Data 59 Ref: https://research.googleblog.com/2017/04/federated-learning-collaborative.html
60. Thank you! Questions? Github: https://github.com/xebia-france/magritte