[DroidCon London] Heat the neurons of your smartphone with Deep Learning

Transcript

1. Heat the Neurons of Your Smartphone with Deep Learning Qian Jin Twitter: @bonbonking | Email: qjin@xebia.fr
2. About Me !"# About This Talk '
3. Sylvain Lequeux Data Engineer @ Xebia Yoann Benoit Data Scientist @ Xebia
4. On-Device Intelligence
6. Android Wear 2.0 Smart Reply Source: https://research.googleblog.com/2017/02/on-device-machine-intelligence.html 6 Learned Projection Model
7. https://en.wikipedia.org/wiki/Moore%27s_law
8. Source: https://www.qualcomm.com/news/snapdragon/2017/01/09/tensorflow-machine-learning-now-optimized-snapdragon-835-and-hexagon-682
9. Credit: 9 Source: https://9to5google.com/2017/01/10/qualcomm-snapdragon-835-machine-learning-tensorflow/
10. The ultimate goal of the on-device intelligence is to improve mobile devices’ ability to understand the world. 10 Image inspiration credit: Moodstocks
11. #datamobile Chat History of the Slack channel
14. Magritte Ceci n’est pas une pomme.
15. René Magritte (1898-1967)
17. Build TensorFlow Android Example With Bazel
19. Android Developer Deep Learning Noob
20. NEURONS NEURONS EVERYWHERE
21. WE CAN RECOGNIZE ALL THE THINGS!
22. I THOUGHT THERE WERE MODELS FOR EVERYTHING...
23. Neural Networks in a Nutshell
24. Here’s a Neural Network 24
25. Prediction on an image - Inference 25
26. Prediction on an image - Inference 26
27. Prediction on an image - Inference 27 Apple: 0.98 Banana: 0.02
28. How to train a model?
30. Back Propagation 30
31. Back Propagation 31 Apple: 0.34 Banana: 0.66
32. Apple: 0.34 Banana: 0.66 Back Propagation 32 Prediction Error
33. Apple: 0.34 Banana: 0.66 Back Propagation 33 Prediction Error
34. Apple: 0.34 Banana: 0.66 Back Propagation 34 Prediction Error
35. Back Propagation 35 Apple: 0.87 Banana: 0.13
36. Back Propagation 36 Banana: 0.93 Apple: 0.07
37. Deep Convolutional Neural Network & Inception Architecture (Please don’t ask me any question about this part because I’ve no idea what are they talking about) Credit: http://nicolovaligi.com/history-inception-deep-learning-architecture.html
38. Network In Network, Lin et al. (2014) 38 Credit: https://arxiv.org/pdf/1312.4400v3.pdf
39. Going deeper with convolutions, Szegedy et al. (2014) Credit: https://arxiv.org/pdf/1409.4842v1.pdf 39
40. Rethinking the inception architecture for computer vision, Szegedy et al. (2015) Credit: https://arxiv.org/pdf/1512.00567v3.pdf 40
41. Deep Convolutional Neural Network 41 Image Credit: https://github.com/tensorflow/models/tree/master/research/inception Visualisation of Inception v3 Model Architecture Edges Shapes High Level Features Classifiers
42. Source: CS231n Convolutional Neural Networks for Visual Recognition http://cs231n.stanford.edu/
43. Source: https://code.facebook.com/posts/1687861518126048/facebook-to-open-source-ai-hardware-design/
44. Transfer Learning
45. Transfer Learning • Use a pre-trained Deep Neural Network • Keep all operations but the last one • Re-train only the last operation to specialize your network to your classes Keep all weights identical except these ones 45
46. Gather Training Data 46
47. 47 Retrain a Model Source: https://codelabs.developers.google.com/codelabs/tensorflow-for-poets/ python -m tensorflow/examples/image_retraining/retrain.py \ --bottleneck_dir=tf_files/bottlenecks \ --how_many_training_steps=500 \ --model_dir=tf_files/models/ \ --summaries_dir=tf_files/training_summaries/ \ --output_graph=tf_files/retrained_graph.pb \ --output_labels=tf_files/retrained_labels.txt \ --image_dir=tf_files/fruit_photos
48. Overfitting
50. Obtain the Retrained Model •2 outputs: • Model as protobuf file: contains a version of the selected network with a final layer retrained on your categories • Labels as text file 50 model.pb label.txt
51. 51 public class ClassifierActivity extends CameraActivity implements OnImageAvailableListener { private static final int INPUT_SIZE = 224; private static final int IMAGE_MEAN = 117; private static final float IMAGE_STD = 1; private static final String INPUT_NAME = "input"; private static final String OUTPUT_NAME = "output"; private static final String MODEL_FILE = "file:///android_asset/ tensorflow_inception_graph.pb"; private static final String LABEL_FILE = "file:///android_asset/ imagenet_comp_graph_label_strings.txt"; } Source: https://github.com/tensorflow/tensorflow/blob/master/tensorflow/examples/android/src/org/tensorflow/demo/ClassifierActivity.java
52. java.lang.UnsupportedOperationException: Op BatchNormWithGlobalNormalization is not available in GraphDef version 21. 52
53. Unsupported Operations • Only keep the operations dedicated to the inference step • Remove decoding, training, loss and evaluation operations 53
54. 54 Optimize for Inference Source: https://codelabs.developers.google.com/codelabs/tensorflow-for-poets/ python -m tensorflow/python/tools/optimize_for_inference \ --input=tf_files/retrained_graph.pb \ --output=tf_files/optimized_graph.pb \ --input_names="input" \ --output_names="final_result"
55. Build Standalone App
56. Pre-Google I/O 2017 • Use nightly build • Library .so • Java API jar android { //… sourceSets { main { jniLibs.srcDirs = ['libs'] } } } 56
57. Post-Google I/O 2017 Source: Android Meets TensorFlow: How to Accelerate Your App with AI (Google I/O '17) https://www.youtube.com/watch?v=25ISTLhz0ys 57 Currently: 1.4.0-rc0
59. App size ~80MB 59
60. Reducing Model Size
61. WHO CARES? MODEL SIZE 61
62. Model Size All weights are stored as they are (64-bit floats) => 80MB 62
63. ~80MB -> ~20MB 63 Weights Quantization 6.372638493746383 => 6.4 Source: https://www.tensorflow.org/performance/quantization
64. 64 Quantize Graph Source: https://codelabs.developers.google.com/codelabs/tensorflow-for-poets/ python -m tensorflow/tools/quantization/quantize_graph.py \ --input=tf_files/optimized_graph.pb \ --output=tf_files/rounded_graph.pb \ --output_node_names=final_result \ --mode=weights_rounded
65. MobileNet Mobile-first computer vision models for TensorFlow 65 Image credit : https://github.com/tensorflow/models/blob/master/research/slim/nets/mobilenet_v1.md
66. Inception V3 v.s. MobileNet 66 Inception V3 78% Accuracy* 85MB MobileNet (Largest configuration) 70.5% Accuracy* 19MB *: accuracy on ImageNet images
68. 68 Optimize for Mobile > IMAGE_SIZE=224 > ARCHITECTURE="mobilenet_0.50_${IMAGE_SIZE}"
69. 69 Optimize for Mobile Source: https://codelabs.developers.google.com/codelabs/tensorflow-for-poets/ python -m tensorflow/examples/image_retraining/retrain.py \ --bottleneck_dir=tf_files/bottlenecks \ --how_many_training_steps=500 \ --model_dir=tf_files/models/ \ --summaries_dir=tf_files/training_summaries/"${ARCHITECTURE}" \ --output_graph=tf_files/retrained_graph.pb \ --output_labels=tf_files/retrained_labels.txt \ --architecture="${ARCHITECTURE}" \ —image_dir=tf_files/fruit_photos
70. ~80Mb => ~20Mb => ~1-5Mb Source: https://research.googleblog.com/2017/06/mobilenets-open-source-models-for.html
72. Underneath the Android App
73. Android SDK (Java) Android NDK (C++) Classifier Implementation TensorFlow JNI wrapper Image (Bitmap) Trained Model top_results Classifications + Confidence input_tensor 1 2 3 4 Camera Preview Ref: https://jalammar.github.io/Supercharging-android-apps-using-tensorflow/ Overlay Display
74. Image Sampling Get Image from Camera Preview Crop the center square Resize Sample Image 74
75. Converts YUV420 (NV21) to ARGB8888 75 public static native void convertYUV420ToARGB8888( byte[] y, byte[] u, byte[] v, int[] output, int width, int height, int yRowStride, int uvRowStride, int uvPixelStride, boolean halfSize );
76. 76 /** * Initializes a native TensorFlow session for classifying images. * * @param assetManager The asset manager to be used to load assets. * @param modelFilename The filepath of the model GraphDef protocol buffer. * @param labels The list of labels * @param inputSize The input size. A square image of inputSize x inputSize is assumed. * @param imageMean The assumed mean of the image values. * @param imageStd The assumed std of the image values. * @param inputName The label of the image input node. * @param outputName The label of the output node. * @throws IOException */ public static Classifier create( AssetManager assetManager, String modelFilename, List<String> labels, int inputSize, int imageMean, float imageStd, String inputName, String outputName) { }
77. 77 @Override public List<Recognition> recognizeImage(final Bitmap bitmap) { // Preprocess bitmap bitmap.getPixels(intValues, 0, bitmap.getWidth(), 0, 0, bitmap.getWidth(), bitmap.getHeight()); for (int i = 0; i < intValues.length; ++i) { final int val = intValues[i]; floatValues[i * 3 + 0] = (((val >> 16) & 0xFF) - imageMean) / imageStd; floatValues[i * 3 + 1] = (((val >> 8) & 0xFF) - imageMean) / imageStd; floatValues[i * 3 + 2] = ((val & 0xFF) - imageMean) / imageStd; } // Copy the input data into TensorFlow. inferenceInterface.feed(inputName, floatValues, 1, inputSize, inputSize, 3); // Run the inference call. inferenceInterface.run(outputNames, logStats); // Copy the output Tensor back into the output array. inferenceInterface.fetch(outputName, outputs); (continue..) Preprocess Bitmap / Create Tensor
78. 78 @Override public List<Recognition> recognizeImage(final Bitmap bitmap) { // Preprocess bitmap bitmap.getPixels(intValues, 0, bitmap.getWidth(), 0, 0, bitmap.getWidth(), bitmap.getHeight()); for (int i = 0; i < intValues.length; ++i) { final int val = intValues[i]; floatValues[i * 3 + 0] = (((val >> 16) & 0xFF) - imageMean) / imageStd; floatValues[i * 3 + 1] = (((val >> 8) & 0xFF) - imageMean) / imageStd; floatValues[i * 3 + 2] = ((val & 0xFF) - imageMean) / imageStd; } // Copy the input data into TensorFlow. inferenceInterface.feed(inputName, floatValues, 1, inputSize, inputSize, 3); // Run the inference call. inferenceInterface.run(outputNames, logStats); // Copy the output Tensor back into the output array. inferenceInterface.fetch(outputName, outputs); (continue..) Feed Input Data to TensorFlow
79. 79 @Override public List<Recognition> recognizeImage(final Bitmap bitmap) { // Preprocess bitmap bitmap.getPixels(intValues, 0, bitmap.getWidth(), 0, 0, bitmap.getWidth(), bitmap.getHeight()); for (int i = 0; i < intValues.length; ++i) { final int val = intValues[i]; floatValues[i * 3 + 0] = (((val >> 16) & 0xFF) - imageMean) / imageStd; floatValues[i * 3 + 1] = (((val >> 8) & 0xFF) - imageMean) / imageStd; floatValues[i * 3 + 2] = ((val & 0xFF) - imageMean) / imageStd; } // Copy the input data into TensorFlow. inferenceInterface.feed(inputName, floatValues, 1, inputSize, inputSize, 3); // Run the inference call. inferenceInterface.run(outputNames, logStats); // Copy the output Tensor back into the output array. inferenceInterface.fetch(outputName, outputs); (continue..) Run the Inference Call
80. 80 @Override public List<Recognition> recognizeImage(final Bitmap bitmap) { // Preprocess bitmap bitmap.getPixels(intValues, 0, bitmap.getWidth(), 0, 0, bitmap.getWidth(), bitmap.getHeight()); for (int i = 0; i < intValues.length; ++i) { final int val = intValues[i]; floatValues[i * 3 + 0] = (((val >> 16) & 0xFF) - imageMean) / imageStd; floatValues[i * 3 + 1] = (((val >> 8) & 0xFF) - imageMean) / imageStd; floatValues[i * 3 + 2] = ((val & 0xFF) - imageMean) / imageStd; } // Copy the input data into TensorFlow. inferenceInterface.feed(inputName, floatValues, 1, inputSize, inputSize, 3); // Run the inference call. inferenceInterface.run(outputNames, logStats); // Copy the output Tensor back into the output array. inferenceInterface.fetch(outputName, outputs); (continue..) Fetch the Output Tensor
81. 81 (continue..) // Find the best classifications. PriorityQueue<Recognition> pq = new PriorityQueue<>( 3, (lhs, rhs) -> { // Intentionally reversed to put high confidence at the head of the queue. return Float.compare(rhs.getConfidence(), lhs.getConfidence()); }); for (int i = 0; i < outputs.length; ++i) { if (outputs[i] > THRESHOLD) { pq.add( new Recognition( "" + i, labels.size() > i ? labels.get(i) : "unknown", outputs[i], null)); } } //... return recognitions; } Find the Best Classification
82. Adding New Models
84. Adding a New Model 84 2 * 20 MB = 40 MB
85. Model Stacking • Start from previous model to keep all specific operations in the graph • Specify all operations to keep when optimizing for inference graph_util.convert_variables_to_constants(sess, graph.as_graph_def(), [“final_result_fruits”, “final_result_vegetables”]
86. Android Makers Paris 2017
87. 87 Source: https://www.youtube.com/watch?v=EnFyneRScQ8
89. Continuous Training Pipeline 89 Source: https://www.tensorflow.org/serving/
90. TensorFlow Serving Hosts the model provides remote access to it 90 Source: https://www.tensorflow.org/serving/
91. model.pb label.txt Continuous Training Pipeline
92. model.pb label.txt Dispensing Model
93. Currently with Project Magritte… Training • Model debug done by overheating a laptop • Model built on personal GPU • Files uploaded manually Model dispensing • API available • Deployment on AWS, currently migrating on Google Cloud 93
94. Android App Evolves
96. Android FilesDir model.pb Labels model.pb label.txt
97. public TensorFlowInferenceInterface(AssetManager assetManager, String model) { prepareNativeRuntime(); this.modelName = model; this.g = new Graph(); this.sess = new Session(g); this.runner = sess.runner(); final boolean hasAssetPrefix = model.startsWith(ASSET_FILE_PREFIX); InputStream is = null; try { String aname = hasAssetPrefix ? model.split(ASSET_FILE_PREFIX)[1] : model; is = assetManager.open(aname); } catch (IOException e) { if (hasAssetPrefix) { throw new RuntimeException("Failed to load model from '" + model + "'", e); } // Perhaps the model file is not an asset but is on disk. try { is = new FileInputStream(model); } catch (IOException e2) { throw new RuntimeException("Failed to load model from '" + model + "'", e); } } } Source: https://github.com/tensorflow/tensorflow/blob/master/tensorflow/contrib/android/java/org/tensorflow/contrib/android/TensorFlowInferenceInterface.java
98. public TensorFlowInferenceInterface(AssetManager assetManager, String model) { prepareNativeRuntime(); this.modelName = model; this.g = new Graph(); this.sess = new Session(g); this.runner = sess.runner(); final boolean hasAssetPrefix = model.startsWith(ASSET_FILE_PREFIX); InputStream is = null; try { String aname = hasAssetPrefix ? model.split(ASSET_FILE_PREFIX)[1] : model; is = assetManager.open(aname); } catch (IOException e) { if (hasAssetPrefix) { throw new RuntimeException("Failed to load model from '" + model + "'", e); } // Perhaps the model file is not an asset but is on disk. try { is = new FileInputStream(model); } catch (IOException e2) { throw new RuntimeException("Failed to load model from '" + model + "'", e); } } } Source: https://github.com/tensorflow/tensorflow/blob/master/tensorflow/contrib/android/java/org/tensorflow/contrib/android/TensorFlowInferenceInterface.java
99. public TensorFlowInferenceInterface(AssetManager assetManager, String model) { prepareNativeRuntime(); this.modelName = model; this.g = new Graph(); this.sess = new Session(g); this.runner = sess.runner(); final boolean hasAssetPrefix = model.startsWith(ASSET_FILE_PREFIX); InputStream is = null; try { String aname = hasAssetPrefix ? model.split(ASSET_FILE_PREFIX)[1] : model; is = assetManager.open(aname); } catch (IOException e) { if (hasAssetPrefix) { throw new RuntimeException("Failed to load model from '" + model + "'", e); } // Perhaps the model file is not an asset but is on disk. try { is = new FileInputStream(model); } catch (IOException e2) { throw new RuntimeException("Failed to load model from '" + model + "'", e); } } } Source: https://github.com/tensorflow/tensorflow/blob/master/tensorflow/contrib/android/java/org/tensorflow/contrib/android/TensorFlowInferenceInterface.java
100. Model Inception V3 Optimized & Quantized
101. Model MobileNets_1.0_224
102. Model MobileNets_0.5_224
103. Model MobileNets_0.25_224
104. Demo Time
107. What’s next?
108. Source: Android Meets TensorFlow: How to Accelerate Your App with AI (Google I/O '17) https://www.youtube.com/watch?v=25ISTLhz0ys
109. Source: Android Meets TensorFlow: How to Accelerate Your App with AI (Google I/O '17) https://www.youtube.com/watch?v=25ISTLhz0ys
110. Federate Learning Collaborative Machine Learning without Centralized Training Data 110 Source: https://research.googleblog.com/2017/04/federated-learning-collaborative.html
111. Resources
112. Resources • Artificial neural network: https://en.wikipedia.org/wiki/Artificial_neural_network • Deep Learning: https://en.wikipedia.org/wiki/Deep_learning • Convolutional Neural Network: https://en.wikipedia.org/wiki/Convolutional_neural_network • TensorFlow for Poets: https://codelabs.developers.google.com/codelabs/tensorflow-for-poets/ • TensorFlow for Poets 2: Optimize for Mobile: https://codelabs.developers.google.com/ codelabs/tensorflow-for-poets-2/ • TensorFlow Glossary: https://www.tensorflow.org/versions/r0.12/resources/glossary • Magritte project blog: http://blog.xebia.fr/2017/07/24/on-device-intelligence-integrez-du- deep-learning-sur-vos-smartphones/ 112
113. Thank you! Github: https://github.com/xebia-france/magritte Twitter: @bonbonking Email: qjin@xebia.fr