Heat the Neurons of Your Smartphone with Deep Learning

7b5a07956eb0b62be7214d043821a987?s=47 jinqian
April 10, 2017

Heat the Neurons of Your Smartphone with Deep Learning

[AndroidMakers 2017] TensorFlow on Android, the project Magritte is an educative application that allows people to learn a language with deep learning on-device!

Video demo here: https://www.youtube.com/watch?v=FCzrWMmrbYQ
Video for the talk: https://www.youtube.com/watch?v=znjGEW6pESQ

7b5a07956eb0b62be7214d043821a987?s=128

jinqian

April 10, 2017
Tweet

Transcript

  1. Heat the Neurons of Your Smartphone with Deep Learning Qian

    Jin @bonbonking Yoann Benoit @yoannbenoit AndroidMakers Paris | 10th April 2017
  2. On-Device Intelligence

  3. 3 Image Credit: Google Research Blog

  4. 4

  5. “In my 34 years in the semiconductor industry, I have

    witnessed the advertised death of Moore’s Law no less than four times. As we progress from 14 nanometer technology to 10 nanometer and plan for 7 nanometer and 5 nanometer and even beyond, our plans are proof that Moore’s Law is alive and well.” 5
  6. 6 Ref: https://www.qualcomm.com/news/snapdragon/2017/01/09/tensorflow-machine-learning-now-optimized-snapdragon-835-and-hexagon-682

  7. 7 Ref: https://9to5google.com/2017/01/10/qualcomm-snapdragon-835-machine-learning-tensorflow/

  8. The ultimate goal of the on-device intelligence is to improve

    mobile devices’ ability to understand the world. 8
  9. Magritte Ceci n’est pas une pomme.

  10. 10

  11. #datamobile Chat History of the Slack channel 11

  12. 12

  13. Build TensorFlow Android Example With Bazel 13

  14. 14

  15. Android Developer Deep Learning Noob

  16. NEURONS NEURONS EVERYWHERE 16

  17. WE CAN RECOGNIZE ALL THE THINGS! 17

  18. I THOUGHT THERE WERE MODELS FOR EVERYTHING... 18

  19. Neural Networks in a Nutshell

  20. Here’s a Neural Network 20

  21. Prediction on an image - Inference 21

  22. Prediction on an image - Inference 22

  23. Prediction on an image - Inference Apple: 0.98 Banana: 0.02

    23
  24. Training a model

  25. 25

  26. Training a model - Back Propagation 26

  27. Training a model - Back Propagation Apple: 0.34 Banana: 0.66

    27
  28. Training a model - Back Propagation Apple: 0.34 Banana: 0.66

    Prediction error 28
  29. Training a model - Back Propagation Apple: 0.34 Banana: 0.66

    Prediction error 29
  30. Training a model - Back Propagation Apple: 0.34 Banana: 0.66

    Prediction error 30
  31. Training a model - Back Propagation Apple: 0.27 Banana: 0.73

    31
  32. Transfer Learning

  33. Deep Convolutional Neural Network 33

  34. Transfer Learning • Use a pre-trained Deep Neural Network •

    Keep all operations but the last one • Re-train only the last operation to specialize your network to your classes Keep all weights identical except these ones 34
  35. Save the model • 2 things to save • Execution

    graph • Weights for each operation • 2 outputs • Model as protobuf file • Labels in text file 35 model.pb label.txt
  36. java.lang.UnsupportedOperationException: Op BatchNormWithGlobalNormalization is not available in GraphDef version 21.

    36
  37. Unsupported Operation • Only keep the operations dedicated to the

    inference step • Remove decoding, training, loss and evaluation operations 37
  38. Data Scientist Android Development Noob

  39. CLICK 7 TIMES ON BUILD NUMBER 39

  40. Build Standalone App

  41. Standalone App • Use nightly build • Library .so •

    Java API jar android { //… sourceSets { main { jniLibs.srcDirs = ['libs'] } } } 41
  42. App size ~80MB 42

  43. Reducing model size

  44. WHO CARES? MODEL SIZE 44

  45. Model Size All weights are stored as they are (64-bit

    floats) => 80MB 45
  46. 80 MB => 20 MB 46 Weights quantization 6.372638493746383 =>

    6.4
  47. Architecture Underneath

  48. Android SDK (Java) Android NDK (C++) Classifier Implementation TensorFlow JNI

    wrapper Image (Bitmap) Trained Model top_results Classifications + Confidence input_tensor 1 2 3 4 Camera Preview Ref: https://jalammar.github.io/Supercharging-android-apps-using-tensorflow/ Overlay Display 48
  49. Image Sampling Get Image from Camera Preview Crop the center

    square Resize Sample Image 49
  50. Converts YUV420 to ARGB8888 public static native void convertYUV420ToARGB8888( byte[]

    y, byte[] u, byte[] v, int[] output, int width, int height, int yRowStride, int uvRowStride, int uvPixelStride, boolean halfSize ); 50
  51. Create Input Tensor From RGB values // Preprocess the image

    data from 0-255 int to normalized float based // on the provided parameters. bitmap.getPixels(intValues, 0, bitmap.getWidth(), 0, 0, bitmap.getWidth(), bitmap.getHeight()); for (int i = 0; i < intValues.length; ++i) { final int val = intValues[i]; floatValues[i * 3 + 0] = (((val >> 16) & 0xFF) - imageMean) / imageStd; floatValues[i * 3 + 1] = (((val >> 8) & 0xFF) - imageMean) / imageStd; floatValues[i * 3 + 2] = ((val & 0xFF) - imageMean) / imageStd; } inferenceInterface.feed(inputName, floatValues, 1, inputSize, inputSize, 3); 51
  52. Adding new models

  53. Adding a new model 53 2 * 20 MB =

    40 MB
  54. Model Stacking 54 • Start from previous model to keep

    all specific operations in the graph • Specify all operations to keep when optimizing for inference graph_util.convert_variables_to_constants(sess, graph.as_graph_def(), [“final_result_fruits”, “final_result_vegetables”]
  55. Demo Time

  56. 56

  57. What’s next? 57

  58. Next: Pulling Model From the Cloud 58 Ref: https://www.youtube.com/watch?v=EnFyneRScQ8

  59. Next: Federate Learning Collaborative Machine Learning without Centralized Training Data

    59 Ref: https://research.googleblog.com/2017/04/federated-learning-collaborative.html
  60. Thank you! Questions? Github: https://github.com/xebia-france/magritte