Faites chauffer les neurones de votre smartphone avec du deep learning on-device

7b5a07956eb0b62be7214d043821a987?s=47 jinqian
October 20, 2017

Faites chauffer les neurones de votre smartphone avec du deep learning on-device

Nous entendons aujourd'hui parler de Deep Learning un peu partout : reconnaissance d'images, de sons, génération de texte, etc. Les annonces sur Android Neural Network API et TensorFlowLite, la release du framework CoreML d’Apple, toutes ces actions nous confirment la tendance d’aller plus loins sur l’on-device intelligence. Mais, bien que les techniques et frameworks soient en train de se démocratiser, il reste difficile d'en voir des applications concrètes en entreprise, et encore moins sur des applications mobiles. Nous avons donc décidé de construire un Proof Of Concept pour relever les défis du domaine. A travers une application mobile à but éducatif utilisant du Deep Learning pour de la reconnaissance d’objets, nous aborderons les questions des impacts de ce type de modèle sur les smartphones, d'architecture pour l'entraînement et le déploiement de modèles sur le service Cloud, ainsi que la construction de l’application mobile avec les dernières nouveautés annoncées.

7b5a07956eb0b62be7214d043821a987?s=128

jinqian

October 20, 2017
Tweet

Transcript

  1. 1.

    Faites chauffer les neurones de votre smartphone avec du deep

    learning on-device Qian Jin, aka "SpeedRabbit" Yoann Benoit, aka "TensorMan" Sylvain Lequeux, aka "TeddyBière" @bonbonking @YoannBENOIT @slequeux
  2. 4.
  3. 5.
  4. 8.

    The ultimate goal of the on-device intelligence is to improve

    mobile devices’ ability to understand the world.
  5. 10.

    10

  6. 11.
  7. 13.

    13

  8. 15.

    15

  9. 21.
  10. 22.
  11. 23.
  12. 26.

    26

  13. 27.
  14. 34.
  15. 35.

    Keep all weights identical except these ones Use a pre-trained

    Deep Neural Network Keep all operations but the last one Re-train only the last operation to specialize your network to your classes
  16. 36.

    Creating the final step in transfer learning with tf.name_scope("input"): bottleneck_input

    = tf.placeholder_with_default(bottleneck_tensor, shape=[None, bottleneck_size]) with tf.name_scope("intermediate_training_ops"): layer_weights = tf.Variable(tf.truncated_normal([bottleneck_size, 500], stddev=0.001)) layer_biases = tf.Variable(tf.zeros([500])) hidden = tf.nn.relu(tf.matmul(bottleneck_input, layer_weights) + layer_biases) dropout = tf.nn.dropout(hidden, keep_prob=keep_prob) with tf.name_scope("final_training_ops"): layer_weights = tf.Variable(tf.truncated_normal([500, class_count], stddev=0.001)) layer_biases = tf.Variable(tf.zeros([class_count])) logits = tf.matmul(dropout, layer_weights) + layer_biases final_tensor = tf.nn.softmax(logits)
  17. 37.

    Two things to save • Execution graph • Weights for

    each operation Two outputs • Model as protobuf file • Labels in text files with gfile.FastGFile(FLAGS.output_graph_path, 'wb') as f: f.write(output_graph_def.SerializeToString()) model.pb label.txt
  18. 39.

    Unsupported operation Only keep the operations dedicated to the inference

    step Remove decoding, training, loss and evaluation operations
  19. 43.

    Use nightly build • Library.so • Java API Jar Pre-Google

    I/O android { //… sourceSets { main { jniLibs.srcDirs = ['libs'] } } }
  20. 45.
  21. 55.
  22. 57.

    Android SDK (Java) Android NDK (C++) Classifier Implementation TensorFlow JNI

    wrapper Image (Bitmap) Trained Model top_results Classifications + Confidence input_tensor 1 2 3 4 Camera Preview Ref: https://jalammar.github.io/Supercharging-android-apps-using-tensorflow/ Overlay Display 57
  23. 58.

    Image sampling on ANdroid device Get Image from Camera Preview

    Crop the center square Resize Sample Image
  24. 59.

    Converts YUV420 to ARGB8888 public static native void convertYUV420ToARGB8888 (

    byte[] y, byte[] u, byte[] v, int[] output, int width, int height, int yRowStride, int uvRowStride, int uvPixelStride, boolean halfSize );
  25. 60.

    Steps of Recognizing Image @Override public List<Recognition> recognizeImage(final Bitmap bitmap)

    { // Preprocess bitmap bitmap.getPixels(intValues, 0, bitmap.getWidth(), 0, 0, bitmap.getWidth(), bitmap.getHeight()); for (int i = 0; i < intValues.length; ++i) { final int val = intValues[i]; floatValues[i * 3 + 0] = (((val >> 16) & 0xFF) - imageMean) / imageStd; floatValues[i * 3 + 1] = (((val >> 8) & 0xFF) - imageMean) / imageStd; floatValues[i * 3 + 2] = ((val & 0xFF) - imageMean) / imageStd; } // Copy the input data into TensorFlow. inferenceInterface.feed(inputName, floatValues, 1, inputSize, inputSize, 3); // Run the inference call. inferenceInterface.run(outputNames, logStats); // Copy the output Tensor back into the output array. inferenceInterface.fetch(outputName, outputs); (continue..)
  26. 61.

    (continue..) // Find the best classifications. PriorityQueue<Recognition> pq = new

    PriorityQueue<>( 3, (lhs, rhs) -> { // Intentionally reversed to put high confidence at the head of the queue. return Float.compare(rhs.getConfidence(), lhs.getConfidence()); }); for (int i = 0; i < outputs.length; ++i) { if (outputs[i] > THRESHOLD) { pq.add( new Recognition( "" + i, labels.size() > i ? labels.get(i) : "unknown", outputs[i], null)); } } //... return recognitions; } Steps of Recognizing Image
  27. 65.

    Start from previous model to keep all specific operations in

    the graph Specify all operations to keep when optimizing for inference Model stacking graph_util.convert_variables_to_constants(sess, graph.as_graph_def(), [“final_result_fruits”, “final_result_vegetables”]
  28. 67.
  29. 78.
  30. 79.

    as for today... Serving • API available • Deployment on

    AWS, currently migrating on Google Cloud
  31. 80.

    as for today... Serving • API available • Deployment on

    AWS, currently migrating on Google Cloud Training • Model debug done by overheating a laptop • Model built on personal GPU • Files uploaded manually
  32. 81.

    as for today... Serving • API available • Deployment on

    AWS, currently migrating on Google Cloud Training • Model debug done by overheating a laptop • Model built on personal GPU • Files uploaded manually
  33. 83.
  34. 85.

    public TensorFlowInferenceInterface(AssetManager var1, String var2) { //... this.modelName = var2;

    this.g = new Graph(); this.sess = new Session(this.g); this.runner = this.sess.runner(); boolean var3 = var2.startsWith("file:///android_asset/"); Object var4 = null; try { String var5 = var3?var2.split("file:///android_asset/")[1]:var2; var4 = var1.open(var5); } catch (IOException var11) { if(var3) { throw new RuntimeException("Failed to load model from '" + var2 + "'", var11); } try { var4 = new FileInputStream(var2); } catch (IOException var8) { throw new RuntimeException("Failed to load model from '" + var2 + "'", var11); } }} }
  35. 90.
  36. 91.

    91

  37. 93.
  38. 94.