Tensor Flow for Android Devs

Introduction to the Tensorflow from an Android developers Point of view. We will go through a simple example, explaining the steps, hurdles of integration of a TensorFlow model into a mobile application. In the end, you will understand the process of working with TensorFlow on Android devices. Hopefully, this will motivate you to implement your crazy AI ideas in the mobile world.

Transcript

1. @ablenessy | @droidconLisbon | #DCLISBON19 for android devs
2. @ablenessy | @droidconLisbon | #DCLISBON19 What can AI do?
3. @ablenessy | @droidconLisbon | #DCLISBON19 What can AI do? Image classification Object detection Gesture recognition Speech recognition Text generation Translate Generate music Generate images
4. @ablenessy | @droidconLisbon | #DCLISBON19 Artificial Intelligence Machine Learning Deep learning: Computer vision, NLP, GAN …
5. @ablenessy | @droidconLisbon | #DCLISBON19 Source: tensorflow.com/about
6. @ablenessy | @droidconLisbon | #DCLISBON19 Source: tensorflow.com/about Neural Network Neural Network
7. @ablenessy | @droidconLisbon | #DCLISBON19 Source: tensorflow.com/about Training
8. @ablenessy | @droidconLisbon | #DCLISBON19 Caffe PyTorch
9. @ablenessy | @droidconLisbon | #DCLISBON19 TensorFlow https://www.tensorflow.org/beta You can write code in: C++, Python, Swift, JavaScript Deploy to: CPU, GPU, TPU Mobile iOS and Android Raspberry Py, Coral (edge TPU), Microcontrollers TensorFlow 2.0 is now in beta Premade Estimators tf.Keras tf.* Low level API High level API Ready made models
10. @ablenessy | @droidconLisbon | #DCLISBON19 Keras High level Neural Networks API, written in Python Integrated in TensorFlow : tf.keras Sequential - best to start with Functional - more flexible Model subclassing - extend a Model class Deep learning with Python By Francois Chollet
11. @ablenessy | @droidconLisbon | #DCLISBON19 import tensorflow as tf mnist = tf.keras.datasets.mnist (x_train, y_train),(x_test, y_test) = mnist.load_data() x_train, x_test = x_train / 255.0, x_test / 255.0 model = tf.keras.models.Sequential([ tf.keras.layers.Flatten(input_shape=(28, 28)), tf.keras.layers.Dense(512, activation=tf.nn.relu), tf.keras.layers.Dropout(0.2), tf.keras.layers.Dense(10, activation=tf.nn.softmax) ]) model.compile(optimizer='adam', loss='sparse_categorical_crossentropy', metrics=['accuracy']) model.fit(x_train, y_train, epochs=5) model.evaluate(x_test, y_test) Digit classifier
12. @ablenessy | @droidconLisbon | #DCLISBON19 On device ML Low latency, no server calls Works offline, no connection needed Privacy, data stays on device
13. @ablenessy | @droidconLisbon | #DCLISBON19 Tensor Flow Open sourced Tensor Flow Mobile TF Lite Developer preview ML Kit Tensor Flow 2.0 TF Lite - final TF Mobile deprecated 2015 2016 2017 2018 2019 Source: https://speakerdeck.com/margaretmz
14. @ablenessy | @droidconLisbon | #DCLISBON19 Application Developer ML Practitioner Data Scientist Firebase ML Kit TensorFlow Lite Neural Networks API
15. @ablenessy | @droidconLisbon | #DCLISBON19 Firebase ML Kit Source: firebase.google.com/products/ml-kit Image labelling Text Recognition Face Detection Barcode scanning Landmark detection Supports Custom models Dynamic downloads New: Auto ML, Smart reply, Object detection, Translation
16. @ablenessy | @droidconLisbon | #DCLISBON19 Text Recognition
17. @ablenessy | @droidconLisbon | #DCLISBON19 val image = FirebaseVisionImage.fromBitmap(selectedImage) val detector = FirebaseVision.getInstance() .getVisionTextDetector() detector.detectInImage(image) .addOnSuccessListener { texts -> processTextRecognitionResult(texts) } .addOnFailureListener(...)
18. @ablenessy | @droidconLisbon | #DCLISBON19 ML workflow Get Data > Train > Convert > Integrate
19. @ablenessy | @droidconLisbon | #DCLISBON19 TensorFlow #pix2pix Source: https://affinelayer.com/pixsrv/
20. @ablenessy | @droidconLisbon | #DCLISBON19 #pix2pix TensorFlow
21. @ablenessy | @droidconLisbon | #DCLISBON19 #pix2pix TensorFlow
22. @ablenessy | @droidconLisbon | #DCLISBON19 How does pix2pix work? Source: https://affinelayer.com/pix2pix/
23. @ablenessy | @droidconLisbon | #DCLISBON19 How does pix2pix work? Source: https://affinelayer.com/pix2pix/
24. @ablenessy | @droidconLisbon | #DCLISBON19 Source: https://dzone.com/articles/working-principles-of-generative-adversarial-netwo
25. @ablenessy | @droidconLisbon | #DCLISBON19 How does pix2pix work? generator INPUT OUTPUT Source: https://affinelayer.com/pix2pix/
26. @ablenessy | @droidconLisbon | #DCLISBON19 How does pix2pix work? Source: https://affinelayer.com/pix2pix/
27. @ablenessy | @droidconLisbon | #DCLISBON19
28. @ablenessy | @droidconLisbon | #DCLISBON19
29. Source: https://affinelayer.com/pix2pix/
30. @ablenessy | @droidconLisbon | #DCLISBON19 TensorBoard
31. @ablenessy | @droidconLisbon | #DCLISBON19 Train > Convert > Integrate
32. ~680 mb ~2600 operations Train > Convert > Integrate
33. ~680 mb ~2600 operations Train > Convert > Integrate
34. ~200 mb ~400 operations Train > Convert > Integrate
35. @ablenessy | @droidconLisbon | #DCLISBON19 Train > Convert > Integrate
36. @ablenessy | @droidconLisbon | #DCLISBON19 Interpreter Core Converter TensorFlow Lite Format Operation Kernels Hardware acceleration ~75 KB vs 1.1 MB in TF ~2.4 MB vs 22.3 MB in Deprecate TF Mobile
37. @ablenessy | @droidconLisbon | #DCLISBON19 Converter TensorFlow Lite Format 
 Command line Python API Android App (Java /C++ API) iOS App (C++ API) Linux (e.g. Raspberry Pi) (C++ API) Trained TensorFlow Model SavedModel HDF5
38. Convert saved_model_dir = PATH + 'saved_model' tf.saved_model.save(generator, saved_model_dir) converter = tf.lite.TFLiteConverter.from_saved_model(saved_model_dir) tf_lite_model = converter.convert() tf_lite_model_file.write_bytes(tf_lite_model)
39. Convert NEW saved_model_dir = PATH + 'saved_model' tf.saved_model.save(generator, saved_model_dir) converter = tf.lite.TFLiteConverter.from_saved_model(saved_model_dir) tf_lite_model = converter.convert() tf_lite_model_file.write_bytes(tf_lite_model) converter.optimizations = [tf.lite.Optimize.DEFAULT] converter.target_spec.supported_types = [tf.float16] tf_lite_fp16_model = converter.convert() tf_lite_f16_model_file.write_bytes(tf_lite_fp16_model)
40. Optimisation options 103 MB 207 MB 154 MB 61 MB - requires custom ops Fails
41. Test test test!
42. Model > Training > Deployment android { aaptOptions { noCompress "tflite" } } implementation 'org.tensorflow:tensorflow-lite:1.14.0'
43. @ablenessy | @droidconLisbon | #DCLISBON19 Model > Training > Deployment
44. @ablenessy | @droidconLisbon | #DCLISBON19 Attila Blénesi Android Engineer @ Babylon Health @ablenessy @ablenessy ablenesi