TENSORFLOW FOR ANDROID DEVS

Introduction to the Tensorflow from an Android developers Point of view. We will go through a simple example, explaining the steps, hurdles of integration a TensorFlow model into a mobile application. In the end, you will understand the process of working with TensorFlow on Android devices. Hopefully, this will motivate you to implement your crazy AI ideas in the mobile world.

Updated version from Droidcon Transylvania

Transcript

1. For Android devs
2. Attila Blénesi Senior Android Engineer @ Halcyon Mobile @ablenessy
3. Is machine learning only for experts ? @ablenessy | @droidconRO | #DroidconT
5. Application Developer ML Practitioner Data Scientist Firebase ML Kit Machine Learning APIs TensorFlow Cloud ML @ablenessy | @droidconRO | #DroidconT
6. Firebase ML Kit @ablenessy | @droidconRO | #DroidconT Source: firebase.google.com/products/ml-kit
7. Text recognition
8. val image = FirebaseVisionImage.fromBitmap(selectedImage) val detector = FirebaseVision.getInstance() .getVisionTextDetector() detector.detectInImage(image) .addOnSuccessListener { texts -> processTextRecognitionResult(texts) }a .addOnFailureListener(...) @ablenessy | @droidconRO | #DroidconT
9. Text recognition
10. Text recognition
11. val image = FirebaseVisionImage.fromBitmap(selectedImage) val options = FirebaseVisionCloudDetectorOptions.Builder() .setModelType(FirebaseVisionCloudDetectorOptions.LATEST_MODEL) .setMaxResults(15) .build() val detector = FirebaseVision.getInstance() .getVisionCloudDocumentTextDetector(options) detector.detectInImage(image) .addOnSuccessListener { texts -> processTextRecognitionResult(texts) }a .addOnFailureListener(...) @ablenessy | @droidconRO | #DroidconT
12. Application Developer ML Practitioner Data Scientist Firebase ML Kit Machine Learning APIs TensorFlow Cloud ML
13. Machine Learning APIs
14. Source: cloud.google.com/vision/
15. google.com/about/stories/machine-learning-qa/
16. google.com/about/stories/machine-learning-qa/
17. Source: cloud.google.com/vision/
18. Source: cloud.google.com/vision/
19. Do I even have to dig deeper ? @ablenessy | @droidconRO | #DroidconT
20. generic model = generic solution Do I even have to dig deeper ? @ablenessy | @droidconRO | #DroidconT
21. Application Developer ML Practitioner Data Scientist Firebase ML Kit Machine Learning APIs TensorFlow Cloud ML
22. TensorFlow Cloud ML #edges2cats
23. TensorFlow Cloud ML #edges2cats
24. TensorFlow Cloud ML #edges2cats Source: https://affinelayer.com/pixsrv/
25. workflow Model > Training > Optimisation > Deployment @ablenessy | @droidconRO | #DroidconT
26. How does pix2pix work? Model > Training > Optimisation > Deployment
27. How does pix2pix work? Model > Training > Optimisation > Deployment
28. How does pix2pix work?
29. How does pix2pix work?
30. How does pix2pix work? generator INPUT OUTPUT
31. How does pix2pix work? generator INPUT OUTPUT That doesn’t look right
32. How does pix2pix work?
33. Training the discriminator
34. Training the generator
35. Model > Training > Optimisation > Deployment
36. Model > Training > Optimisation > Deployment
37. Model > Training > Optimisation > Deployment
38. Model > Training > Optimisation > Deployment
39. Local ~ 36 hours ~ 12 hours Cloud vs @ablenessy | @droidconRO | #DroidconT
40. Model > Training > Optimisation > Deployment Cloud ML
41. Model > Training > Optimisation > Deployment gcloud ml-engine jobs submit training $JOB_NAME \ --job-dir $JOB_DIR \ --module-name trainer.pix2pix \ --package-path ./trainer \ --region $REGION \ --config=trainer/cloudml-gpu.yaml \ -- \ --mode train \ --input-dir gs://$BUCKET_NAME/train Cloud ML
42. Model > Training > Optimisation > Deployment gcloud ml-engine jobs submit training $JOB_NAME \ --job-dir $JOB_DIR \ --module-name trainer.pix2pix \ --package-path ./trainer \ --region $REGION \ --config=trainer/cloudml-gpu.yaml \ -- \ --mode train \ --input-dir gs://$BUCKET_NAME/train Cloud ML
43. trainingInput: scaleTier: CUSTOM masterType: standard_gpu # 1 GPU pythonVersion: "3.5" runtimeVersion: "1.8" Model > Training > Optimisation > Deployment Cloud ML
44. trainingInput: scaleTier: CUSTOM masterType: complex_model_m_gpu # 4 GPUs pythonVersion: "3.5" runtimeVersion: "1.8" Model > Training > Optimisation > Deployment Cloud ML
45. Model > Training > Optimisation > Deployment gcloud ml-engine jobs submit training $JOB_NAME \ --job-dir $JOB_DIR \ --module-name trainer.pix2pix \ --package-path ./trainer \ --region $REGION \ --config=trainer/cloudml-gpu.yaml \ -- \ --mode train \ --input-dir gs://$BUCKET_NAME/train Cloud ML
46. Model > Training > Optimisation > Deployment gcloud ml-engine jobs submit training $JOB_NAME \ --job-dir $JOB_DIR \ --module-name trainer.pix2pix \ --package-path ./trainer \ --region $REGION \ --config=trainer/cloudml-gpu.yaml \ -- \ --mode train \ --input-dir gs://$BUCKET_NAME/train Cloud ML
47. Model > Training > Optimisation > Deployment
48. Model > Training > Optimisation > Deployment TensorBoard
49. Model > Training > Optimisation > Deployment TensorBoard
50. Model > Training > Optimisation > Deployment TensorBoard tensorboard —logdir /path/to/model http://localhost:6006/
51. Model > Training > Optimisation > Deployment TensorBoard
52. Model > Training > Optimisation > Deployment
53. Model > Training > Optimisation > Deployment ~680 mb ~2600 operations
54. Model > Training > Optimisation > Deployment ~680 mb ~2600 operations
55. Model > Training > Optimisation > Deployment ~200 mb ~400 operations
56. Model > Training > Optimisation > Deployment checkpoint reduced_model.data-00000-of-00001 reduced_model.meta reduced_model.index frozen_model.pb
57. Model > Training > Optimisation > Deployment # We start a session using a temporary fresh Graph with tf.Session(graph=tf.Graph()) as sess: # We import the meta graph in the current default Graph saver = tf.train.import_meta_graph(input_checkpoint + ‘.meta’,…) saver.restore(sess, input_checkpoint) # We restore the weights output_graph_def = tf.graph_util.convert_variables_to_constants( sess, # The session is used to retrieve the weights tf.get_default_graph().as_graph_def(), # retrieve the nodes output_node_names # select the useful nodes ) with tf.gfile.GFile(output_graph, "wb") as f: f.write(output_graph_def.SerializeToString())
58. Model > Training > Optimisation > Deployment checkpoint reduced_model.data-00000-of-00001 reduced_model.meta reduced_model.index frozen_model.pb
59. Model > Training > Optimisation > Deployment // TensorFlow Android implementation ‘org.tensorflow:tensorflow-android:1.10.0’
60. Model > Training > Optimisation > Deployment
61. Model > Training > Optimisation > Deployment output_tensor Generated image jalammar.github.io/Supercharging-android-apps-using-tensorflow/ Generator
62. Model > Training > Optimisation > Deployment val interpreter = TensorFlowInferenceInterface( assetManager, "file:///android_asset/frozen_model.pb" ) inputValues = generateInput(inputDrawing) interpreter.feed("input_node", inputValues, inputDimensions) interpreter.run(arrayOf("output_node"), false) interpreter.fetch(“output_node", outputValues) val result = generateBitmapFromOutput(outputValues)
65. Model > Training > Optimisation > Deployment // TensorFlow Lite implementation ‘org.tensorflow:tensorflow-lite:1.10.0’
66. Model > Training > Optimisation > Deployment val interpreter = Interpreter( File(URI.create("file://android_asset/frozen_model.tflite")) )a inputValues = generateInput(inputDrawing) interpreter.run(inputValues, outputValues) val result = generateBitmapFromOutput(outputValues)
67. Demo @ablenessy | @droidconRO | #DroidconT
68. @ablenessy | @droidconRO | #DroidconT Machine learning is NOT just for experts
69. @ablenessy | @droidconRO | #DroidconT Machine learning is NOT just for experts
70. @ablenessy | @droidconRO | #DroidconT