Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Machine Learning for Android Developers

Dan J
July 18, 2017

Machine Learning for Android Developers

The slides are for a talk I gave at AnDevCon DC 2017:
http://www.andevcon.com/dc2017/sessions#MachineLearningforAndroidDevelopers

A screencast video of it is available here:
https://www.youtube.com/watch?v=fX7eQ9oRp-U

------------------------------------

Find out how to use machine learning on Android!

This talk explains what machine learning can do for you, how it works, and shows how to use a TensorFlow image classifier in your Android app.

We cover:
- Example machine learning applications, including sentiment analysis and artistic image filters.
- Machine learning APIs and frameworks.
- An overview of machine learning: training, evaluation and using models.
- Training data sets: NIST/MNIST, ImageNet, Kaggle.
- How to retrain a TensorFlow model and use it in your Android application: demo and sample code!

Dan J

July 18, 2017
Tweet

More Decks by Dan J

Other Decks in Programming

Transcript

  1. • Part 1 - Fundamentals ◦ Context and example applications

    ◦ Overview of training • Part 2 - Building stuff ◦ APIs and frameworks ◦ Using a pre-trained TensorFlow model ◦ Custom TensorFlow image classifier Agenda
  2. • 1997 - Deep Blue supercomputer beats chess world champion

    • 2006 - Personal computer beats chess world champion • 2011 - IBM Watson beats Jeopardy champions • 2015 - ML image classifiers start to beat humans • 2016 - AlphaGo beats Go Master Tipping Point
  3. “I honestly can't think of any recent product development that

    Microsoft has been involved in that hasn't involved machine learning” “Everything we do now is influenced, one way or another, by machine learning.” - Peter Lee, Microsoft’s Director of Research (Feb 2015)
  4. Jeff Dean, Google Senior Fellow - “Machine Learning - The

    Future Is Now” https://www.youtube.com/watch?v=UdHUDvxcH9o
  5. Sentiment Analysis • Does author feel positively, neutrally, or negatively?

    • Movie reviews, social media • Order matters
  6. When To Use Machine Learning • You cannot code the

    solution • You cannot scale with humans • You have training data
  7. • Subset of NIST ◦ 60k for training ◦ 10k

    for testing • Normalized • 20x20 pixels per character MNIST - http://yann.lecun.com/exdb/mnist/
  8. • English lexical database • Concepts = “Synsets” ◦ >114k

    synsets ◦ >80k are nouns • No longer maintained WORDNET - http://wordnet.princeton.edu
  9. • Images for some WORDNET ◦ >14 million images ◦

    >21k synsets • Annual competitions IMAGENET - http://imagenet.stanford.edu
  10. • Convert into appropriate formats • Label the data (for

    supervised learning) • Distort your data to generalize your model ◦ Rotate, zoom in/out, brighten, crop … mirror? ◦ Takes longer to train, so make a good model first 2 - Clean & Prepare Data
  11. • Very computationally expensive (CPU -> GPU -> TPU) •

    Only experts can configure training from scratch 3 - Train Model
  12. • Computer system modelled on the human brain • Nonlinear

    processing units extract features Neural Networks https://stackoverflow.com/a/32527774
  13. • Computer system modelled on the human brain • Nonlinear

    processing units extract features Neural Networks https://stackoverflow.com/a/32527774
  14. • Input data are already labelled • Model trains by

    iterating • Common examples: ◦ Regression 1 - Supervised
  15. • Input data are already labelled • Model trains by

    iterating • Common examples: ◦ Regression ◦ Classification 1 - Supervised
  16. • Input data are not labelled • Model deduces structures

    • Common examples: ◦ Clustering 2 - Unsupervised
  17. • Input data are not labelled • Model deduces structures

    • Common examples: ◦ Clustering ◦ Dimensionality Reduction 2 - Unsupervised
  18. • Learn by using a Reward function • Common examples:

    ◦ Games 3 - Reinforcement http://karpathy.github.io/2016/05/31/rl/
  19. • Learn by using a Reward function • Common examples:

    ◦ Games 3 - Reinforcement https://nihit.github.io/resources/spaceinvaders.pdf
  20. Machine Learning APIs • Google Cloud https://cloud.google.com/products/machine-learning/ • IBM Watson

    https://www.ibm.com/watson/developercloud/discovery.html • Amazon AWS https://aws.amazon.com/machine-learning/ • Microsoft Azure https://docs.microsoft.com/en-us/rest/api/machinelearning/
  21. Google Cloud APIs • Machine Learning Engine • Job Search

    • Video • Vision • Speech • Natural Language • Translation
  22. Why TensorFlow? • Excellent tutorials, tools and demos • Great

    developer engagement • Cross platform ◦ Windows, Mac, Linux, Android, iOS, etc ◦ Android Things too! • Built to scale
  23. • Training is done in Python or C++ • Requires

    NDK • Bazel is the build tool • Tutorials lag platform • Big model files (10MB to 90MB for image classifiers) ◦ TensorFlow Lite to the rescue? TensorFlow Limitations
  24. Install Android NDK Add to your PATH Build native library

    from command line: cd jni-build make Building the Native Library
  25. ndk-build Android NDK: WARNING:jni/Android.mk:tensorflow_mnist: non-system libraries in linker flags: jni/libs/armeabi-v7a/libprotos_all_cc.a

    jni/libs/armeabi-v7a/libprotobuf.a jni/libs/armeabi-v7a/libprotobuf_lite.a /Users/xgl413/Library/Android/android-ndk-r15b/sources/cxx-stl/gnu-libstdc++/4.9/libs/ armeabi-v7a/libgnustl_static.a /Users/xgl413/Library/Android/android-ndk-r15b/sources/cxx-stl/gnu-libstdc++/4.9/libs/ armeabi-v7a/libsupc++.a Android NDK: This is likely to result in incorrect builds. Try using LOCAL_STATIC_LIBRARIES Android NDK: or LOCAL_SHARED_LIBRARIES instead to list the library dependencies of the Android NDK: current module make[1]: Entering directory `/development/TensorFlowAndroidMNIST/jni-build' [armeabi-v7a] Compile++ arm : tensorflow_mnist <= tensorflow_jni.cc [armeabi-v7a] Compile++ arm : tensorflow_mnist <= jni_utils.cc [armeabi-v7a] SharedLibrary : libtensorflow_mnist.so [armeabi-v7a] Install : libtensorflow_mnist.so => libs/armeabi-v7a/libtensorflow_mnist.so make[1]: Leaving directory `/development/TensorFlowAndroidMNIST/jni-build'
  26. ndk-build Android NDK: WARNING:jni/Android.mk:tensorflow_mnist: non-system libraries in linker flags: jni/libs/armeabi-v7a/libprotos_all_cc.a

    jni/libs/armeabi-v7a/libprotobuf.a jni/libs/armeabi-v7a/libprotobuf_lite.a /Users/xgl413/Library/Android/android-ndk-r15b/sources/cxx-stl/gnu-libstdc++/4.9/libs/ armeabi-v7a/libgnustl_static.a /Users/xgl413/Library/Android/android-ndk-r15b/sources/cxx-stl/gnu-libstdc++/4.9/libs/ armeabi-v7a/libsupc++.a Android NDK: This is likely to result in incorrect builds. Try using LOCAL_STATIC_LIBRARIES Android NDK: or LOCAL_SHARED_LIBRARIES instead to list the library dependencies of the Android NDK: current module make[1]: Entering directory `/development/TensorFlowAndroidMNIST/jni-build' [armeabi-v7a] Compile++ arm : tensorflow_mnist <= tensorflow_jni.cc [armeabi-v7a] Compile++ arm : tensorflow_mnist <= jni_utils.cc [armeabi-v7a] SharedLibrary : libtensorflow_mnist.so [armeabi-v7a] Install : libtensorflow_mnist.so => libs/armeabi-v7a/libtensorflow_mnist.so make[1]: Leaving directory `/development/TensorFlowAndroidMNIST/jni-build'
  27. Performance • Android build = ~20 seconds • JNI build

    = <10 seconds • Classification = ~70ms first time, 10-20ms thereafter (Nexus 5)
  28. 1 - Install TensorFlow Install Docker from www.docker.com Download and

    start the TensorFlow binary image: docker run -it gcr.io/tensorflow/tensorflow:1.2.0 bash
  29. 2 - Download Training Script Download the scripts into our

    Docker TensorFlow container: curl -O https://raw.githubusercontent.com/miyosuda/TensorFlowAndr oidMNIST/master/trainer-script/expert.py curl -O https://raw.githubusercontent.com/miyosuda/TensorFlowAndr oidMNIST/master/trainer-script/input_data.py
  30. # python expert.py Successfully downloaded train-images-idx3-ubyte.gz 9912422 bytes. Extracting data/train-images-idx3-ubyte.gz

    Successfully downloaded train-labels-idx1-ubyte.gz 28881 bytes. Extracting data/train-labels-idx1-ubyte.gz Successfully downloaded t10k-images-idx3-ubyte.gz 1648877 bytes. Extracting data/t10k-images-idx3-ubyte.gz Successfully downloaded t10k-labels-idx1-ubyte.gz 4542 bytes. Extracting data/t10k-labels-idx1-ubyte.gz 2017-07-15 20:45:24.352372: W tensorflow/core/platform/cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use SSE4.1 instructions, but these are available on your machine and could speed up CPU computations. ... step 0, training accuracy 0.14 step 100, training accuracy 0.86 step 200, training accuracy 0.96 step 300, training accuracy 0.88 step 400, training accuracy 1 step 500, training accuracy 0.94 ...
  31. Docker Gotchas • Changes are lost unless you commit them

    • Only certain local folders can be mounted (e.g. $HOME) • Weird workarounds to avoid out of memory errors
  32. ... step 10000, training accuracy 1 step 10100, training accuracy

    1 step 10200, training accuracy 0.98 step 10300, training accuracy 1 step 10400, training accuracy 0.98 step 10500, training accuracy 1 step 10600, training accuracy 1 ...
  33. ... step 19500, training accuracy 1 step 19600, training accuracy

    1 step 19700, training accuracy 1 step 19800, training accuracy 1 step 19900, training accuracy 1 Killed
  34. Installing TensorBoard • By default has a dependency on TensorFlow

    docker run -p 6006:6006 -it gcr.io/tensorflow/tensorflow:1.2.0 bash • Standalone version available: https://github.com/dmlc/tensorboard pip install tensorboard
  35. TensorBoard - Limitations • No easy way to import a

    pretrained model :-( ◦ import_pb_to_tensorboard.py in master, but doesn’t work • Requires code into your training script • Try to find training examples built for TensorBoard
  36. # Uncomment and update the paths in these entries to

    build the demo. android_sdk_repository( name = "androidsdk", api_level = 23, build_tools_version = " 25.0.2", # Replace with path to Android SDK on your system path = "/android", ) android_ndk_repository( name="androidndk", path="/android/android-ndk-r12b ", api_level= 21)
  37. ... ERROR: /Users/Dan/Code/tensorflow/WORKSPACE:8:1: Traceback (most recent call last): File "/Users/Dan/Code/tensorflow/WORKSPACE",

    line 8 check_version("0.4.2") File "/Users/Dan/Code/tensorflow/tensorflow/workspace.bzl", line 33, in check_version fail(" Current Bazel version is {}, e...)) Current Bazel version is 0.3.2-homebrew, expected at least 0.4.2 . ERROR: Error evaluating WORKSPACE file. ERROR: error loading package 'external': Package 'external' contains errors. INFO: Elapsed time: 0.214s
  38. ... ERROR: /Users/Dan/Code/tensorflow/WORKSPACE:8:1: Traceback (most recent call last): File "/Users/Dan/Code/tensorflow/WORKSPACE",

    line 8 check_version("0.4.2") File "/Users/Dan/Code/tensorflow/tensorflow/workspace.bzl", line 33, in check_version fail(" Current Bazel version is {}, e...)) Current Bazel version is 0.3.2-homebrew, expected at least 0.4.2 . ERROR: Error evaluating WORKSPACE file. ERROR: error loading package 'external': Package 'external' contains errors. INFO: Elapsed time: 0.214s
  39. WARNING: /Users/Dan/Code/tensorflow/tensorflow/core/BUILD:826:12: in srcs attribute of cc_library rule //tensorflow/core:android_tensorflow_lib_lite: please

    do not import '//tensorflow/core/util/tensor_bundle:tensor_bundle.h' directly. You should either move the file to this package or depend on an appropriate rule there. ERROR: /Users/Dan/Code/tensorflow/WORKSPACE:11:1: no such package '@androidsdk//': Bazel requires Android build tools version 24.0.3 or newer, 23.0.1 was provided and referenced by '//external:android/sdk'. ERROR: Analysis of target '//tensorflow/examples/android:tensorflow_demo' failed; build aborted. INFO: Elapsed time: 28.826s
  40. WARNING: /Users/Dan/Code/tensorflow/tensorflow/core/BUILD:826:12: in srcs attribute of cc_library rule //tensorflow/core:android_tensorflow_lib_lite: please

    do not import '//tensorflow/core/util/tensor_bundle:tensor_bundle.h' directly. You should either move the file to this package or depend on an appropriate rule there. ERROR: /Users/Dan/Code/tensorflow/WORKSPACE:11:1: no such package '@androidsdk//': Bazel requires Android build tools version 24.0.3 or newer, 23.0.1 was provided and referenced by '//external:android/sdk'. ERROR: Analysis of target '//tensorflow/examples/android:tensorflow_demo' failed; build aborted. INFO: Elapsed time: 28.826s
  41. Building From Source Start our Docker container docker run -it

    danjarvis/tensorflow-android:1.0.0 From in the container bazel build -c opt --local_resources 4096,4.0,1.0 -j 1 //tensorflow/examples/android:tensorflow_demo
  42. tensorflow/core/kernels/split_op.cc:159:64: warning: narrowing conversion of '(tensorflow::int64)split_dim_output_size' from 'tensorflow::int64 {aka long

    long int}' to 'int' inside { } [-Wnarrowing] Slow read: a 756175192-byte read from /root/.cache/bazel/_bazel_root/68a62076e91007a7908bc42a32e4cff9/execroot/tensorflow/ba zel-out/android-arm-linux-androideabi-4.9-v7a-gnu-libstdcpp-opt/bin/tensorflow/core/ke rnels/libandroid_tensorflow_kernels.lo took 5883ms. Target //tensorflow/examples/android:tensorflow_demo up-to-date: bazel-bin/tensorflow/examples/android/tensorflow_demo_deploy.jar bazel-bin/tensorflow/examples/android/tensorflow_demo_unsigned.apk bazel-bin/tensorflow/examples/android/tensorflow_demo.apk INFO: Elapsed time: 1325.598s, Critical Path: 95.81s
  43. tensorflow/core/kernels/split_op.cc:159:64: warning: narrowing conversion of '(tensorflow::int64)split_dim_output_size' from 'tensorflow::int64 {aka long

    long int}' to 'int' inside { } [-Wnarrowing] Slow read: a 756175192-byte read from /root/.cache/bazel/_bazel_root/68a62076e91007a7908bc42a32e4cff9/execroot/tensorflow/ba zel-out/android-arm-linux-androideabi-4.9-v7a-gnu-libstdcpp-opt/bin/tensorflow/core/ke rnels/libandroid_tensorflow_kernels.lo took 5883ms. Target //tensorflow/examples/android:tensorflow_demo up-to-date: bazel-bin/tensorflow/examples/android/tensorflow_demo_deploy.jar bazel-bin/tensorflow/examples/android/tensorflow_demo_unsigned.apk bazel-bin/tensorflow/examples/android/tensorflow_demo.apk INFO: Elapsed time: 1325.598s, Critical Path: 95.81s
  44. tensorflow/core/kernels/split_op.cc:159:64: warning: narrowing conversion of '(tensorflow::int64)split_dim_output_size' from 'tensorflow::int64 {aka long

    long int}' to 'int' inside { } [-Wnarrowing] Slow read: a 756175192-byte read from /root/.cache/bazel/_bazel_root/68a62076e91007a7908bc42a32e4cff9/execroot/tensorflow/ba zel-out/android-arm-linux-androideabi-4.9-v7a-gnu-libstdcpp-opt/bin/tensorflow/core/ke rnels/libandroid_tensorflow_kernels.lo took 5883ms. Target //tensorflow/examples/android:tensorflow_demo up-to-date: bazel-bin/tensorflow/examples/android/tensorflow_demo_deploy.jar bazel-bin/tensorflow/examples/android/tensorflow_demo_unsigned.apk bazel-bin/tensorflow/examples/android/tensorflow_demo.apk INFO: Elapsed time: 1325.598s, Critical Path: 95.81s
  45. Image Classifier Model - Inception-v3 • https://github.com/tensorflow/models/tree/master/inception • Trained on

    1000 IMAGENET categories from 2012 https://medium.com/towards-data-science/transfer-learning-using-keras-d804b2e04ef8
  46. Transfer Learning • https://www.tensorflow.org/tutorials/image_retraining • Retraining a classifier is easy!

    https://medium.com/towards-data-science/transfer-learning-using-keras-d804b2e04ef8
  47. docker run -it -v $HOME/tf_files:/tf_files danjarvis/tensorflow-android:1.0.0 python tensorflow/examples/image_retraining/retrain.py \ --bottleneck_dir=/tf_files/bottlenecks

    \ --how_many_training_steps 500 \ --model_dir=/tf_files/inception \ --output_graph=/tf_files/retrained_graph.pb \ --output_labels=/tf_files/retrained_labels.txt \ --image_dir /tf_files/photos Retraining The Model
  48. ... 2017–02–27 02:39:58.461678: Step 499: Train accuracy = 100.0% 2017–02–27

    02:39:58.461841: Step 499: Cross entropy = 0.036179 2017–02–27 02:39:59.081111: Step 499: Validation accuracy = 99.0% (N=100) Final test accuracy = 97.2% (N=145) Converted 2 variables to const ops.
  49. bazel build --local_resources 4096,4.0,1.0 -j 1 tensorflow/python/tools:strip_unused bazel-bin/tensorflow/python/tools/strip_unused \ --input_graph=/tf_files/retrained_graph.pb

    \ --output_graph=/tf_files/stripped_retrained_graph.pb \ --input_node_names="Mul" \ --output_node_names="final_result" \ --input_binary=true Prepare The Model
  50. From easiest to hardest: 1. Use machine learning cloud APIs

    2. Embed pre-trained models 3. Follow instructions to train a model 4. Use transfer learning to customize a pre-trained model 5. Train your own model from scratch 6. Train your model dynamically on the device How To Use Machine Learning
  51. • Blog posts: https://medium.com/@daj • Docker container: https://hub.docker.com/r/danjarvis/ • Legs

    or hot dogs images: https://github.com/daj/legs-or-hotdogs-images • Slides: https://speakerdeck.com/daj Links