Slide 1
Slide 1 text
Ready for AI?
ML on Moble
» Machine Learning & TensorFlow
» Build your own model and optimize it
DroidKaigi 2018 @rejasupotaro 1
Slide 2
Slide 2 text
» Kentaro Takiguchi / @rejasupotaro
» Software Engineer
» Android (since 2010)
» ! Web
» ! Search
» ! ML
» Discovery team at Cookpad UK
2
Slide 3
Slide 3 text
Discovery
= Search + Recommendation
Provide users with the right contents at the right time
3
Slide 4
Slide 4 text
Deep Learning Achievements Over The
Past Year
https://blog.statsbot.co/deep-learning-achievements-4c563e034257 4
Slide 5
Slide 5 text
Artifical Intelligence, Blockchain, IoT, VR,
Serverless architecture, ...
5
Slide 6
Slide 6 text
Best Apps in 2017
Socratic, Pinterest, Faceapp 6
Slide 7
Slide 7 text
Google Flights will now predict airline
delays before the airlines do
7
Slide 8
Slide 8 text
One Day
» When I send a resume to a company, AI predicts my
future performance, then they reject without
having an interview
8
Slide 9
Slide 9 text
Another Day
» When I send a resume to a company, AI predicts my
future performance, then they reject without
having an interview
» When I attempt to buy an insurance, AI detects a
risk of my future illness, then they reject my
application
9
Slide 10
Slide 10 text
Another Day
» When I send a resume to a company, AI predicts my
future performance, then they reject without
having an interview
» When I attempt to buy an insurance, AI detects a
risk of my future illness, then they reject my
application
» When I pass in front of a surveillance camera, AI
detects that I will commit a crime, then I'm
arrested
10
Slide 11
Slide 11 text
! Sounds fun !
11
Slide 12
Slide 12 text
! This is the talk for you
» ✅ "I'm just interested in AI"
» Welcome!
» ✅ "I've investigated the feasibility. I want to
release"
» This talk focuses on what's happening inside
» You would be able to release your app with
confidence
12
Slide 13
Slide 13 text
API development
is matured
Server: "Call this API with the param"
GET /users?id=1
Client: "Okay"
» We get to know how to use it by just looking at
this one line
13
Slide 14
Slide 14 text
Productioning ML
has just started
ML: "I trained a model. Use this"
awesome_model.pb
Client: ???
» Machine learning researchers need to know the real
environment
» Product developers need to learn machine learning
14
Slide 15
Slide 15 text
My Motivation
It's like Android development around 2012 (Everything
was difficult...)
I feel we need to share knowledge.
! I want to share what I learned.
15
Slide 16
Slide 16 text
Build
This
AI
App
Together
https://github.com/rejasupotaro/logic-gate 16
Slide 17
Slide 17 text
Why TensorFlow?
» There are many libraries such as PyTorch, Caffe2,
Theano, Chainer, ...
17
Slide 18
Slide 18 text
Runtime
We would ❤ the official way
Android iOS
TensorFlow
Deeplearning4j
Core ML
TensorFlow
” Let's ask if the library they love supports these
platforms
18
Slide 19
Slide 19 text
Train ! (IR) ! Inference
19
Slide 20
Slide 20 text
What is TensorFlow?
“TensorFlow is an interface for expressing machine
learning algorithms, and an implementation for
executing such algorithms, focusing on a wide variety
of heterogeneous systems, ranging from mobile devices
up to large-scale distributed systems of hundred
machines and thousands of computational devices”
https://static.googleusercontent.com/media/research.google.com/en//pubs/archive/45166.pdf 20
Slide 21
Slide 21 text
What is TensorFlow?
https://www.tensorflow.org/programmers_guide/graphs 21
Slide 22
Slide 22 text
What is TensorFlow?
import tensorflow as tf
a = tf.constant(2)
b = tf.constant(3)
with tf.Session() as session:
print(session.run(a + b)) # => 5
22
Slide 23
Slide 23 text
Why TensorFlow?
» There are many libraries such as PyTorch, Caffe2,
Theano, Chainer, ...
» ! Most of libraries are designed for research, but
TensorFlow is designed to run models in
production, for various environments including
mobile.
23
Slide 24
Slide 24 text
Architecture of TensorFlow
24
Slide 25
Slide 25 text
TensorFlow Mobile & Lite
25
Slide 26
Slide 26 text
Arhictecture of TensorFlow Lite
https://developer.android.com/ndk/guides/neuralnetworks/index.html 26
Slide 27
Slide 27 text
Arm support for Android NNAPI
gives >4x performance boost
https://arxiv.org/pdf/1801.06274.pdf 27
Slide 28
Slide 28 text
TensorFlow Mobile (17.2 MB)
TensorFlow Lite (1.5 MB)
28
Slide 29
Slide 29 text
TensorFlow Mobile & Lite
TensorFlow Mobile TensorFlow Lite
1.4.0 0.1.1
» TensorFlow Mobile
» = TensorFlow Java + Android
» TensorFlow Lite
» = Minimum implementation optimized for mobile
and embedded devices
as of 2018/02/01 29
Slide 30
Slide 30 text
“Google will make
Android the best
platform for
machine learning”
ૉΒ͍͠NDKͷੈք / Building High Performance Android Apps with NDK ʵ 2018/02/08 15:40-16:30 30
Slide 31
Slide 31 text
Let's See How to run a trained
model
31
Slide 32
Slide 32 text
Run a trained model
» Add the dependency.
implementation "org.tensorflow:tensorflow-android:1.4.0"
» Put a model in src/main/assets
» Run the computation graph
inferenceInterface.feed(inputName, input, shape.first, shape.second)
inferenceInterface.run(arrayOf(outputName))
inferenceInterface.fetch(outputName, output)
» Give [0, 1] ! Get [0]
32
Slide 33
Slide 33 text
That's it!
» It can be done within 5 min if I have a trained
model
» You don't need to write C++. You don't need to use
CMake. Just use the library.
33
Slide 34
Slide 34 text
Logic AND Gate
Input Output
0, 0 0
0, 1 0
1, 0 0
1, 1 1
34
Slide 35
Slide 35 text
» , : input
» : output
» , : variables
» Fomula:
35
Slide 36
Slide 36 text
Let's define a graph
36
Slide 37
Slide 37 text
x = tf.placeholder(tf.float32, shape=[None, 2], name='x')
y = tf.placeholder(tf.float32, shape=[None, 1], name='y')
w = tf.Variable(tf.zeros([2, 1]), name='weight')
b = tf.Variable(tf.zeros([1]), name='bias')
y_pred = tf.nn.sigmoid(tf.matmul(x, w) + bias, name='y_pred')
with tf.name_scope("loss"):
loss = tf.reduce_sum(tf.square(y_pred - y), name='loss')
with tf.name_scope("train"):
optimizer = tf.train.AdamOptimizer(learning_rate=0.1, name='optimizer')
train_step = optimizer.minimize(loss, name='train_step')
with tf.Session() as session:
session.run(tf.global_variables_initializer())
for epoch in range(self.args.epochs):
_, summary, l = session.run(
[train_step, merged, loss],
feed_dict={
x: input, y: output
}
)
37
Slide 38
Slide 38 text
x = tf.placeholder(tf.float32, shape=[None, 2], name='x')
y = tf.placeholder(tf.float32, shape=[None, 1], name='y')
w = tf.Variable(tf.zeros([2, 1]), name='weight')
b = tf.Variable(tf.zeros([1]), name='bias')
y_pred = tf.nn.sigmoid(tf.matmul(x, w) + bias, name='y_pred')
with tf.name_scope("loss"):
loss = tf.reduce_sum(tf.square(y_pred - y), name='loss')
with tf.name_scope("train"):
optimizer = tf.train.AdamOptimizer(learning_rate=0.1, name='optimizer')
train_step = optimizer.minimize(loss, name='train_step')
with tf.Session() as session:
session.run(tf.global_variables_initializer())
for epoch in range(self.args.epochs):
_, summary, l = session.run(
[train_step, merged, loss],
feed_dict={
x: input, y: output
}
)
38
Slide 39
Slide 39 text
x = tf.placeholder(tf.float32, shape=[None, 2], name='x')
y = tf.placeholder(tf.float32, shape=[None, 1], name='y')
w = tf.Variable(tf.zeros([2, 1]), name='weight')
b = tf.Variable(tf.zeros([1]), name='bias')
y_pred = tf.nn.sigmoid(tf.matmul(x, w) + bias, name='y_pred')
with tf.name_scope("loss"):
loss = tf.reduce_sum(tf.square(y_pred - y), name='loss')
with tf.name_scope("train"):
optimizer = tf.train.AdamOptimizer(learning_rate=0.1, name='optimizer')
train_step = optimizer.minimize(loss, name='train_step')
with tf.Session() as session:
session.run(tf.global_variables_initializer())
for epoch in range(self.args.epochs):
_, summary, l = session.run(
[train_step, merged, loss],
feed_dict={
x: input, y: output
}
)
39
Slide 40
Slide 40 text
Training
» Define the problem (inputs and expected outputs)
» Build the computation graph
» : tf.placeholder
» , : tf.Variable
» : tf.nn.sigmoid(tf.matmul(x, w) + b)
» Find the appropriate values by iterating training
steps
40
Slide 41
Slide 41 text
Visualizing a Model
41
Slide 42
Slide 42 text
Congrats! !
» Once you understand how model is built, you can
build any models (by modifying the graph a little)
» You don't need to hardcode the logic. Machine
learns how to do it instead of you.
42
Slide 43
Slide 43 text
Hardcode Logic
fun predict(x1: Int, x2: Int): Int {
return if (x1 == 0 && x2 == 0) {
0
} else if (x1 == 1 && x2 == 0) {
0
} else if (x1 == 0 && x2 == 1) {
0
} else if (x1 == 1 && x2 == 1) {
1
} else {
throw RuntimeException("I hope it won't happen")
}
}
” List all possible cases
43
Slide 44
Slide 44 text
Machine Learns
// [[0, 0], [0, 1], [1, 0], [1, 1]]
fun predict(x1: Int, x2: Int): Int {
??? // Machine learns what value to return
}
// => [[0], [0], [0], [1]]
” You don't need to write the logic
44
Slide 45
Slide 45 text
XOR Gate
// [[0, 0], [0, 1], [1, 0], [1, 1]]
fun predict(x1: Int, x2: Int): Int {
??? // Machine learns what value to return
}
// => [[0], [1], [1], [0]]
Nest more layers when we work on a complex problem 45
Slide 46
Slide 46 text
Estimate Weight
// [170cm, Man, ...]
fun predict(...): Int {
??? // Machine learns what value to return
}
// => 70kg
46
Slide 47
Slide 47 text
Translation
» "I am learning machine learning"
» ! ["i", "am", "learning", "machine", "learning"]
» ! [0.53177921, 0.7965138, 0.66959208, ...]
» ! ["ࢲ", "", "ػցֶश", "Λ", "ษڧ", "͠", "ͯ",
"͍", "·͢"]
» ! "ࢲػցֶशΛษڧ͍ͯ͠·͢"
47
Slide 48
Slide 48 text
Image Classification
» [#000000, #00001A, ...]
» ! [[0, 0, 0], [0, 0, 0.1], ...]
» ! "Cat" or "Burrito"
48
Slide 49
Slide 49 text
Deploy to Production
» Optimize Model
» Deploy Model
49
Slide 50
Slide 50 text
Practically, we don't run such a
graph in production
50
Slide 51
Slide 51 text
Unnecessary nodes +
No need to be variables
51
Slide 52
Slide 52 text
Training
» Define a problem (inputs and expected outputs)
» Build a computation graph
» Find the appropriate values by iterating training
steps
Inference
» Get the predicted value
52
Slide 53
Slide 53 text
Why we need to transform graph
1.To enable the model to run on mobile
» Supported operations are limited
2.For better performance
» Computer resources are restricted
» Shouldn't drain users' phone battery
53
Slide 54
Slide 54 text
1. To enable the model to run on
mobile
$ find ./tensorflow/python/ -type f | grep '.py$' | wc -l
968
$ find ./tensorflow/java/ -type f | grep '.java$' | wc -l
42
Feature Python Java
Run a predefined Graph tf.import_graph_def, tf.Session TensorFlowInferenceInterface
Graph construction Yes
Gradients tf.gradients
Functions tf.python.framework.function.Defun
Control Flow tf.cond, tf.while_loop
Neural Network library tf.train, tf.nn, tf.contrib.layers,
tf.contrib.slim
54
Slide 55
Slide 55 text
2. For better performance
Pixel 2 XL (my phone) Oppo A37f (most used)
Chipset Qualcomm MSM8996
Snapdragon 821
Qualcomm MSM8916
Snapdragon 410
CPU Quad-core (2x2.15
GHz Kryo & 2x1.6
GHz Kryo)
Quad-core 1.2 GHz
Cortex-A53
GPU Adreno 530 Adreno 306
RAM 4 GB 2 GB
There is a big gap between developers and users 55
Slide 56
Slide 56 text
Graph Transformation
Model Size !, Persormance ", (Accuracy !)
56
Slide 57
Slide 57 text
Freeze: Variable to Constant
node {
name: "x"
op: "Placeholder"
...
}
node {
name: "bias"
op: "VariableV2"
attr {
key: "container"
value {
s: ""
}
}
attr {
key: "shape"
value {
shape {
dim {
size: 1
}
}
}
}
...
57
Slide 58
Slide 58 text
Freeze: Variable to Constant
» ! Train
» ! Checkpoint file is created
» ! Freeze
$ bazel build tensorflow/python/tools:optimize_for_inference
$ bazel-bin/tensorflow/python/tools/optimize_for_inference \
--input=../logic-gate/logic-gate-python/models/and.pb \
--output=../logic-gate/logic-gate-python/models/optimized_and.pb \
--frozen_graph=True \
--input_names=x \
--output_names=y_pred
58
Slide 59
Slide 59 text
Freeze: Variable to Constant
node {
name: "x"
op: "Placeholder"
...
}
node {
name: "bias"
op: "Const"
attr {
key: "value"
value {
tensor {
dtype: DT_FLOAT
tensor_shape {
dim {
size: 1
}
}
float_val: -15.988138198852539
}
}
}
...
bias became Const with a fixed float value 59
Slide 60
Slide 60 text
Before (Training)
60
Slide 61
Slide 61 text
After (Inference)
61
Slide 62
Slide 62 text
Performance
» Computational Cost
» Model Size
62
Slide 63
Slide 63 text
Pre-trained Models for Image
Classification
» Inception, AlexNet, MobileNet, ...
63
Slide 64
Slide 64 text
Model Comparison
» Retrained Inception-v3, Retrained MobileNet,
PatchNet (Modified Inception-v3)
» Specialized for food
64
Slide 65
Slide 65 text
https://speakerdeck.com/lunardog/cooking-with-food-photos 65
Slide 66
Slide 66 text
MobileNet ! PatchNet
66
Slide 67
Slide 67 text
Performance Comparison
Retrained
Inception-v3
Retrained
MobileNet
PatchNet (Modified
Inception-v3)
Model Size 77.4 MB 11.6 MB 85.9 MB
Execution
Time
05m 44s
166ms
01m 19s
535ms
18m 34s
624ms
CPU Usage 80% 50% 80%
Memory Usage 500 MB 150 MB 500 MB
67
Slide 68
Slide 68 text
68
Slide 69
Slide 69 text
Graph Complexity
» ≈ Computational Cost
» ≈ Model Size
69
Slide 70
Slide 70 text
Inception-v3, MobileNet, PatchNet 70
Slide 71
Slide 71 text
Benchmark
$ bazel build tensorflow/tools/benchmark/benchmark_model
$ bazel-bin/tensorflow/tools/benchmark/benchmark_model \
--graph=... \
--input_layer=... \
--input_layer_shape=... \
--input_layer_type=... \
--output_layer=... \
...
71
Slide 72
Slide 72 text
Benchmark
Retrained Inception-v3 Retrained MobileNet PatchNet
Number of nodes
executed
507 264 704
AVG Timings
(microseconds)
368284 35551.5 1.28228e+06
Actual
Execution Time
05m 44s 166ms 01m 19s 535ms 18m 34s 624ms
How much faster
than PatchNet
3x faster 18x faster -
72
Slide 73
Slide 73 text
Graph Transformation
$ bazel build tensorflow/tools/graph_transforms:transform_graph
$ bazel-bin/tensorflow/tools/graph_transforms/transform_graph \
--in_graph=../models/mobilenet.pb \
--out_graph=../models/optimized-mobilenet.pb \
--inputs='input_1:0' \
--outputs='final:0' \
--transforms='
strip_unused_nodes(type=float, shape="1,224,224,3")
remove_nodes(op=Identity, op=CheckNumerics)
fold_constants(ignore_errors=true)
fold_batch_norms
fold_old_batch_norms
quantize_weights
sort_by_execution_order'
73
Slide 74
Slide 74 text
Graph Transformation
Original quantize_weights More Options
Model Size 12 MB 3.3 MB 3.1 MB
Accuracy 0.97 0.97 0.9633
Benchmark 01m 19s 535ms 01m 17s 716ms 01m 13s 816ms
CPU Usage 40-60% 40-60% 40-60%
Memory Usage 120 MB 90 MB 90 MB
Number of nodes
executed
264 264 236
” MobileNet 74
Slide 75
Slide 75 text
Accuracy
Basically, accuracy should be the same if we run the
same computation graph with the same inputs but...
» Computation Graph
» Transformed for mobile
» Data preprocessing
» Vectorization
» Feature Scaling
75
Slide 76
Slide 76 text
Feature Scaling
Make sure that features are on the same scale.
! One particular feature which has a broad range of
values governs the result "
76
Slide 77
Slide 77 text
Normalization (Number)
Rescaling
Mean normalization
Standardization
77
Slide 78
Slide 78 text
Normalization (Color)
val pixels: IntArray ! How to process this data?
78
Slide 79
Slide 79 text
Load from Asset Manager / File
constructor(assets: AssetManager) {
this.inferenceInterface = TensorFlowInferenceInterface(assets, modelName)
}
constructor(file: File) {
this.inferenceInterface = file.inputStream().use {
TensorFlowInferenceInterface(it)
}
}
79
Slide 80
Slide 80 text
Still worry about app size?
» Put the model in assets (bundling)
» Pros: No need to manage versions
» Cons: Increased app size
» Download a model into a file
» Pros: Reduced app size
» Cons: Need to manage versions
80
Slide 81
Slide 81 text
Still worry about library size?
» Splitting APK
android {
...
splits {
abi {
enable true
reset()
include 'x86_64', 'x86', 'arm64-v8a', 'armeabi-v7a', 'arm64-v8a'
}
}
project.ext.abiCodes = ['x86_64': 1, 'x86': 2, 'arm64-v8a': 3, 'armeabi-v7a': 4].withDefault {
0
}
}
android.applicationVariants.all { variant ->
variant.outputs.each { output ->
def baseAbiVersionCode = project.ext.abiCodes.get(output.getFilter(OutputFile.ABI))
if (baseAbiVersionCode != null) {
output.versionCodeOverride = baseAbiVersionCode * 1000 + variant.versionCode
}
}
}
81
Slide 82
Slide 82 text
Asynchronous download + Splitting APK
! Get power of AI by just adding 345 MB
82
Slide 83
Slide 83 text
[Summary]
» How to train a model
» Difference between TensorFlow Mobile and Lite
» How to transform a graph ! Freeze/Optimize/
benchmark
» How to implement client-side
» Model is too big? ! Consider asyncronous
download
» APK still is too big? ! Split APK
83
Slide 84
Slide 84 text
Thank you for listening!
Any questions?
! @rejasupotaro 84