Slide 1
Slide 1 text
Dive into Deep Learning -
with Scala
Sören Brunk @soebrunk
https://brunk.io
Slide 2
Slide 2 text
“AI is the new Electricity”
Andrew Ng
Slide 3
Slide 3 text
Source: https://petewarden.com/2017/11/13/deep-learning-is-eating-software/
Source: https://medium.com/intuitionmachine/is-deep-learning-software-2-0-cc7ad46b138f
Slide 4
Slide 4 text
Source: Dukesy68 CC-BY-SA 4.0
Slide 5
Slide 5 text
(might look slightly different in your country)
Slide 6
Slide 6 text
(might look slightly different in your country)
Slide 7
Slide 7 text
import awesome.deeplearn.library.magic
val result = magic(input)
(might look slightly different in your country)
Slide 8
Slide 8 text
Let's Discover Deep Learning from a Scala
Perspective
Slide 9
Slide 9 text
No content
Slide 10
Slide 10 text
Deep Learning ≈ Artificial Neural
Networks
Slide 11
Slide 11 text
Neural Network Examples
Slide 12
Slide 12 text
Computer Vision
=>
Slide 13
Slide 13 text
Speech Recognition
“The Quick brown fox jumps
over the lazy dog.”
=>
Slide 14
Slide 14 text
Machine Translation
=>
Slide 15
Slide 15 text
What do these Examples have in Common?
=> “The Quick brown fox
jumps over the lazy dog.”
=>
=>
Slide 16
Slide 16 text
=>
=>
What do these Examples have in Common?
“The Quick brown fox jumps
over the lazy dog.”
Slide 17
Slide 17 text
=>
A => B
A Neural Network is a Function
Slide 18
Slide 18 text
A Neural Network is a Function
type Model[A, B] = A => B
Slide 19
Slide 19 text
val classifier: Model[Image, ImageClass] = ???
A Neural Network is a Function
How do we implement
that thing?
Slide 20
Slide 20 text
Neural Network Mechanics
Slide 21
Slide 21 text
Neural Networks Work on Numeric Tensors
Slide 22
Slide 22 text
Representing Values as Tensors
Model[Tensor, Tensor]
encoder: A => Tensor
decoder: Tensor => B
Model[A, B]
64
432
67
62
7 9 203
90
56
64
230
54 76 4
51
83
24
8
41
40
17
5
230
98
23
7
129
34
93
73
Slide 23
Slide 23 text
The Artificial Neuron
w1
∑
x1
w2
x2
w3
x3
wn
xn
…
…
Apply
Activation
Function
y
Multiply inputs and weights
Sum up
results
Slide 24
Slide 24 text
The Artificial Neuron
w
1
∑
x
1
w
2
x
2
w
3
x
3
w
n
x
n
…
…
y
∑
Slide 25
Slide 25 text
Neural Networks are Layers of Neurons
∑
∑
Input
Layer
1. Layer 2. Layer
(Output)
∑
Slide 26
Slide 26 text
Source: https://github.com/tensorflow/models/tree/master/research/inception
Neural Networks are Layers of Neurons
Slide 27
Slide 27 text
But How Do We Set Those Weights?
∑
∑
Input
Layer
1. Layer 2. Layer
(Output)
∑
Slide 28
Slide 28 text
“What we want is a machine that can
learn from experience.”
Alan Turing, 1947
Slide 29
Slide 29 text
A Different Programming Paradigm
Traditional Programming Machine Learning
Slide 30
Slide 30 text
A Neural Network Learns its Function
def train(examples: Map[A, B]): Model[A, B]
Supervised Learning
Slide 31
Slide 31 text
Training Data: Examples with Answers
Example
Label (Answer)
(1 = cat, 0 = not cat)
1
0
1
val examples: Map[Image, Either[Cat, NotCat]]
Slide 32
Slide 32 text
Training a Neural Network
Forward Pass
Compute Loss
Backpropagation
Update Weights
Initialize
Randomly
Slide 33
Slide 33 text
Initialize Randomly
Slide 34
Slide 34 text
Initialize Randomly
Slide 35
Slide 35 text
Forward Pass Forward Pass
Compute Loss
Backpropagation
Update Weights
Slide 36
Slide 36 text
Forward Pass
Cat
Probability
= 0.4
Forward Pass
Compute Loss
Backpropagation
Update Weights
Slide 37
Slide 37 text
Compute the Loss
Loss (simplified) =
correct answer - predicted answer
Cat = 1 1
0.4
0.6
Forward Pass
Compute Loss
Backpropagation
Update Weights
Slide 38
Slide 38 text
Backpropagation Forward Pass
Compute Loss
Backpropagation
Update Weights
Slide 39
Slide 39 text
Update Weights Forward Pass
Compute Loss
Backpropagation
Update Weights
Slide 40
Slide 40 text
Update Weights Forward Pass
Compute Loss
Backpropagation
Update Weights
Slide 41
Slide 41 text
Error
Weight
Loss Function
Gradient Descent Forward Pass
Compute Loss
Backpropagation
Update Weights
Source: https://www.coursera.org/learn/machine-learning
Slide 42
Slide 42 text
What Makes Deep Learning so
powerful?
Slide 43
Slide 43 text
Performance / Accuracy
Amount of data
Traditional Machine
Learning
Deep Learning
Neural Networks Scale
Slide 44
Slide 44 text
Neural Networks Compose
Source: Vinyals et al. Show and Tell: A Neural Image Caption Generator
Slide 45
Slide 45 text
Limitations of Deep Learning
Source: https://www.technologyreview.com/s/608321/this-image-is-why-self-driving-cars-
come-loaded-with-many-types-of-sensors
Source: Brown et al, Adversarial Patch (https://arxiv.org/abs/1712.09665)
Slide 46
Slide 46 text
No content
Slide 47
Slide 47 text
Why Scala?
Prototyping vs. Production
Slide 48
Slide 48 text
Why Scala?
Type safe tensor operations
case class Tensor(
dataType: DataType,
shape: Shape
)
class TypedTensor[DataType, Shape]
val tensor: TypedTensor[Float, (Width, Height, Channel)] = ...
Slide 49
Slide 49 text
Deep Learning Libraries for Scala
Slide 50
Slide 50 text
Deep Learning Libraries for Scala
Source: https://brunk.io/deep-learning-in-scala-part-1-basics-and-libraries.html
• Mature, Large
Community &
Commercial Support
• Java API
• Native Scala API
(ScalNet) in alpha
• Scala API part of main
project
• API very pythonic
• Most idiomatic & type
safe Scala API
• Not part of main
TensorFlow (yet)
Slide 51
Slide 51 text
Deep Learning Libraries for Scala
Source: https://brunk.io/deep-learning-in-scala-part-1-basics-and-libraries.html
• Mature, Large
Community &
Commercial Support
• Java API
• Native Scala API
(ScalNet) in alpha
• Scala API part of main
project
• API very pythonic
• Most idiomatic & type
safe Scala API
• Not part of main
TensorFlow (yet)
Slide 52
Slide 52 text
No content
Slide 53
Slide 53 text
? => ?
1. Frame Your Problem
Slide 54
Slide 54 text
1. Frame Your Problem
=>
Slide 55
Slide 55 text
1. Frame Your Problem
=>Either[ScalaLogo,
NotScalaLogo]
Slide 56
Slide 56 text
2. Look for Existing Solutions
Slide 57
Slide 57 text
3. Collect and Prepare Training Data
Filter Resize Normalize Vectorize ...
encoder: A => Tensor
decoder: Tensor => B
Training Set Test Set
Slide 58
Slide 58 text
3. Collect and Prepare Training Data
def readImage(filename: Output): Output = {
val rawImage = tf.data.readFile(filename)
val image = tf.image.decodeJpeg(rawImage, numChannels = 3)
tf.image.resizeBilinear(image.expandDims(axis=0), Seq(250, 250))
.squeeze(Seq(0)).cast(UINT8)
}
val trainData: Dataset[(Tensor, Tensor), (Output, Output),
(DataType, DataType), (Shape, Shape)] =
tf.data.TensorSlicesDataset(filenamesWithLabels(trainDir))
.shuffle(bufferSize = 30000, Some(seed))
.map({ case (filename, label) => (readImage(filename), label)})
.repeat()
.batch(128)
.prefetch(100)
Slide 59
Slide 59 text
4. Choose the Right Architecture
By Aphex34 - Own work, CC BY-SA 4.0, https://commons.wikimedia.org/w/index.php?curid=45679374
Source: https://karpathy.github.io/2015/05/21/rnn-effectiveness/
Fully Connected Neural Network Convolutional Neural Network
Recurrent Neural Network
∑
∑
∑
Slide 60
Slide 60 text
4. Choose the Right Architecture
By Aphex34 - Own work, CC BY-SA 4.0, https://commons.wikimedia.org/w/index.php?curid=45679374
1. Layer
2. Layer
3. Layer
4. Layer
Convolutional Neural Network
Slide 61
Slide 61 text
5. Train, Evaluate, Refine
Create /
Improve
Train
Evaluate
The Model
Development
Cycle
Slide 62
Slide 62 text
Build a Simple Model
Source: https://www.flickr.com/photos/foto_db/12163399363 CC-BY-SA
Slide 63
Slide 63 text
Build a Simple Model
import org.platanios.tensorflow.api._
val input = tf.learn.Input(UINT8, Shape(-1, 250, 250, 3)) // type and shape of our images
val labelInput = tf.learn.Input(UINT8, Shape(-1)) // type and shape of our labels
val layer =
tf.learn.Flatten("Input/Flatten") >> // flatten the images into a single vector
tf.learn.Cast("Input/Cast", FLOAT32) >> // cast input to float
tf.learn.Linear("Layer_1/Linear", units = 64) >> // hidden layer with 512 neurons
tf.learn.ReLU("Layer_1/ReLU“, 0.01f) >> // hidden layer activation
tf.learn.Linear("OutputLayer/Linear", units = 2) // output layer
val trainInputLayer = tf.learn.Cast("TrainInput/Cast", INT64) // cast labels to long
val loss =
tf.learn.SparseSoftmaxCrossEntropy("Loss/CrossEntropy") >> // loss/error function
tf.learn.Mean("Loss/Mean") >>
tf.learn.ScalarSummary("Loss/Summary", "Loss")
val optimizer = tf.train.GradientDescent(learningRate = 0.001) // the optimizer updates our weights
// create a Model for training
val model = tf.learn.Model.supervised(input, layer, labelInput, trainInputLayer, loss, optimizer)
Slide 64
Slide 64 text
Train and Evaluate the Model
val estimator = tf.learn.InMemoryEstimator(model)
estimator.train(() => trainData)
Learn / Hooks / Evaluation - Step 0 Evaluator:
Learn / Hooks / Evaluation - ╔═══════╤════════════╗
Learn / Hooks / Evaluation - ║ │ Accuracy ║
Learn / Hooks / Evaluation - ╟───────┼────────────╢
Learn / Hooks / Evaluation - ║ Train │ 0.1219 ║
Learn / Hooks / Evaluation - ║ Test │ 0.1217 ║
Learn / Hooks / Evaluation - ╚═══════╧════════════╝
Learn / Hooks / Loss Logger - ( 2.831 s) Step: 100, Loss: 2250.2437
Learn / Hooks / Loss Logger - ( 0.561 s) Step: 200, Loss: 2735.0537
...
Learn / Hooks / Loss Logger - ( 0.537 s) Step: 900, Loss: 877.2033
Learn / Hooks / Loss Logger - ( 0.699 s) Step: 1000, Loss: 1207.4133
Learn / Hooks / Evaluation - Step 1000 Evaluator:
Learn / Hooks / Evaluation - ╔═══════╤════════════╗
Learn / Hooks / Evaluation - ║ │ Accuracy ║
Learn / Hooks / Evaluation - ╟───────┼────────────╢
Learn / Hooks / Evaluation - ║ Train │ 0.8105 ║
Learn / Hooks / Evaluation - ║ Test │ 0.7888 ║
Learn / Hooks / Evaluation - ╚═══════╧════════════╝
Learn / Hooks / Loss Logger - ( 2.599 s) Step: 1100, Loss: 989.0901
Learn / Hooks / Loss Logger - ( 0.529 s) Step: 1200, Loss: 440.6418
...
Slide 65
Slide 65 text
Refine the Model
val layer = tf.learn.Cast("Input/Cast", FLOAT32) >>
tf.learn.Flatten("Input/Flatten") >>
tf.learn.Linear("Layer_1/Linear", units = 64) >> // hidden layer
tf.learn.ReLU("Layer_1/ReLU", 0.01f) >> // hidden layer activation
tf.learn.Linear("OutputLayer/Linear", units = 2) // output layer
Slide 66
Slide 66 text
Refine the Model
val layer = tf.learn.Cast("Input/Cast", FLOAT32) >>
tf.learn.Flatten("Input/Flatten") >>
tf.learn.Linear("Layer_1/Linear", units = 128) >> // hidden layer
tf.learn.ReLU("Layer_1/ReLU", 0.01f) >> // hidden layer activation
tf.learn.Linear("OutputLayer/Linear", units = 2) // output layer
Slide 67
Slide 67 text
Refine the Model
val layer = tf.learn.Cast("Input/Cast", FLOAT32) >>
tf.learn.Flatten("Input/Flatten") >>
tf.learn.Linear("Layer_1/Linear", units = 128) >> // hidden layer
tf.learn.ReLU("Layer_1/ReLU", 0.01f) >> // hidden layer activation
tf.learn.Linear("OutputLayer/Linear", units = 2) // output layer
val estimator = tf.learn.InMemoryEstimator(model)
estimator.train(() => trainData)
Slide 68
Slide 68 text
Refine the Model
val layer = tf.learn.Cast("Input/Cast", FLOAT32) >>
tf.learn.Flatten("Input/Flatten") >>
tf.learn.Linear("Layer_1/Linear", units = 512) >> // hidden layer
tf.learn.ReLU("Layer_1/ReLU", 0.01f) >> // hidden layer activation
tf.learn.Linear("OutputLayer/Linear", units = 2) // output layer
val estimator = tf.learn.InMemoryEstimator(model)
estimator.train(() => trainData)
Slide 69
Slide 69 text
Refine the Model
tf.learn.Conv2D("Layer_0/Conv2D", Shape(3, 3, 1, 32), stride1 = 2, stride2 = 2, ValidConvPadding) >>
tf.learn.AddBias("Layer_0/Bias") >>
tf.learn.ReLU("Layer_0/ReLU", 0.01f) >>
tf.learn.MaxPool("Layer_0/MaxPool", windowSize = Seq(1, 2, 2, 1), stride1 = 2, stride2 = 2, ValidConvPadding) >>
val layer = tf.learn.Cast("Input/Cast", FLOAT32) >>
tf.learn.Flatten("Input/Flatten") >>
tf.learn.Linear("Layer_1/Linear", units = 512) >>
tf.learn.ReLU("Layer_1/ReLU") >>
tf.learn.Linear("OutputLayer/Linear", units = 2)
Slide 70
Slide 70 text
Refine the Model
tf.learn.Conv2D("Layer_0/Conv2D", Shape(3, 3, 1, 32), stride1 = 2, stride2 = 2, ValidConvPadding) >>
tf.learn.AddBias("Layer_0/Bias") >>
tf.learn.ReLU("Layer_0/ReLU", 0.01f) >>
tf.learn.MaxPool("Layer_0/MaxPool", windowSize = Seq(1, 2, 2, 1), stride1 = 2, stride2 = 2, ValidConvPadding) >>
val layer = tf.learn.Cast("Input/Cast", FLOAT32) >>
tf.learn.Flatten("Input/Flatten") >>
tf.learn.Linear("Layer_1/Linear", units = 512) >>
tf.learn.ReLU("Layer_1/ReLU") >>
tf.learn.Linear("OutputLayer/Linear", units = 2)
val estimator = tf.learn.InMemoryEstimator(model)
estimator.train(() => trainData)
Slide 71
Slide 71 text
Run the Model
val image: Tensor = readImage(filename)
val decoder = Seq("not_scala", "scala")
val result: Tensor = estimator.infer(() => image)
val label = getLabel(result))
drawLabel(image, s"Class: $label ($decoder(label))")
Slide 72
Slide 72 text
7. Deploy to Production
Slide 73
Slide 73 text
Takeaways
● Neural networks are functions
● We usually learn them from known examples
● Training can be challenging
● Scala is a great language for Deep Learning
● Interested? Give it a try
● Get involved to make it even better
Slide 74
Slide 74 text
Thank You for Listening
Sören Brunk @soebrunk
https://brunk.io