Dive into Deep Learning with Scala

Deep learning is an emerging field with the potential to transform whole industries, and to significantly affect software development.
This talk is an introduction to deep learning for Scala developers. We'll see why deep learning enables a new programming paradigm, and how it's related to functional programming.
We'll discuss what makes Scala an interesting choice for deep learning, and look at the current landscape of deep learning libraries for Scala. Equipped with that knowledge, we'll walk through a concrete example that shows you how to create, train and deploy a neural network using Scala.

Talk from Scala Days Europe 2018.

Transcript

1. Dive into Deep Learning - with Scala Sören Brunk @soebrunk https://brunk.io
2. “AI is the new Electricity” Andrew Ng
3. Source: https://petewarden.com/2017/11/13/deep-learning-is-eating-software/ Source: https://medium.com/intuitionmachine/is-deep-learning-software-2-0-cc7ad46b138f
4. Source: Dukesy68 CC-BY-SA 4.0
5. (might look slightly different in your country)
6. (might look slightly different in your country)
7. import awesome.deeplearn.library.magic val result = magic(input) (might look slightly different in your country)
8. Let's Discover Deep Learning from a Scala Perspective
10. Deep Learning ≈ Artificial Neural Networks
11. Neural Network Examples
12. Computer Vision =>
13. Speech Recognition “The Quick brown fox jumps over the lazy dog.” =>
14. Machine Translation =>
15. What do these Examples have in Common? => “The Quick brown fox jumps over the lazy dog.” => =>
16. => => What do these Examples have in Common? “The Quick brown fox jumps over the lazy dog.”
17. => A => B A Neural Network is a Function
18. A Neural Network is a Function type Model[A, B] = A => B
19. val classifier: Model[Image, ImageClass] = ??? A Neural Network is a Function How do we implement that thing?
20. Neural Network Mechanics
21. Neural Networks Work on Numeric Tensors
22. Representing Values as Tensors Model[Tensor, Tensor] encoder: A => Tensor decoder: Tensor => B Model[A, B] 64 432 67 62 7 9 203 90 56 64 230 54 76 4 51 83 24 8 41 40 17 5 230 98 23 7 129 34 93 73
23. The Artificial Neuron w1 ∑ x1 w2 x2 w3 x3 wn xn … … Apply Activation Function y Multiply inputs and weights Sum up results
24. The Artificial Neuron w 1 ∑ x 1 w 2 x 2 w 3 x 3 w n x n … … y ∑
25. Neural Networks are Layers of Neurons ∑ ∑ Input Layer 1. Layer 2. Layer (Output) ∑
26. Source: https://github.com/tensorflow/models/tree/master/research/inception Neural Networks are Layers of Neurons
27. But How Do We Set Those Weights? ∑ ∑ Input Layer 1. Layer 2. Layer (Output) ∑
28. “What we want is a machine that can learn from experience.” Alan Turing, 1947
29. A Different Programming Paradigm Traditional Programming Machine Learning
30. A Neural Network Learns its Function def train(examples: Map[A, B]): Model[A, B] Supervised Learning
31. Training Data: Examples with Answers Example Label (Answer) (1 = cat, 0 = not cat) 1 0 1 val examples: Map[Image, Either[Cat, NotCat]]
32. Training a Neural Network Forward Pass Compute Loss Backpropagation Update Weights Initialize Randomly
33. Initialize Randomly
34. Initialize Randomly
35. Forward Pass Forward Pass Compute Loss Backpropagation Update Weights
36. Forward Pass Cat Probability = 0.4 Forward Pass Compute Loss Backpropagation Update Weights
37. Compute the Loss Loss (simplified) = correct answer - predicted answer Cat = 1 1 0.4 0.6 Forward Pass Compute Loss Backpropagation Update Weights
38. Backpropagation Forward Pass Compute Loss Backpropagation Update Weights
39. Update Weights Forward Pass Compute Loss Backpropagation Update Weights
40. Update Weights Forward Pass Compute Loss Backpropagation Update Weights
41. Error Weight Loss Function Gradient Descent Forward Pass Compute Loss Backpropagation Update Weights Source: https://www.coursera.org/learn/machine-learning
42. What Makes Deep Learning so powerful?
43. Performance / Accuracy Amount of data Traditional Machine Learning Deep Learning Neural Networks Scale
44. Neural Networks Compose Source: Vinyals et al. Show and Tell: A Neural Image Caption Generator
45. Limitations of Deep Learning Source: https://www.technologyreview.com/s/608321/this-image-is-why-self-driving-cars- come-loaded-with-many-types-of-sensors Source: Brown et al, Adversarial Patch (https://arxiv.org/abs/1712.09665)
47. Why Scala? Prototyping vs. Production
48. Why Scala? Type safe tensor operations case class Tensor( dataType: DataType, shape: Shape ) class TypedTensor[DataType, Shape] val tensor: TypedTensor[Float, (Width, Height, Channel)] = ...
49. Deep Learning Libraries for Scala
50. Deep Learning Libraries for Scala Source: https://brunk.io/deep-learning-in-scala-part-1-basics-and-libraries.html • Mature, Large Community & Commercial Support • Java API • Native Scala API (ScalNet) in alpha • Scala API part of main project • API very pythonic • Most idiomatic & type safe Scala API • Not part of main TensorFlow (yet)
51. Deep Learning Libraries for Scala Source: https://brunk.io/deep-learning-in-scala-part-1-basics-and-libraries.html • Mature, Large Community & Commercial Support • Java API • Native Scala API (ScalNet) in alpha • Scala API part of main project • API very pythonic • Most idiomatic & type safe Scala API • Not part of main TensorFlow (yet)
53. ? => ? 1. Frame Your Problem
54. 1. Frame Your Problem =>
55. 1. Frame Your Problem =>Either[ScalaLogo, NotScalaLogo]
56. 2. Look for Existing Solutions
57. 3. Collect and Prepare Training Data Filter Resize Normalize Vectorize ... encoder: A => Tensor decoder: Tensor => B Training Set Test Set
58. 3. Collect and Prepare Training Data def readImage(filename: Output): Output = { val rawImage = tf.data.readFile(filename) val image = tf.image.decodeJpeg(rawImage, numChannels = 3) tf.image.resizeBilinear(image.expandDims(axis=0), Seq(250, 250)) .squeeze(Seq(0)).cast(UINT8) } val trainData: Dataset[(Tensor, Tensor), (Output, Output), (DataType, DataType), (Shape, Shape)] = tf.data.TensorSlicesDataset(filenamesWithLabels(trainDir)) .shuffle(bufferSize = 30000, Some(seed)) .map({ case (filename, label) => (readImage(filename), label)}) .repeat() .batch(128) .prefetch(100)
59. 4. Choose the Right Architecture By Aphex34 - Own work, CC BY-SA 4.0, https://commons.wikimedia.org/w/index.php?curid=45679374 Source: https://karpathy.github.io/2015/05/21/rnn-effectiveness/ Fully Connected Neural Network Convolutional Neural Network Recurrent Neural Network ∑ ∑ ∑
60. 4. Choose the Right Architecture By Aphex34 - Own work, CC BY-SA 4.0, https://commons.wikimedia.org/w/index.php?curid=45679374 1. Layer 2. Layer 3. Layer 4. Layer Convolutional Neural Network
61. 5. Train, Evaluate, Refine Create / Improve Train Evaluate The Model Development Cycle
62. Build a Simple Model Source: https://www.flickr.com/photos/foto_db/12163399363 CC-BY-SA
63. Build a Simple Model import org.platanios.tensorflow.api._ val input = tf.learn.Input(UINT8, Shape(-1, 250, 250, 3)) // type and shape of our images val labelInput = tf.learn.Input(UINT8, Shape(-1)) // type and shape of our labels val layer = tf.learn.Flatten("Input/Flatten") >> // flatten the images into a single vector tf.learn.Cast("Input/Cast", FLOAT32) >> // cast input to float tf.learn.Linear("Layer_1/Linear", units = 64) >> // hidden layer with 512 neurons tf.learn.ReLU("Layer_1/ReLU“, 0.01f) >> // hidden layer activation tf.learn.Linear("OutputLayer/Linear", units = 2) // output layer val trainInputLayer = tf.learn.Cast("TrainInput/Cast", INT64) // cast labels to long val loss = tf.learn.SparseSoftmaxCrossEntropy("Loss/CrossEntropy") >> // loss/error function tf.learn.Mean("Loss/Mean") >> tf.learn.ScalarSummary("Loss/Summary", "Loss") val optimizer = tf.train.GradientDescent(learningRate = 0.001) // the optimizer updates our weights // create a Model for training val model = tf.learn.Model.supervised(input, layer, labelInput, trainInputLayer, loss, optimizer)
64. Train and Evaluate the Model val estimator = tf.learn.InMemoryEstimator(model) estimator.train(() => trainData) Learn / Hooks / Evaluation - Step 0 Evaluator: Learn / Hooks / Evaluation - ╔═══════╤════════════╗ Learn / Hooks / Evaluation - ║ │ Accuracy ║ Learn / Hooks / Evaluation - ╟───────┼────────────╢ Learn / Hooks / Evaluation - ║ Train │ 0.1219 ║ Learn / Hooks / Evaluation - ║ Test │ 0.1217 ║ Learn / Hooks / Evaluation - ╚═══════╧════════════╝ Learn / Hooks / Loss Logger - ( 2.831 s) Step: 100, Loss: 2250.2437 Learn / Hooks / Loss Logger - ( 0.561 s) Step: 200, Loss: 2735.0537 ... Learn / Hooks / Loss Logger - ( 0.537 s) Step: 900, Loss: 877.2033 Learn / Hooks / Loss Logger - ( 0.699 s) Step: 1000, Loss: 1207.4133 Learn / Hooks / Evaluation - Step 1000 Evaluator: Learn / Hooks / Evaluation - ╔═══════╤════════════╗ Learn / Hooks / Evaluation - ║ │ Accuracy ║ Learn / Hooks / Evaluation - ╟───────┼────────────╢ Learn / Hooks / Evaluation - ║ Train │ 0.8105 ║ Learn / Hooks / Evaluation - ║ Test │ 0.7888 ║ Learn / Hooks / Evaluation - ╚═══════╧════════════╝ Learn / Hooks / Loss Logger - ( 2.599 s) Step: 1100, Loss: 989.0901 Learn / Hooks / Loss Logger - ( 0.529 s) Step: 1200, Loss: 440.6418 ...
65. Refine the Model val layer = tf.learn.Cast("Input/Cast", FLOAT32) >> tf.learn.Flatten("Input/Flatten") >> tf.learn.Linear("Layer_1/Linear", units = 64) >> // hidden layer tf.learn.ReLU("Layer_1/ReLU", 0.01f) >> // hidden layer activation tf.learn.Linear("OutputLayer/Linear", units = 2) // output layer
66. Refine the Model val layer = tf.learn.Cast("Input/Cast", FLOAT32) >> tf.learn.Flatten("Input/Flatten") >> tf.learn.Linear("Layer_1/Linear", units = 128) >> // hidden layer tf.learn.ReLU("Layer_1/ReLU", 0.01f) >> // hidden layer activation tf.learn.Linear("OutputLayer/Linear", units = 2) // output layer
67. Refine the Model val layer = tf.learn.Cast("Input/Cast", FLOAT32) >> tf.learn.Flatten("Input/Flatten") >> tf.learn.Linear("Layer_1/Linear", units = 128) >> // hidden layer tf.learn.ReLU("Layer_1/ReLU", 0.01f) >> // hidden layer activation tf.learn.Linear("OutputLayer/Linear", units = 2) // output layer val estimator = tf.learn.InMemoryEstimator(model) estimator.train(() => trainData)
68. Refine the Model val layer = tf.learn.Cast("Input/Cast", FLOAT32) >> tf.learn.Flatten("Input/Flatten") >> tf.learn.Linear("Layer_1/Linear", units = 512) >> // hidden layer tf.learn.ReLU("Layer_1/ReLU", 0.01f) >> // hidden layer activation tf.learn.Linear("OutputLayer/Linear", units = 2) // output layer val estimator = tf.learn.InMemoryEstimator(model) estimator.train(() => trainData)
69. Refine the Model tf.learn.Conv2D("Layer_0/Conv2D", Shape(3, 3, 1, 32), stride1 = 2, stride2 = 2, ValidConvPadding) >> tf.learn.AddBias("Layer_0/Bias") >> tf.learn.ReLU("Layer_0/ReLU", 0.01f) >> tf.learn.MaxPool("Layer_0/MaxPool", windowSize = Seq(1, 2, 2, 1), stride1 = 2, stride2 = 2, ValidConvPadding) >> val layer = tf.learn.Cast("Input/Cast", FLOAT32) >> tf.learn.Flatten("Input/Flatten") >> tf.learn.Linear("Layer_1/Linear", units = 512) >> tf.learn.ReLU("Layer_1/ReLU") >> tf.learn.Linear("OutputLayer/Linear", units = 2)
70. Refine the Model tf.learn.Conv2D("Layer_0/Conv2D", Shape(3, 3, 1, 32), stride1 = 2, stride2 = 2, ValidConvPadding) >> tf.learn.AddBias("Layer_0/Bias") >> tf.learn.ReLU("Layer_0/ReLU", 0.01f) >> tf.learn.MaxPool("Layer_0/MaxPool", windowSize = Seq(1, 2, 2, 1), stride1 = 2, stride2 = 2, ValidConvPadding) >> val layer = tf.learn.Cast("Input/Cast", FLOAT32) >> tf.learn.Flatten("Input/Flatten") >> tf.learn.Linear("Layer_1/Linear", units = 512) >> tf.learn.ReLU("Layer_1/ReLU") >> tf.learn.Linear("OutputLayer/Linear", units = 2) val estimator = tf.learn.InMemoryEstimator(model) estimator.train(() => trainData)
71. Run the Model val image: Tensor = readImage(filename) val decoder = Seq("not_scala", "scala") val result: Tensor = estimator.infer(() => image) val label = getLabel(result)) drawLabel(image, s"Class: $label ($decoder(label))")
72. 7. Deploy to Production
73. Takeaways • Neural networks are functions • We usually learn them from known examples • Training can be challenging • Scala is a great language for Deep Learning • Interested? Give it a try • Get involved to make it even better
74. Thank You for Listening Sören Brunk @soebrunk https://brunk.io