Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Dive into Deep Learning with Scala

A3820a925274dd585cf62f4c6371ca91?s=47 Sören Brunk
May 16, 2018
6.3k

Dive into Deep Learning with Scala

Deep learning is an emerging field with the potential to transform whole industries, and to significantly affect software development.
This talk is an introduction to deep learning for Scala developers. We'll see why deep learning enables a new programming paradigm, and how it's related to functional programming.
We'll discuss what makes Scala an interesting choice for deep learning, and look at the current landscape of deep learning libraries for Scala. Equipped with that knowledge, we'll walk through a concrete example that shows you how to create, train and deploy a neural network using Scala.

Talk from Scala Days Europe 2018.

A3820a925274dd585cf62f4c6371ca91?s=128

Sören Brunk

May 16, 2018
Tweet

Transcript

  1. Dive into Deep Learning - with Scala Sören Brunk @soebrunk

    https://brunk.io
  2. “AI is the new Electricity” Andrew Ng

  3. Source: https://petewarden.com/2017/11/13/deep-learning-is-eating-software/ Source: https://medium.com/intuitionmachine/is-deep-learning-software-2-0-cc7ad46b138f

  4. Source: Dukesy68 CC-BY-SA 4.0

  5. (might look slightly different in your country)

  6. (might look slightly different in your country)

  7. import awesome.deeplearn.library.magic val result = magic(input) (might look slightly different

    in your country)
  8. Let's Discover Deep Learning from a Scala Perspective

  9. None
  10. Deep Learning ≈ Artificial Neural Networks

  11. Neural Network Examples

  12. Computer Vision =>

  13. Speech Recognition “The Quick brown fox jumps over the lazy

    dog.” =>
  14. Machine Translation =>

  15. What do these Examples have in Common? => “The Quick

    brown fox jumps over the lazy dog.” => =>
  16. => => What do these Examples have in Common? “The

    Quick brown fox jumps over the lazy dog.”
  17. => A => B A Neural Network is a Function

  18. A Neural Network is a Function type Model[A, B] =

    A => B
  19. val classifier: Model[Image, ImageClass] = ??? A Neural Network is

    a Function How do we implement that thing?
  20. Neural Network Mechanics

  21. Neural Networks Work on Numeric Tensors

  22. Representing Values as Tensors Model[Tensor, Tensor] encoder: A => Tensor

    decoder: Tensor => B Model[A, B] 64 432 67 62 7 9 203 90 56 64 230 54 76 4 51 83 24 8 41 40 17 5 230 98 23 7 129 34 93 73
  23. The Artificial Neuron w1 ∑ x1 w2 x2 w3 x3

    wn xn … … Apply Activation Function y Multiply inputs and weights Sum up results
  24. The Artificial Neuron w 1 ∑ x 1 w 2

    x 2 w 3 x 3 w n x n … … y ∑
  25. Neural Networks are Layers of Neurons ∑ ∑ Input Layer

    1. Layer 2. Layer (Output) ∑
  26. Source: https://github.com/tensorflow/models/tree/master/research/inception Neural Networks are Layers of Neurons

  27. But How Do We Set Those Weights? ∑ ∑ Input

    Layer 1. Layer 2. Layer (Output) ∑
  28. “What we want is a machine that can learn from

    experience.” Alan Turing, 1947
  29. A Different Programming Paradigm Traditional Programming Machine Learning

  30. A Neural Network Learns its Function def train(examples: Map[A, B]):

    Model[A, B] Supervised Learning
  31. Training Data: Examples with Answers Example Label (Answer) (1 =

    cat, 0 = not cat) 1 0 1 val examples: Map[Image, Either[Cat, NotCat]]
  32. Training a Neural Network Forward Pass Compute Loss Backpropagation Update

    Weights Initialize Randomly
  33. Initialize Randomly

  34. Initialize Randomly

  35. Forward Pass Forward Pass Compute Loss Backpropagation Update Weights

  36. Forward Pass Cat Probability = 0.4 Forward Pass Compute Loss

    Backpropagation Update Weights
  37. Compute the Loss Loss (simplified) = correct answer - predicted

    answer Cat = 1 1 0.4 0.6 Forward Pass Compute Loss Backpropagation Update Weights
  38. Backpropagation Forward Pass Compute Loss Backpropagation Update Weights

  39. Update Weights Forward Pass Compute Loss Backpropagation Update Weights

  40. Update Weights Forward Pass Compute Loss Backpropagation Update Weights

  41. Error Weight Loss Function Gradient Descent Forward Pass Compute Loss

    Backpropagation Update Weights Source: https://www.coursera.org/learn/machine-learning
  42. What Makes Deep Learning so powerful?

  43. Performance / Accuracy Amount of data Traditional Machine Learning Deep

    Learning Neural Networks Scale
  44. Neural Networks Compose Source: Vinyals et al. Show and Tell:

    A Neural Image Caption Generator
  45. Limitations of Deep Learning Source: https://www.technologyreview.com/s/608321/this-image-is-why-self-driving-cars- come-loaded-with-many-types-of-sensors Source: Brown et

    al, Adversarial Patch (https://arxiv.org/abs/1712.09665)
  46. None
  47. Why Scala? Prototyping vs. Production

  48. Why Scala? Type safe tensor operations case class Tensor( dataType:

    DataType, shape: Shape ) class TypedTensor[DataType, Shape] val tensor: TypedTensor[Float, (Width, Height, Channel)] = ...
  49. Deep Learning Libraries for Scala

  50. Deep Learning Libraries for Scala Source: https://brunk.io/deep-learning-in-scala-part-1-basics-and-libraries.html • Mature, Large

    Community & Commercial Support • Java API • Native Scala API (ScalNet) in alpha • Scala API part of main project • API very pythonic • Most idiomatic & type safe Scala API • Not part of main TensorFlow (yet)
  51. Deep Learning Libraries for Scala Source: https://brunk.io/deep-learning-in-scala-part-1-basics-and-libraries.html • Mature, Large

    Community & Commercial Support • Java API • Native Scala API (ScalNet) in alpha • Scala API part of main project • API very pythonic • Most idiomatic & type safe Scala API • Not part of main TensorFlow (yet)
  52. None
  53. ? => ? 1. Frame Your Problem

  54. 1. Frame Your Problem =>

  55. 1. Frame Your Problem =>Either[ScalaLogo, NotScalaLogo]

  56. 2. Look for Existing Solutions

  57. 3. Collect and Prepare Training Data Filter Resize Normalize Vectorize

    ... encoder: A => Tensor decoder: Tensor => B Training Set Test Set
  58. 3. Collect and Prepare Training Data def readImage(filename: Output): Output

    = { val rawImage = tf.data.readFile(filename) val image = tf.image.decodeJpeg(rawImage, numChannels = 3) tf.image.resizeBilinear(image.expandDims(axis=0), Seq(250, 250)) .squeeze(Seq(0)).cast(UINT8) } val trainData: Dataset[(Tensor, Tensor), (Output, Output), (DataType, DataType), (Shape, Shape)] = tf.data.TensorSlicesDataset(filenamesWithLabels(trainDir)) .shuffle(bufferSize = 30000, Some(seed)) .map({ case (filename, label) => (readImage(filename), label)}) .repeat() .batch(128) .prefetch(100)
  59. 4. Choose the Right Architecture By Aphex34 - Own work,

    CC BY-SA 4.0, https://commons.wikimedia.org/w/index.php?curid=45679374 Source: https://karpathy.github.io/2015/05/21/rnn-effectiveness/ Fully Connected Neural Network Convolutional Neural Network Recurrent Neural Network ∑ ∑ ∑
  60. 4. Choose the Right Architecture By Aphex34 - Own work,

    CC BY-SA 4.0, https://commons.wikimedia.org/w/index.php?curid=45679374 1. Layer 2. Layer 3. Layer 4. Layer Convolutional Neural Network
  61. 5. Train, Evaluate, Refine Create / Improve Train Evaluate The

    Model Development Cycle
  62. Build a Simple Model Source: https://www.flickr.com/photos/foto_db/12163399363 CC-BY-SA

  63. Build a Simple Model import org.platanios.tensorflow.api._ val input = tf.learn.Input(UINT8,

    Shape(-1, 250, 250, 3)) // type and shape of our images val labelInput = tf.learn.Input(UINT8, Shape(-1)) // type and shape of our labels val layer = tf.learn.Flatten("Input/Flatten") >> // flatten the images into a single vector tf.learn.Cast("Input/Cast", FLOAT32) >> // cast input to float tf.learn.Linear("Layer_1/Linear", units = 64) >> // hidden layer with 512 neurons tf.learn.ReLU("Layer_1/ReLU“, 0.01f) >> // hidden layer activation tf.learn.Linear("OutputLayer/Linear", units = 2) // output layer val trainInputLayer = tf.learn.Cast("TrainInput/Cast", INT64) // cast labels to long val loss = tf.learn.SparseSoftmaxCrossEntropy("Loss/CrossEntropy") >> // loss/error function tf.learn.Mean("Loss/Mean") >> tf.learn.ScalarSummary("Loss/Summary", "Loss") val optimizer = tf.train.GradientDescent(learningRate = 0.001) // the optimizer updates our weights // create a Model for training val model = tf.learn.Model.supervised(input, layer, labelInput, trainInputLayer, loss, optimizer)
  64. Train and Evaluate the Model val estimator = tf.learn.InMemoryEstimator(model) estimator.train(()

    => trainData) Learn / Hooks / Evaluation - Step 0 Evaluator: Learn / Hooks / Evaluation - ╔═══════╤════════════╗ Learn / Hooks / Evaluation - ║ │ Accuracy ║ Learn / Hooks / Evaluation - ╟───────┼────────────╢ Learn / Hooks / Evaluation - ║ Train │ 0.1219 ║ Learn / Hooks / Evaluation - ║ Test │ 0.1217 ║ Learn / Hooks / Evaluation - ╚═══════╧════════════╝ Learn / Hooks / Loss Logger - ( 2.831 s) Step: 100, Loss: 2250.2437 Learn / Hooks / Loss Logger - ( 0.561 s) Step: 200, Loss: 2735.0537 ... Learn / Hooks / Loss Logger - ( 0.537 s) Step: 900, Loss: 877.2033 Learn / Hooks / Loss Logger - ( 0.699 s) Step: 1000, Loss: 1207.4133 Learn / Hooks / Evaluation - Step 1000 Evaluator: Learn / Hooks / Evaluation - ╔═══════╤════════════╗ Learn / Hooks / Evaluation - ║ │ Accuracy ║ Learn / Hooks / Evaluation - ╟───────┼────────────╢ Learn / Hooks / Evaluation - ║ Train │ 0.8105 ║ Learn / Hooks / Evaluation - ║ Test │ 0.7888 ║ Learn / Hooks / Evaluation - ╚═══════╧════════════╝ Learn / Hooks / Loss Logger - ( 2.599 s) Step: 1100, Loss: 989.0901 Learn / Hooks / Loss Logger - ( 0.529 s) Step: 1200, Loss: 440.6418 ...
  65. Refine the Model val layer = tf.learn.Cast("Input/Cast", FLOAT32) >> tf.learn.Flatten("Input/Flatten")

    >> tf.learn.Linear("Layer_1/Linear", units = 64) >> // hidden layer tf.learn.ReLU("Layer_1/ReLU", 0.01f) >> // hidden layer activation tf.learn.Linear("OutputLayer/Linear", units = 2) // output layer
  66. Refine the Model val layer = tf.learn.Cast("Input/Cast", FLOAT32) >> tf.learn.Flatten("Input/Flatten")

    >> tf.learn.Linear("Layer_1/Linear", units = 128) >> // hidden layer tf.learn.ReLU("Layer_1/ReLU", 0.01f) >> // hidden layer activation tf.learn.Linear("OutputLayer/Linear", units = 2) // output layer
  67. Refine the Model val layer = tf.learn.Cast("Input/Cast", FLOAT32) >> tf.learn.Flatten("Input/Flatten")

    >> tf.learn.Linear("Layer_1/Linear", units = 128) >> // hidden layer tf.learn.ReLU("Layer_1/ReLU", 0.01f) >> // hidden layer activation tf.learn.Linear("OutputLayer/Linear", units = 2) // output layer val estimator = tf.learn.InMemoryEstimator(model) estimator.train(() => trainData)
  68. Refine the Model val layer = tf.learn.Cast("Input/Cast", FLOAT32) >> tf.learn.Flatten("Input/Flatten")

    >> tf.learn.Linear("Layer_1/Linear", units = 512) >> // hidden layer tf.learn.ReLU("Layer_1/ReLU", 0.01f) >> // hidden layer activation tf.learn.Linear("OutputLayer/Linear", units = 2) // output layer val estimator = tf.learn.InMemoryEstimator(model) estimator.train(() => trainData)
  69. Refine the Model tf.learn.Conv2D("Layer_0/Conv2D", Shape(3, 3, 1, 32), stride1 =

    2, stride2 = 2, ValidConvPadding) >> tf.learn.AddBias("Layer_0/Bias") >> tf.learn.ReLU("Layer_0/ReLU", 0.01f) >> tf.learn.MaxPool("Layer_0/MaxPool", windowSize = Seq(1, 2, 2, 1), stride1 = 2, stride2 = 2, ValidConvPadding) >> val layer = tf.learn.Cast("Input/Cast", FLOAT32) >> tf.learn.Flatten("Input/Flatten") >> tf.learn.Linear("Layer_1/Linear", units = 512) >> tf.learn.ReLU("Layer_1/ReLU") >> tf.learn.Linear("OutputLayer/Linear", units = 2)
  70. Refine the Model tf.learn.Conv2D("Layer_0/Conv2D", Shape(3, 3, 1, 32), stride1 =

    2, stride2 = 2, ValidConvPadding) >> tf.learn.AddBias("Layer_0/Bias") >> tf.learn.ReLU("Layer_0/ReLU", 0.01f) >> tf.learn.MaxPool("Layer_0/MaxPool", windowSize = Seq(1, 2, 2, 1), stride1 = 2, stride2 = 2, ValidConvPadding) >> val layer = tf.learn.Cast("Input/Cast", FLOAT32) >> tf.learn.Flatten("Input/Flatten") >> tf.learn.Linear("Layer_1/Linear", units = 512) >> tf.learn.ReLU("Layer_1/ReLU") >> tf.learn.Linear("OutputLayer/Linear", units = 2) val estimator = tf.learn.InMemoryEstimator(model) estimator.train(() => trainData)
  71. Run the Model val image: Tensor = readImage(filename) val decoder

    = Seq("not_scala", "scala") val result: Tensor = estimator.infer(() => image) val label = getLabel(result)) drawLabel(image, s"Class: $label ($decoder(label))")
  72. 7. Deploy to Production

  73. Takeaways • Neural networks are functions • We usually learn

    them from known examples • Training can be challenging • Scala is a great language for Deep Learning • Interested? Give it a try • Get involved to make it even better
  74. Thank You for Listening Sören Brunk @soebrunk https://brunk.io