Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Dive into Deep Learning with Scala

Sören Brunk
May 16, 2018
7.4k

Dive into Deep Learning with Scala

Deep learning is an emerging field with the potential to transform whole industries, and to significantly affect software development.
This talk is an introduction to deep learning for Scala developers. We'll see why deep learning enables a new programming paradigm, and how it's related to functional programming.
We'll discuss what makes Scala an interesting choice for deep learning, and look at the current landscape of deep learning libraries for Scala. Equipped with that knowledge, we'll walk through a concrete example that shows you how to create, train and deploy a neural network using Scala.

Talk from Scala Days Europe 2018.

Sören Brunk

May 16, 2018
Tweet

Transcript

  1. What do these Examples have in Common? => “The Quick

    brown fox jumps over the lazy dog.” => =>
  2. => => What do these Examples have in Common? “The

    Quick brown fox jumps over the lazy dog.”
  3. val classifier: Model[Image, ImageClass] = ??? A Neural Network is

    a Function How do we implement that thing?
  4. Representing Values as Tensors Model[Tensor, Tensor] encoder: A => Tensor

    decoder: Tensor => B Model[A, B] 64 432 67 62 7 9 203 90 56 64 230 54 76 4 51 83 24 8 41 40 17 5 230 98 23 7 129 34 93 73
  5. The Artificial Neuron w1 ∑ x1 w2 x2 w3 x3

    wn xn … … Apply Activation Function y Multiply inputs and weights Sum up results
  6. The Artificial Neuron w 1 ∑ x 1 w 2

    x 2 w 3 x 3 w n x n … … y ∑
  7. But How Do We Set Those Weights? ∑ ∑ Input

    Layer 1. Layer 2. Layer (Output) ∑
  8. “What we want is a machine that can learn from

    experience.” Alan Turing, 1947
  9. Training Data: Examples with Answers Example Label (Answer) (1 =

    cat, 0 = not cat) 1 0 1 val examples: Map[Image, Either[Cat, NotCat]]
  10. Compute the Loss Loss (simplified) = correct answer - predicted

    answer Cat = 1 1 0.4 0.6 Forward Pass Compute Loss Backpropagation Update Weights
  11. Error Weight Loss Function Gradient Descent Forward Pass Compute Loss

    Backpropagation Update Weights Source: https://www.coursera.org/learn/machine-learning
  12. Why Scala? Type safe tensor operations case class Tensor( dataType:

    DataType, shape: Shape ) class TypedTensor[DataType, Shape] val tensor: TypedTensor[Float, (Width, Height, Channel)] = ...
  13. Deep Learning Libraries for Scala Source: https://brunk.io/deep-learning-in-scala-part-1-basics-and-libraries.html • Mature, Large

    Community & Commercial Support • Java API • Native Scala API (ScalNet) in alpha • Scala API part of main project • API very pythonic • Most idiomatic & type safe Scala API • Not part of main TensorFlow (yet)
  14. Deep Learning Libraries for Scala Source: https://brunk.io/deep-learning-in-scala-part-1-basics-and-libraries.html • Mature, Large

    Community & Commercial Support • Java API • Native Scala API (ScalNet) in alpha • Scala API part of main project • API very pythonic • Most idiomatic & type safe Scala API • Not part of main TensorFlow (yet)
  15. 3. Collect and Prepare Training Data Filter Resize Normalize Vectorize

    ... encoder: A => Tensor decoder: Tensor => B Training Set Test Set
  16. 3. Collect and Prepare Training Data def readImage(filename: Output): Output

    = { val rawImage = tf.data.readFile(filename) val image = tf.image.decodeJpeg(rawImage, numChannels = 3) tf.image.resizeBilinear(image.expandDims(axis=0), Seq(250, 250)) .squeeze(Seq(0)).cast(UINT8) } val trainData: Dataset[(Tensor, Tensor), (Output, Output), (DataType, DataType), (Shape, Shape)] = tf.data.TensorSlicesDataset(filenamesWithLabels(trainDir)) .shuffle(bufferSize = 30000, Some(seed)) .map({ case (filename, label) => (readImage(filename), label)}) .repeat() .batch(128) .prefetch(100)
  17. 4. Choose the Right Architecture By Aphex34 - Own work,

    CC BY-SA 4.0, https://commons.wikimedia.org/w/index.php?curid=45679374 Source: https://karpathy.github.io/2015/05/21/rnn-effectiveness/ Fully Connected Neural Network Convolutional Neural Network Recurrent Neural Network ∑ ∑ ∑
  18. 4. Choose the Right Architecture By Aphex34 - Own work,

    CC BY-SA 4.0, https://commons.wikimedia.org/w/index.php?curid=45679374 1. Layer 2. Layer 3. Layer 4. Layer Convolutional Neural Network
  19. Build a Simple Model import org.platanios.tensorflow.api._ val input = tf.learn.Input(UINT8,

    Shape(-1, 250, 250, 3)) // type and shape of our images val labelInput = tf.learn.Input(UINT8, Shape(-1)) // type and shape of our labels val layer = tf.learn.Flatten("Input/Flatten") >> // flatten the images into a single vector tf.learn.Cast("Input/Cast", FLOAT32) >> // cast input to float tf.learn.Linear("Layer_1/Linear", units = 64) >> // hidden layer with 512 neurons tf.learn.ReLU("Layer_1/ReLU“, 0.01f) >> // hidden layer activation tf.learn.Linear("OutputLayer/Linear", units = 2) // output layer val trainInputLayer = tf.learn.Cast("TrainInput/Cast", INT64) // cast labels to long val loss = tf.learn.SparseSoftmaxCrossEntropy("Loss/CrossEntropy") >> // loss/error function tf.learn.Mean("Loss/Mean") >> tf.learn.ScalarSummary("Loss/Summary", "Loss") val optimizer = tf.train.GradientDescent(learningRate = 0.001) // the optimizer updates our weights // create a Model for training val model = tf.learn.Model.supervised(input, layer, labelInput, trainInputLayer, loss, optimizer)
  20. Train and Evaluate the Model val estimator = tf.learn.InMemoryEstimator(model) estimator.train(()

    => trainData) Learn / Hooks / Evaluation - Step 0 Evaluator: Learn / Hooks / Evaluation - ╔═══════╤════════════╗ Learn / Hooks / Evaluation - ║ │ Accuracy ║ Learn / Hooks / Evaluation - ╟───────┼────────────╢ Learn / Hooks / Evaluation - ║ Train │ 0.1219 ║ Learn / Hooks / Evaluation - ║ Test │ 0.1217 ║ Learn / Hooks / Evaluation - ╚═══════╧════════════╝ Learn / Hooks / Loss Logger - ( 2.831 s) Step: 100, Loss: 2250.2437 Learn / Hooks / Loss Logger - ( 0.561 s) Step: 200, Loss: 2735.0537 ... Learn / Hooks / Loss Logger - ( 0.537 s) Step: 900, Loss: 877.2033 Learn / Hooks / Loss Logger - ( 0.699 s) Step: 1000, Loss: 1207.4133 Learn / Hooks / Evaluation - Step 1000 Evaluator: Learn / Hooks / Evaluation - ╔═══════╤════════════╗ Learn / Hooks / Evaluation - ║ │ Accuracy ║ Learn / Hooks / Evaluation - ╟───────┼────────────╢ Learn / Hooks / Evaluation - ║ Train │ 0.8105 ║ Learn / Hooks / Evaluation - ║ Test │ 0.7888 ║ Learn / Hooks / Evaluation - ╚═══════╧════════════╝ Learn / Hooks / Loss Logger - ( 2.599 s) Step: 1100, Loss: 989.0901 Learn / Hooks / Loss Logger - ( 0.529 s) Step: 1200, Loss: 440.6418 ...
  21. Refine the Model val layer = tf.learn.Cast("Input/Cast", FLOAT32) >> tf.learn.Flatten("Input/Flatten")

    >> tf.learn.Linear("Layer_1/Linear", units = 64) >> // hidden layer tf.learn.ReLU("Layer_1/ReLU", 0.01f) >> // hidden layer activation tf.learn.Linear("OutputLayer/Linear", units = 2) // output layer
  22. Refine the Model val layer = tf.learn.Cast("Input/Cast", FLOAT32) >> tf.learn.Flatten("Input/Flatten")

    >> tf.learn.Linear("Layer_1/Linear", units = 128) >> // hidden layer tf.learn.ReLU("Layer_1/ReLU", 0.01f) >> // hidden layer activation tf.learn.Linear("OutputLayer/Linear", units = 2) // output layer
  23. Refine the Model val layer = tf.learn.Cast("Input/Cast", FLOAT32) >> tf.learn.Flatten("Input/Flatten")

    >> tf.learn.Linear("Layer_1/Linear", units = 128) >> // hidden layer tf.learn.ReLU("Layer_1/ReLU", 0.01f) >> // hidden layer activation tf.learn.Linear("OutputLayer/Linear", units = 2) // output layer val estimator = tf.learn.InMemoryEstimator(model) estimator.train(() => trainData)
  24. Refine the Model val layer = tf.learn.Cast("Input/Cast", FLOAT32) >> tf.learn.Flatten("Input/Flatten")

    >> tf.learn.Linear("Layer_1/Linear", units = 512) >> // hidden layer tf.learn.ReLU("Layer_1/ReLU", 0.01f) >> // hidden layer activation tf.learn.Linear("OutputLayer/Linear", units = 2) // output layer val estimator = tf.learn.InMemoryEstimator(model) estimator.train(() => trainData)
  25. Refine the Model tf.learn.Conv2D("Layer_0/Conv2D", Shape(3, 3, 1, 32), stride1 =

    2, stride2 = 2, ValidConvPadding) >> tf.learn.AddBias("Layer_0/Bias") >> tf.learn.ReLU("Layer_0/ReLU", 0.01f) >> tf.learn.MaxPool("Layer_0/MaxPool", windowSize = Seq(1, 2, 2, 1), stride1 = 2, stride2 = 2, ValidConvPadding) >> val layer = tf.learn.Cast("Input/Cast", FLOAT32) >> tf.learn.Flatten("Input/Flatten") >> tf.learn.Linear("Layer_1/Linear", units = 512) >> tf.learn.ReLU("Layer_1/ReLU") >> tf.learn.Linear("OutputLayer/Linear", units = 2)
  26. Refine the Model tf.learn.Conv2D("Layer_0/Conv2D", Shape(3, 3, 1, 32), stride1 =

    2, stride2 = 2, ValidConvPadding) >> tf.learn.AddBias("Layer_0/Bias") >> tf.learn.ReLU("Layer_0/ReLU", 0.01f) >> tf.learn.MaxPool("Layer_0/MaxPool", windowSize = Seq(1, 2, 2, 1), stride1 = 2, stride2 = 2, ValidConvPadding) >> val layer = tf.learn.Cast("Input/Cast", FLOAT32) >> tf.learn.Flatten("Input/Flatten") >> tf.learn.Linear("Layer_1/Linear", units = 512) >> tf.learn.ReLU("Layer_1/ReLU") >> tf.learn.Linear("OutputLayer/Linear", units = 2) val estimator = tf.learn.InMemoryEstimator(model) estimator.train(() => trainData)
  27. Run the Model val image: Tensor = readImage(filename) val decoder

    = Seq("not_scala", "scala") val result: Tensor = estimator.infer(() => image) val label = getLabel(result)) drawLabel(image, s"Class: $label ($decoder(label))")
  28. Takeaways • Neural networks are functions • We usually learn

    them from known examples • Training can be challenging • Scala is a great language for Deep Learning • Interested? Give it a try • Get involved to make it even better