Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Deep Learning Introduction with KotlinDL

Deep Learning Introduction with KotlinDL

376cd2fd5ffded946c96d5a45766350f?s=128

Alexey Zinoviev

February 23, 2021
Tweet

Transcript

  1. Introduction to Deep Learning with KotlinDL Alexey Zinovyev, ML Engineer,

    Apache Ignite PMC JetBrains
  2. Bio 1. Java & Kotlin developer 2. Distributed ML enthusiast

    3. Apache Ignite PMC 4. TensorFlow Contributor 5. ML engineer at JetBrains 6. Happy father and husband 7. https://github.com/zaleslaw
  3. Motivation 1. Kotlin took a course to become a convenient

    language for data science 2. No modern data science without Neural Networks 3. All deep learning frameworks are good enough at image recognition 4. Convolutional neural networks (CNNs) are the gold standard for image recognition 5. Training, Transfer Learning, and Inference are now available for different CNN architectures on Kotlin with KotlinDL library
  4. Agenda 1. Neural Network Intro 2. Deep Learning 3. Required

    and optional math knowledge 4. Primitives or building blocks a. Activation Functions b. Loss Functions c. Initializers d. Optimizers e. Layers 5. 5 Major Scientific breakthroughs in DL 6. Kotlin DL Demo
  5. Some basic terms 1. Model

  6. Some basic terms 1. Model 2. Inference

  7. Some basic terms 1. Model 2. Inference 3. Training

  8. Some basic terms 1. Model 2. Inference 3. Training 4.

    Transfer Learning
  9. Some basic terms 1. Model 2. Inference 3. Training 4.

    Transfer Learning 5. Evaluation
  10. Some basic terms 1. Model 2. Inference 3. Training 4.

    Transfer Learning 5. Evaluation 6. Train/validation/test datasets
  11. Neural Network Intro

  12. The life of one neuron

  13. The place of one neuron in his family

  14. Forward propagation

  15. Cat/Dog neural network architecture in Kotlin

  16. MNIST example

  17. MNIST Subset

  18. Backward propagation

  19. Full training with some maths

  20. Deep Learning is just...

  21. Way to approximate unknown function

  22. math

  23. Need to keep in mind • Some trigonometry, exponentials and

    logarithms; • Linear Algebra: vectors, vector space; • Linear Algebra: inverse and transpose matrices, matrix decomposition, eigenvectors, Kronecker-Capelli’s theorem; • Mathematical Analysis: continuous, monotonous, differentiable functions; • Mathematical Analysis: derivative, partial derivative, Jacobian; • Methods of one-dimensional and multidimensional optimization; • Gradient Descent and all its variations; • Optimization methods and convex analysis will not be superfluous in your luggage
  24. N-dimensional space in theory

  25. Matrix multiplication (friendly reminder)

  26. Loss Functions

  27. Loss Functions • Each loss function could be reused as

    metric • Should be differentiable • Not every metric could be a loss function ( metrics could have not the derivative ) • Loss function could be very complex • Are different for regression and classification tasks
  28. An optimization problem [Loss Optimization Problem]

  29. Most widely used:

  30. Gradients

  31. Gradient Descent

  32. Optimizers

  33. Optimizers: SGD with memory

  34. Optimizers • SGD • SGD with Momentum • Adam •

    RMSProp • AdaDelta ...
  35. It’s faster as a result

  36. Wire it together with KotlinDL

  37. Activation Functions

  38. Activation functions • Activation functions change the outputs coming out

    of each layer of a neural network. • Required to add non-linearity • Should have a derivative (to be used in backward propagation)
  39. Linear

  40. Sigmoid

  41. Tanh

  42. ReLU

  43. Hm..

  44. Initializers

  45. Vanishing gradient problem

  46. Exploding gradient problem

  47. Initializers • Zeros • Ones • Random [Uniform or Normal]

    • Xavier / Glorot • He ...
  48. Layers

  49. Dense

  50. Cat/Dog neural network architecture in Kotlin

  51. Conv2d: filters

  52. Output with filters

  53. Pooling (subsampling layer)

  54. Dropout

  55. Everything now available in Kotlin

  56. How it works?

  57. How it works?

  58. How it works?

  59. How it works?

  60. KotlinDL Demo

  61. KotlinDL Limitations 1. Now useful for Image Recognition task and

    Regression ML 2. Limited number of layers is supported 3. Tiny number of preprocessing methods 4. Only VGG-like architectures are supported 5. No Android support
  62. KotlinDL Roadmap 1. New models: Inception, ResNet, DenseNet 2. Rich

    Dataset API 3. GPU settings 4. Maven Central Availability 5. Functional API 6. New layers: BatchNorm, Add, Concatenate, DepthwiseConv2d 7. Regularization for layers will be added 8. New metrics framework 9. Conversion to TFLite (for mobile devices) 10. ONNX support 11. ML algorithms
  63. Useful links 1. https://github.com/JetBrains/KotlinDL 2. https://kotlinlang.org/docs/data-science-overview.html#kotlin-libraries 3. #deeplearning channel on

    Kotlin slack (join it, if you are not yet) 4. Feel free to join discussions on Github 5. Follow @zaleslaw and @KotlinForData on Twitter
  64. The End

  65. LSTM consist of LSTM “neurons”

  66. 5 Major Scientific breakthroughs

  67. 5 Major Scientific breakthroughs 1. Non-linearity in early 90’s (could

    solve new problems) 2. ReLU and simplest and cheap non-linearity (could converge on new tasks) 3. Batch Normalization (could help convergence and give more acceleration) 4. Xe Initialization (solves vanishing/exploding gradient problem) 5. Adam optimizer (give more performance)