Alexey Zinoviev
February 23, 2021
410

# Deep Learning Introduction with KotlinDL

## Alexey Zinoviev

February 23, 2021

## Transcript

1. Introduction to Deep Learning
with KotlinDL
Alexey Zinovyev, ML Engineer, Apache Ignite PMC
JetBrains

2. Bio
1. Java & Kotlin developer
2. Distributed ML enthusiast
3. Apache Ignite PMC
4. TensorFlow Contributor
5. ML engineer at JetBrains
6. Happy father and husband
7. https://github.com/zaleslaw

3. Motivation
1. Kotlin took a course to become a convenient language for data science
2. No modern data science without Neural Networks
3. All deep learning frameworks are good enough at image recognition
4. Convolutional neural networks (CNNs) are the gold standard for image
recognition
5. Training, Transfer Learning, and Inference are now available for diﬀerent CNN
architectures on Kotlin with KotlinDL library

4. Agenda
1. Neural Network Intro
2. Deep Learning
3. Required and optional math knowledge
4. Primitives or building blocks
a. Activation Functions
b. Loss Functions
c. Initializers
d. Optimizers
e. Layers
5. 5 Major Scientiﬁc breakthroughs in DL
6. Kotlin DL Demo

5. Some basic terms
1. Model

6. Some basic terms
1. Model
2. Inference

7. Some basic terms
1. Model
2. Inference
3. Training

8. Some basic terms
1. Model
2. Inference
3. Training
4. Transfer Learning

9. Some basic terms
1. Model
2. Inference
3. Training
4. Transfer Learning
5. Evaluation

10. Some basic terms
1. Model
2. Inference
3. Training
4. Transfer Learning
5. Evaluation
6. Train/validation/test datasets

11. Neural Network Intro

12. The life of one neuron

13. The place of one neuron in his family

14. Forward propagation

15. Cat/Dog neural network architecture in Kotlin

16. MNIST example

17. MNIST Subset

18. Backward propagation

19. Full training with some maths

20. Deep Learning is just...

21. Way to approximate unknown function

22. math

23. Need to keep in mind

Some trigonometry, exponentials and logarithms;

Linear Algebra: vectors, vector space;

Linear Algebra: inverse and transpose matrices, matrix decomposition,
eigenvectors, Kronecker-Capelli’s theorem;

Mathematical Analysis: continuous, monotonous, diﬀerentiable functions;

Mathematical Analysis: derivative, partial derivative, Jacobian;

Methods of one-dimensional and multidimensional optimization;

Gradient Descent and all its variations;

Optimization methods and convex analysis will not be superﬂuous in your
luggage

24. N-dimensional space in theory

25. Matrix multiplication (friendly reminder)

26. Loss Functions

27. Loss Functions

Each loss function could be reused as metric

Should be diﬀerentiable

Not every metric could be a loss function ( metrics could have not the derivative )

Loss function could be very complex

Are diﬀerent for regression and classiﬁcation tasks

28. An optimization problem [Loss Optimization Problem]

29. Most widely used:

32. Optimizers

33. Optimizers: SGD with memory

34. Optimizers

SGD

SGD with Momentum

RMSProp

...

35. It’s faster as a result

36. Wire it together with KotlinDL

37. Activation Functions

38. Activation functions

Activation functions change the outputs coming out of each layer of a neural
network.

Should have a derivative (to be used in backward propagation)

39. Linear

40. Sigmoid

41. Tanh

42. ReLU

43. Hm..

44. Initializers

47. Initializers

Zeros

Ones

Random [Uniform or Normal]

Xavier / Glorot

He
...

48. Layers

49. Dense

50. Cat/Dog neural network architecture in Kotlin

51. Conv2d: ﬁlters

52. Output with ﬁlters

53. Pooling (subsampling layer)

54. Dropout

55. Everything now available in Kotlin

56. How it works?

57. How it works?

58. How it works?

59. How it works?

60. KotlinDL Demo

61. KotlinDL Limitations
1. Now useful for Image Recognition task and Regression ML
2. Limited number of layers is supported
3. Tiny number of preprocessing methods
4. Only VGG-like architectures are supported
5. No Android support

1. New models: Inception, ResNet, DenseNet
2. Rich Dataset API
3. GPU settings
4. Maven Central Availability
5. Functional API
6. New layers: BatchNorm, Add, Concatenate, DepthwiseConv2d
7. Regularization for layers will be added
8. New metrics framework
9. Conversion to TFLite (for mobile devices)
10. ONNX support
11. ML algorithms

1. https://github.com/JetBrains/KotlinDL
2. https://kotlinlang.org/docs/data-science-overview.html#kotlin-libraries
3. #deeplearning channel on Kotlin slack (join it, if you are not yet)
4. Feel free to join discussions on Github