Introduction to Deep Learning with KotlinDL

Links from the last slide:
KotlinDL: https://github.com/jetbrains/kotlindl
KInference: https://github.com/JetBrains-Research/kinference
Multik: https://github.com/Kotlin/multik
Kotlin kernel for Jupyter: https://github.com/Kotlin/kotlin-jupyter
Lets-Plot: https://github.com/JetBrains/lets-plot-kotlin

Transcript

1. @MariaKhalusova With KotlinDL Introduction to Deep Learning NOT HOTDOG Photo by Peter Bravo de los Rios on Unsplash
2. About me Developer Advocate at JetBrains Python person converted to Kotlin PyData Montreal Organizer mariakhalusova.com @MariaKhalusova on Twitter
3. Source: https://blogs.nvidia.com/blog/2016/07/29/whats-di ff erence-arti fi cial-intelligence-machine-learning-deep-learning-ai/
4. Image Data
5. Text Data https://huggingface.co/distilgpt2
6. Other Complex Data
7. What makes Deep Learning so popular? No need to hand-craft rules for complex use cases More accurate than humans Specialized hardware More labeled data http://karpathy.github.io/2011/04/27/manually-classifying-cifar10/
8. Deep Learning isn’t magic Multilayer Perceptron ReLu Activation Function Kernel Initializer Gradient Descent Sigmoid Backpropagation Loss function Conv2D LSTM Tensor Epoch Learning rate Computation graph TensorFlow Keras PyTorch Dense layers MaxPooling2D Transfer Learning Transformers Softmax Dropout Embeddings Batch Normalization
9. Agenda Simple neural network architecture Building blocks How the training works Inference
10. Multilayer Perceptron Source: https://becominghuman.ai/multi-layer-perceptron-mlp-models-on-real-world-banking-data-f6dd3d7e998f Feed-forward Dense Layers
11. Multilayer Perceptron Source: https://becominghuman.ai/multi-layer-perceptron-mlp-models-on-real-world-banking-data-f6dd3d7e998f
12. Multilayer Perceptron Source: https://becominghuman.ai/multi-layer-perceptron-mlp-models-on-real-world-banking-data-f6dd3d7e998f
13. Hidden Layer’s node Weighted sum: Activation function: - ReLU, or - Sigmoid, or - Tanh, or - …
14. Activation functions
15. Kernel and Bias Initializers What should the initial weights be before training? All zeros? A constant? Random values? Random values from a uniform distribution? …
16. “Delving Deep into Recti fi ers: 
 Surpassing Human-Level Performance on ImageNet Classi fi cation ” by Kaiming He et al. 201 5 “… we derive a robust initialization method that particularly considers the recti fi er nonlinearities.”
17. But how do you train it?
18. What does learning mean? Leaning = finding optimal weights and biases: w1, w2,… b1, b2,…
19. Loss function Classi fi cation Regression SOFT_MAX_CROSS_ENTROPY_WITH_LOGITS MSE BINARY_CROSSENTROPY MSLE … …
20. RMSE aka Root Mean Squared Error
21. Binary Cross-Entropy
22. Binary Cross-Entropy Takes into account uncertainty of predictions Larger penalty for confident wrong predictions True label Predicted prob. of class 1 loss 1 0.9 0.105360515657 1 0.55 0.597837000755 1 0.10 2.302585092994 0 0.95 2.995732273553
23. All about finding minima
24. Gradient Descent
25. Compiling the model
26. Training Split the data: - Why? - How much data should go into test set? - How to split? Epochs & Batches
27. MNIST Fashion Label Class 0 T-shirt/top 1 Trouser 2 Pullover 3 Dress 4 Coat 5 Sandal 6 Shirt 7 Sneaker 8 Bag 9 Ankle boot https://github.com/zalandoresearch/fashion-mnist
28. MNIST Fashion 70 000 images 28 x 28 pixels Grayscale val (train, test) = Dataset.createTrainAndTestDatasets ( trainFeaturesPath = "datasets/fashionmnist/train-images-idx3-ubyte.gz" , trainLabelsPath = "datasets/fashionmnist/train-labels-idx1-ubyte.gz" , testFeaturesPath = "datasets/fashionmnistmnist/t10k-images-idx3-ubyte.gz" , testLabelsPath = "datasets/fashionmnistmnist/t10k-labels-idx1-ubyte.gz" , numClasses = 10 , ::extractImages , ::extractLabel s ) val (trainingSet, testSet) = train.split(splitRatio = 0.95 )
29. private val model = Sequential.of ( Input(28, 28, 1) , Flatten() , Dense(300) , Dense(100) , Dense(10 ) ) fun main() { model.use { it.compile ( optimizer = Adam() , loss = Losses.SOFT_MAX_CROSS_ENTROPY_WITH_LOGITS , metric = Metrics.ACCURAC Y ) it.summary( ) it.fit ( dataset = trainingSet , epochs = 10 , batchSize = 10 0 ) val accuracy = it.evaluate ( dataset = testSet , batchSize = 10 0 ).metrics[Metrics.ACCURACY ] it.save(File("src/model/my_model") ) println("Accuracy: $accuracy" ) } }
30. val PATH_TO_MODEL = "src/model/my_model " val stringLabels = mapOf(0 to "T-shirt/top" , 1 to "Trouser" , 2 to "Pullover" , 3 to "Dress" , 4 to "Coat" , 5 to "Sandal" , 6 to "Shirt" , 7 to "Sneaker" , 8 to "Bag" , 9 to "Ankle boot " ) fun reshapeInput(inputData: FloatArray): Array<Array<FloatArray>> { val reshaped = Array ( 1 ) { Array(28) { FloatArray(28) } } for (i in inputData.indices) reshaped[0][i / 28][i % 28] = inputData[i ] return reshape d } fun main() { InferenceModel.load(File(PATH_TO_MODEL)).use { it.reshape(::reshapeInput ) val prediction = it.predict(test.getX(0) ) val actualLabel = test.getLabel(0 ) println("Predicted label is: $prediction. This corresponds to class ${stringLabels[prediction]}." ) println("Actual label is: $actualLabel." ) } }
31. What else you can do with KotlinDL Build CNNs Train on GPU Import a model trained with Keras 2.x Transfer Learning
32. Current KotlinDL limitations Limited number of layers No Android support Minimal data preprocessing functionality
33. Links KotlinDL: https://github.com/jetbrains/kotlindl KInference: https://github.com/JetBrains- Research/kinference Multik: https://github.com/Kotlin/multik Kotlin kernel for Jupyter: https://github.com/ Kotlin/kotlin-jupyter Lets-Plot: https://github.com/JetBrains/lets-plot- kotlin