DEEP LEARNING ATLAS Introduction to Neural Networks Michał Karzyński • 4Developers 2019 

ABOUT THIS TALK • What are neural networks? • How do they learn? • Convolutional Neural Networks (CNNs) • Recurrent neural networks (RNNs) • Generative adversarial networks (GANs) 

ABOUT ME • Michał Karzyński (@postrational) • Full stack geek (C++, Python, JavaScript) • I blog at http://michal.karzynski.pl • I’m a tech lead at working on 

No content

WHAT IS A NEURAL NETWORK? 

A universal function approximator. 

WHAT IS A FUNCTION? Input Output Function (mapping) 

WHAT IS A FUNCTION? Hotdog Not Hotdog Function (mapping) 

NEURAL NETWORK 
 CAN APPROXIMATE ANY FUNCTION Hotdog Not Hotdog 1 1 1 ⦦ ⦦ ⦦ ⦦ ⦦ ⦦ 

WHAT IS A NEURAL NETWORK? 1 1 1 ⦦ ⦦ ⦦ ⦦ ⦦ ⦦ ⦦ 1 Input Layer Hidden Layer Bias Output Layer 

1 b W HOW DO NEURAL NETWORKS WORK? 1 1 ⦦ ⦦ ⦦ Hotdog Not hotdog 0 10 255 168 x ⦦ ⦦ ⦦ 

1 b W HOW DO NEURAL NETWORKS WORK? 1 1 ⦦ ⦦ ⦦ Hotdog Not hotdog 0 10 255 168 n ∑ i=1 wi xi + b x ⦦ ⦦ ⦦ 

ACTIVATION FUNCTIONS ReLU 0 1,5 3 4,5 6 -6 -3 0 3 6 Sigmoid 0 0,25 0,5 0,75 1 -6 -3 0 3 6 Tanh -1 -0,5 0 0,5 1 -6 -3 0 3 6 

HOW DO NEURAL NETWORKS WORK? 1 1 1 ⦦ ⦦ ⦦ ⦦ ⦦ ⦦ Hotdog Not hotdog 0 10 255 168 ReLU ( n ∑ i=1 wi xi + b ) 

HOW DO NEURAL NETWORKS WORK? 1 1 1 ⦦ ⦦ ⦦ ⦦ ⦦ ⦦ Hotdog Not hotdog 0 10 255 168 

HOW DO NEURAL NETWORKS WORK? 1 1 1 ⦦ ⦦ ⦦ ⦦ ⦦ ⦦ Hotdog Not hotdog 0.10 0.90 Prediction 0 10 255 168 

HOW DO NEURAL NETWORKS LEARN? 

INFERENCE TRAINING 

HOW DO NEURAL NETWORKS LEARN? 1 1 1 ⦦ ⦦ ⦦ ⦦ ⦦ ⦦ Hotdog Not hotdog 0.50 0.90 Prediction 0 10 255 168 

HOW DO NEURAL NETWORKS LEARN? 1 1 1 ⦦ ⦦ ⦦ ⦦ ⦦ ⦦ Hotdog Not hotdog 0.50 0 0.90 1 Prediction G round truth 0 10 255 168 

HOW DO NEURAL NETWORKS LEARN? 1 1 1 ⦦ ⦦ ⦦ ⦦ ⦦ ⦦ Hotdog Not hotdog ⏐0.50 - 0⏐ = 0.50 ⏐0.90 - 1⏐ = 0.10 0.60 Prediction G round truth Loss 0 10 255 168 

HOW DO NEURAL NETWORKS LEARN? 1 1 1 ⦦ ⦦ ⦦ ⦦ ⦦ ⦦ Hotdog Not hotdog ⏐0.55 - 0⏐ = 0.55 ⏐0.85 - 1⏐ = 0.15 0.70 Prediction G round truth Loss 0 10 255 168 

HOW DO NEURAL NETWORKS LEARN? 1 1 1 ⦦ ⦦ ⦦ ⦦ ⦦ ⦦ Hotdog Not hotdog ⏐0.45 - 0⏐ = 0.45 ⏐0.95 - 1⏐ = 0.05 0.50 Prediction G round truth Loss 0 10 255 168 

HOW DO NEURAL NETWORKS LEARN? • Backpropagation - calculate the gradients using derivatives • Stochastic gradient descent - use gradients to adjust weights 

BACKPROPAGATION 1 1 1 ⦦ ⦦ ⦦ ⦦ ⦦ ⦦ Hotdog Not hotdog ⏐0.45 - 0⏐ = 0.45 ⏐0.95 - 1⏐ = 0.05 0.50 Prediction G round truth Loss 0 10 255 168 ∇W3 = dLoss dW3 ∇W2 = dLoss dW2 ∇W1 = dLoss dW1 

BACKPROPAGATION 1 1 1 ⦦ ⦦ ⦦ ⦦ ⦦ ⦦ Hotdog Not hotdog ⏐0.45 - 0⏐ = 0.45 ⏐0.95 - 1⏐ = 0.05 0.50 Prediction G round truth Loss 0 10 255 168 ∇W3 ∇W2 ∇W1 

STOCHASTIC GRADIENT DESCENT one step per mini-batch 

CONVOLUTIONAL NEURAL NETWORKS 

WHAT IS A NEURAL NETWORK? 1 1 1 ⦦ ⦦ ⦦ ⦦ ⦦ ⦦ ⦦ 1 Input Layer Hidden Layer Bias Output Layer 

CONVOLUTIONAL NEURAL NETWORK 1 1 1 ⦦ ⦦ ⦦ ⦦ 1 Input Layer Fully-connected Bias Output Layer Convolutional layer 

CONVOLUTION OPERATION Input image Weights ∗ = Output image 

DETECT VERTICAL EDGES 5 5 5 0 0 0 5 5 5 0 0 0 5 5 5 0 0 0 5 5 5 0 0 0 5 5 5 0 0 0 5 5 5 0 0 0 1 0 -1 2 0 -2 1 0 -1 Input data Sobel filter ∗ = 

DETECT VERTICAL EDGES 5 5 5 0 0 0 5 5 5 0 0 0 5 5 5 0 0 0 5 5 5 0 0 0 5 5 5 0 0 0 5 5 5 0 0 0 1 0 -1 2 0 -2 1 0 -1 0 Input data Sobel filter ∗ = ∑ m ∑ n I(m,n) K(m,n) 

DETECT VERTICAL EDGES 5 5 5 0 0 0 5 5 5 0 0 0 5 5 5 0 0 0 5 5 5 0 0 0 5 5 5 0 0 0 5 5 5 0 0 0 1 0 -1 2 0 -2 1 0 -1 0 20 Input data Sobel filter ∗ = 

DETECT VERTICAL EDGES 5 5 5 0 0 0 5 5 5 0 0 0 5 5 5 0 0 0 5 5 5 0 0 0 5 5 5 0 0 0 5 5 5 0 0 0 1 0 -1 2 0 -2 1 0 -1 0 20 20 Input data Sobel filter ∗ = 

DETECT VERTICAL EDGES 5 5 5 0 0 0 5 5 5 0 0 0 5 5 5 0 0 0 5 5 5 0 0 0 5 5 5 0 0 0 5 5 5 0 0 0 1 0 -1 2 0 -2 1 0 -1 0 20 20 0 Input data Sobel filter ∗ = 

DETECT VERTICAL EDGES 5 5 5 0 0 0 5 5 5 0 0 0 5 5 5 0 0 0 5 5 5 0 0 0 5 5 5 0 0 0 5 5 5 0 0 0 1 0 -1 2 0 -2 1 0 -1 0 20 20 0 0 ... Input data Sobel filter ∗ = 

DETECT VERTICAL EDGES 5 5 5 0 0 0 5 5 5 0 0 0 5 5 5 0 0 0 5 5 5 0 0 0 5 5 5 0 0 0 5 5 5 0 0 0 1 0 -1 2 0 -2 1 0 -1 0 20 20 0 0 20 20 0 0 20 20 0 0 20 20 0 Input data Sobel filter Feature map ∗ = 

USING MULTIPLE FILTERS 5 5 5 0 0 0 5 5 5 0 0 0 5 5 5 0 0 0 5 5 5 0 0 0 5 5 5 0 0 0 5 5 5 0 0 0 Feature map 1 ∗ = Feature map 2 

LEARNING MULTIPLE FILTERS [Krizhevsky et al., NIPS'12] 

CONVOLUTIONAL NETWORK: VGG-16 224x224x3 Convolution & ReLU Max Pooling Fully Connected & ReLU FC + SoftMax 224x224x64 112x112x128 56x56x256 28x28x512 14x14x512 7x7x512 4096 1000 112x112x64 56x56x128 28x28x256 14x14x512 

VISUALIZATION OF CONVOLUTIONAL NETWORKS [Zeiler and Fergus, ECCV 2014] 

VISUALIZATION OF CONVOLUTIONAL NETWORKS [Zeiler and Fergus, ECCV 2014] 

[Zeiler and Fergus, ECCV 2014] 

APPLICATIONS OF CNNS • Image recognition • Facial recognition • Detecting objects in an image • Handwriting recognition • Video analysis • Drug discovery • Medical diagnostics 

No content

RECURRENT NEURAL NETWORKS 

WHAT IS A NEURAL NETWORK? 1 1 1 ⦦ ⦦ ⦦ ⦦ ⦦ ⦦ ⦦ 1 Input Layer Hidden Layer Bias Output Layer b W1 x W2 W3 ^ y 

WHAT IS A NEURAL NETWORK? Input Layer Hidden Layer Output Layer W1 x W2 W3 ^ y 

RECURRENT NEURAL NETWORKS (WORKING WITH SEQUENCES) xt WI ^ yt Wy WR xt WI ^ yt Wy xt+1 WI ^ yt+1 Wy xt+2 WI ^ yt+2 Wy WR WR 

RECURRENT NEURAL NETWORKS 1 0 0 0 "e" 0 0 1 0 "l" 0 0 1 0 "l" e 0 h 1 l 0 o 0 "h" = "h" 0 1 0 0 0.7 0.1 0.1 0.1 "e" 0.2 0.2 0.4 0.2 "l" 0.1 0.0 0.9 0.0 "l" 0.1 0.1 0.1 0.7 "o" 

RECURRENT NEURAL NETWORKS Naturalism and decision for the majority of Arab countries' capitalide was grounded by the Irish language by [[John Clair]], [[An Imperial Japanese Revolt]], associated with Guangzham's sovereignty. His generals were the powerful ruler of the Portugal in the [[Protestant Immineners]], which could be said to be directly in Cantonese Communication, which followed a ceremony and set inspired prison, training. The emperor travelled back to [[Antioch, Perth, October 25|21]] to note, the Kingdom of Costa Rica, unsuccessful fashioned the [[Thrales]], [[Cynth's Dajoard]], known in western [[Scotland]], near Italy to the conquest of India with the conflict. [http://karpathy.github.io/2015/05/21/rnn-effectiveness/] 

RECURRENT NEURAL NETWORKS this movie was really good positive 0.81 negative 0.19 

RECURRENT NEURAL NETWORKS he learned about neural networks uczył się o sieciach neuronowych 

APPLICATIONS OF RNNS • Language modelling • Machine translation • Speech recognition • Music composition • Time series prediction • Human action recognition • Protein homology detection 

No content

HIGHER-ORDER NETWORKS 

VIDEO CAPTIONING NETWORK Conv Conv Conv Conv people racing to the top of sand dunes 

GENERATIVE ADVERSARIAL NETWORKS Conv Random noise Deconvolution real 0.81 fake 0.19 Fake image Real images Generator Discriminator 

this person does not exist.com 

this person does not exist.com 

No content

DEEP LEARNING SPECIALIZATION ON COURSERA 

DEEP LEARNING SOFTWARE 

DEEP LEARNING FRAMEWORK ARCHITECTURE Python API Graph Intermediate Representation CPU GPU Accelerator Execution backends 

ngraph.ai 

THANK YOU michal.karzynski.pl References ImageNet Classification with Deep Convolutional Neural Networks (NIPS, 2012)
 Alex Krizhevsky, Ilya Sutskever, Geoffrey Hinton
 https://papers.nips.cc/paper/4824-imagenet-classification-with-deep-convolutional-neural-networks Visualizing and Understanding Convolutional Networks (ECCV, 2014)
 Matthew Zeiler, Rob Fergus
 https://arxiv.org/abs/1311.2901 The Unreasonable Effectiveness of Recurrent Neural Networks
 Andrej Karpathy
 http://karpathy.github.io/2015/05/21/rnn-effectiveness/ Star Trek images via TrekCore.com. 
 All Star Trek trademarks and images are owned by CBS Studios Inc. Hot-dog image by TheCulinaryGeek on Flickr
 https://www.flickr.com/photos/preppybyday/ 5076302395 All other images from pexels.com
 https://www.pexels.com/ * Other names and brands may be claimed as the property of others