Deep Learning Atlas

What are neural networks? What makes them recurrent or convolutional? This talk will give you an introduction to and a whirlwind tour of the field of Deep Learning. We'll start by explaining how simple artificial neural networks work and how they learn though backpropagation. We'll then go over different network architectures used in computer vision, sequence analysis, neural machine translation and other areas.

Transcript

1. DEEP LEARNING ATLAS Introduction to Neural Networks Michał Karzyński • 4Developers 2019
2. ABOUT THIS TALK • What are neural networks? • How do they learn? • Convolutional Neural Networks (CNNs) • Recurrent neural networks (RNNs) • Generative adversarial networks (GANs)
3. ABOUT ME • Michał Karzyński (@postrational) • Full stack geek (C++, Python, JavaScript) • I blog at http://michal.karzynski.pl • I’m a tech lead at working on
5. WHAT IS A NEURAL NETWORK?
6. A universal function approximator.
7. WHAT IS A FUNCTION? Input Output Function (mapping)
8. WHAT IS A FUNCTION? Hotdog Not Hotdog Function (mapping)
9. NEURAL NETWORK 
 CAN APPROXIMATE ANY FUNCTION Hotdog Not Hotdog 1 1 1 ⦦ ⦦ ⦦ ⦦ ⦦ ⦦
10. WHAT IS A NEURAL NETWORK? 1 1 1 ⦦ ⦦ ⦦ ⦦ ⦦ ⦦ ⦦ 1 Input Layer Hidden Layer Bias Output Layer
11. 1 b W HOW DO NEURAL NETWORKS WORK? 1 1 ⦦ ⦦ ⦦ Hotdog Not hotdog 0 10 255 168 x ⦦ ⦦ ⦦
12. 1 b W HOW DO NEURAL NETWORKS WORK? 1 1 ⦦ ⦦ ⦦ Hotdog Not hotdog 0 10 255 168 n ∑ i=1 wi xi + b x ⦦ ⦦ ⦦
13. ACTIVATION FUNCTIONS ReLU 0 1,5 3 4,5 6 -6 -3 0 3 6 Sigmoid 0 0,25 0,5 0,75 1 -6 -3 0 3 6 Tanh -1 -0,5 0 0,5 1 -6 -3 0 3 6
14. HOW DO NEURAL NETWORKS WORK? 1 1 1 ⦦ ⦦ ⦦ ⦦ ⦦ ⦦ Hotdog Not hotdog 0 10 255 168 ReLU ( n ∑ i=1 wi xi + b )
15. HOW DO NEURAL NETWORKS WORK? 1 1 1 ⦦ ⦦ ⦦ ⦦ ⦦ ⦦ Hotdog Not hotdog 0 10 255 168
16. HOW DO NEURAL NETWORKS WORK? 1 1 1 ⦦ ⦦ ⦦ ⦦ ⦦ ⦦ Hotdog Not hotdog 0.10 0.90 Prediction 0 10 255 168
17. HOW DO NEURAL NETWORKS LEARN?
18. INFERENCE TRAINING
19. HOW DO NEURAL NETWORKS LEARN? 1 1 1 ⦦ ⦦ ⦦ ⦦ ⦦ ⦦ Hotdog Not hotdog 0.50 0.90 Prediction 0 10 255 168
20. HOW DO NEURAL NETWORKS LEARN? 1 1 1 ⦦ ⦦ ⦦ ⦦ ⦦ ⦦ Hotdog Not hotdog 0.50 0 0.90 1 Prediction G round truth 0 10 255 168
21. HOW DO NEURAL NETWORKS LEARN? 1 1 1 ⦦ ⦦ ⦦ ⦦ ⦦ ⦦ Hotdog Not hotdog ⏐0.50 - 0⏐ = 0.50 ⏐0.90 - 1⏐ = 0.10 0.60 Prediction G round truth Loss 0 10 255 168
22. HOW DO NEURAL NETWORKS LEARN? 1 1 1 ⦦ ⦦ ⦦ ⦦ ⦦ ⦦ Hotdog Not hotdog ⏐0.55 - 0⏐ = 0.55 ⏐0.85 - 1⏐ = 0.15 0.70 Prediction G round truth Loss 0 10 255 168
23. HOW DO NEURAL NETWORKS LEARN? 1 1 1 ⦦ ⦦ ⦦ ⦦ ⦦ ⦦ Hotdog Not hotdog ⏐0.45 - 0⏐ = 0.45 ⏐0.95 - 1⏐ = 0.05 0.50 Prediction G round truth Loss 0 10 255 168
24. HOW DO NEURAL NETWORKS LEARN? • Backpropagation - calculate the gradients using derivatives • Stochastic gradient descent - use gradients to adjust weights
25. BACKPROPAGATION 1 1 1 ⦦ ⦦ ⦦ ⦦ ⦦ ⦦ Hotdog Not hotdog ⏐0.45 - 0⏐ = 0.45 ⏐0.95 - 1⏐ = 0.05 0.50 Prediction G round truth Loss 0 10 255 168 ∇W3 = dLoss dW3 ∇W2 = dLoss dW2 ∇W1 = dLoss dW1
26. BACKPROPAGATION 1 1 1 ⦦ ⦦ ⦦ ⦦ ⦦ ⦦ Hotdog Not hotdog ⏐0.45 - 0⏐ = 0.45 ⏐0.95 - 1⏐ = 0.05 0.50 Prediction G round truth Loss 0 10 255 168 ∇W3 ∇W2 ∇W1
27. STOCHASTIC GRADIENT DESCENT one step per mini-batch
28. CONVOLUTIONAL NEURAL NETWORKS
29. WHAT IS A NEURAL NETWORK? 1 1 1 ⦦ ⦦ ⦦ ⦦ ⦦ ⦦ ⦦ 1 Input Layer Hidden Layer Bias Output Layer
30. CONVOLUTIONAL NEURAL NETWORK 1 1 1 ⦦ ⦦ ⦦ ⦦ 1 Input Layer Fully-connected Bias Output Layer Convolutional layer
31. CONVOLUTION OPERATION Input image Weights ∗ = Output image
32. DETECT VERTICAL EDGES 5 5 5 0 0 0 5 5 5 0 0 0 5 5 5 0 0 0 5 5 5 0 0 0 5 5 5 0 0 0 5 5 5 0 0 0 1 0 -1 2 0 -2 1 0 -1 Input data Sobel filter ∗ =
33. DETECT VERTICAL EDGES 5 5 5 0 0 0 5 5 5 0 0 0 5 5 5 0 0 0 5 5 5 0 0 0 5 5 5 0 0 0 5 5 5 0 0 0 1 0 -1 2 0 -2 1 0 -1 0 Input data Sobel filter ∗ = ∑ m ∑ n I(m,n) K(m,n)
34. DETECT VERTICAL EDGES 5 5 5 0 0 0 5 5 5 0 0 0 5 5 5 0 0 0 5 5 5 0 0 0 5 5 5 0 0 0 5 5 5 0 0 0 1 0 -1 2 0 -2 1 0 -1 0 20 Input data Sobel filter ∗ =
35. DETECT VERTICAL EDGES 5 5 5 0 0 0 5 5 5 0 0 0 5 5 5 0 0 0 5 5 5 0 0 0 5 5 5 0 0 0 5 5 5 0 0 0 1 0 -1 2 0 -2 1 0 -1 0 20 20 Input data Sobel filter ∗ =
36. DETECT VERTICAL EDGES 5 5 5 0 0 0 5 5 5 0 0 0 5 5 5 0 0 0 5 5 5 0 0 0 5 5 5 0 0 0 5 5 5 0 0 0 1 0 -1 2 0 -2 1 0 -1 0 20 20 0 Input data Sobel filter ∗ =
37. DETECT VERTICAL EDGES 5 5 5 0 0 0 5 5 5 0 0 0 5 5 5 0 0 0 5 5 5 0 0 0 5 5 5 0 0 0 5 5 5 0 0 0 1 0 -1 2 0 -2 1 0 -1 0 20 20 0 0 ... Input data Sobel filter ∗ =
38. DETECT VERTICAL EDGES 5 5 5 0 0 0 5 5 5 0 0 0 5 5 5 0 0 0 5 5 5 0 0 0 5 5 5 0 0 0 5 5 5 0 0 0 1 0 -1 2 0 -2 1 0 -1 0 20 20 0 0 20 20 0 0 20 20 0 0 20 20 0 Input data Sobel filter Feature map ∗ =
39. USING MULTIPLE FILTERS 5 5 5 0 0 0 5 5 5 0 0 0 5 5 5 0 0 0 5 5 5 0 0 0 5 5 5 0 0 0 5 5 5 0 0 0 Feature map 1 ∗ = Feature map 2
40. LEARNING MULTIPLE FILTERS [Krizhevsky et al., NIPS'12]
41. CONVOLUTIONAL NETWORK: VGG-16 224x224x3 Convolution & ReLU Max Pooling Fully Connected & ReLU FC + SoftMax 224x224x64 112x112x128 56x56x256 28x28x512 14x14x512 7x7x512 4096 1000 112x112x64 56x56x128 28x28x256 14x14x512
42. VISUALIZATION OF CONVOLUTIONAL NETWORKS [Zeiler and Fergus, ECCV 2014]
43. VISUALIZATION OF CONVOLUTIONAL NETWORKS [Zeiler and Fergus, ECCV 2014]
44. [Zeiler and Fergus, ECCV 2014]
45. APPLICATIONS OF CNNS • Image recognition • Facial recognition • Detecting objects in an image • Handwriting recognition • Video analysis • Drug discovery • Medical diagnostics
47. RECURRENT NEURAL NETWORKS
48. WHAT IS A NEURAL NETWORK? 1 1 1 ⦦ ⦦ ⦦ ⦦ ⦦ ⦦ ⦦ 1 Input Layer Hidden Layer Bias Output Layer b W1 x W2 W3 ^ y
49. WHAT IS A NEURAL NETWORK? Input Layer Hidden Layer Output Layer W1 x W2 W3 ^ y
50. RECURRENT NEURAL NETWORKS (WORKING WITH SEQUENCES) xt WI ^ yt Wy WR xt WI ^ yt Wy xt+1 WI ^ yt+1 Wy xt+2 WI ^ yt+2 Wy WR WR
51. RECURRENT NEURAL NETWORKS 1 0 0 0 "e" 0 0 1 0 "l" 0 0 1 0 "l" e 0 h 1 l 0 o 0 "h" = "h" 0 1 0 0 0.7 0.1 0.1 0.1 "e" 0.2 0.2 0.4 0.2 "l" 0.1 0.0 0.9 0.0 "l" 0.1 0.1 0.1 0.7 "o"
52. RECURRENT NEURAL NETWORKS Naturalism and decision for the majority of Arab countries' capitalide was grounded by the Irish language by [[John Clair]], [[An Imperial Japanese Revolt]], associated with Guangzham's sovereignty. His generals were the powerful ruler of the Portugal in the [[Protestant Immineners]], which could be said to be directly in Cantonese Communication, which followed a ceremony and set inspired prison, training. The emperor travelled back to [[Antioch, Perth, October 25|21]] to note, the Kingdom of Costa Rica, unsuccessful fashioned the [[Thrales]], [[Cynth's Dajoard]], known in western [[Scotland]], near Italy to the conquest of India with the conflict. [http://karpathy.github.io/2015/05/21/rnn-effectiveness/]
53. RECURRENT NEURAL NETWORKS this movie was really good <eos> positive 0.81 negative 0.19
54. RECURRENT NEURAL NETWORKS he learned about neural networks<eos> uczył się o sieciach neuronowych
55. APPLICATIONS OF RNNS • Language modelling • Machine translation • Speech recognition • Music composition • Time series prediction • Human action recognition • Protein homology detection
57. HIGHER-ORDER NETWORKS
58. VIDEO CAPTIONING NETWORK Conv Conv Conv Conv people racing to the top of sand dunes
59. GENERATIVE ADVERSARIAL NETWORKS Conv Random noise Deconvolution real 0.81 fake 0.19 Fake image Real images Generator Discriminator
60. this person does not exist.com
61. this person does not exist.com
63. DEEP LEARNING SPECIALIZATION ON COURSERA
64. DEEP LEARNING SOFTWARE
65. DEEP LEARNING FRAMEWORK ARCHITECTURE Python API Graph Intermediate Representation CPU GPU Accelerator Execution backends
66. ngraph.ai
67. THANK YOU michal.karzynski.pl References ImageNet Classification with Deep Convolutional Neural Networks (NIPS, 2012)
 Alex Krizhevsky, Ilya Sutskever, Geoffrey Hinton
 https://papers.nips.cc/paper/4824-imagenet-classification-with-deep-convolutional-neural-networks Visualizing and Understanding Convolutional Networks (ECCV, 2014)
 Matthew Zeiler, Rob Fergus
 https://arxiv.org/abs/1311.2901 The Unreasonable Effectiveness of Recurrent Neural Networks
 Andrej Karpathy
 http://karpathy.github.io/2015/05/21/rnn-effectiveness/ Star Trek images via TrekCore.com. 
 All Star Trek trademarks and images are owned by CBS Studios Inc. Hot-dog image by TheCulinaryGeek on Flickr
 https://www.flickr.com/photos/preppybyday/ 5076302395 All other images from pexels.com
 https://www.pexels.com/ * Other names and brands may be claimed as the property of others