Deep Learning Atlas

Deep Learning Atlas

What are neural networks? What makes them recurrent or convolutional? This talk will give you an introduction to and a whirlwind tour of the field of Deep Learning. We'll start by explaining how simple artificial neural networks work and how they learn though backpropagation. We'll then go over different network architectures used in computer vision, sequence analysis, neural machine translation and other areas.

6082203cf72bc1220f6b7984bfbbad11?s=128

Michał Karzyński

April 08, 2019
Tweet

Transcript

  1. DEEP LEARNING ATLAS Introduction to Neural Networks Michał Karzyński •

    4Developers 2019
  2. ABOUT THIS TALK • What are neural networks? • How

    do they learn? • Convolutional Neural Networks (CNNs) • Recurrent neural networks (RNNs) • Generative adversarial networks (GANs)
  3. ABOUT ME • Michał Karzyński (@postrational) • Full stack geek

    (C++, Python, JavaScript) • I blog at http://michal.karzynski.pl • I’m a tech lead at working on
  4. None
  5. WHAT IS A NEURAL NETWORK?

  6. A universal function approximator.

  7. WHAT IS A FUNCTION? Input Output Function (mapping)

  8. WHAT IS A FUNCTION? Hotdog Not Hotdog Function (mapping)

  9. NEURAL NETWORK 
 CAN APPROXIMATE ANY FUNCTION Hotdog Not Hotdog

    1 1 1 ⦦ ⦦ ⦦ ⦦ ⦦ ⦦
  10. WHAT IS A NEURAL NETWORK? 1 1 1 ⦦ ⦦

    ⦦ ⦦ ⦦ ⦦ ⦦ 1 Input Layer Hidden Layer Bias Output Layer
  11. 1 b W HOW DO NEURAL NETWORKS WORK? 1 1

    ⦦ ⦦ ⦦ Hotdog Not hotdog 0 10 255 168 x ⦦ ⦦ ⦦
  12. 1 b W HOW DO NEURAL NETWORKS WORK? 1 1

    ⦦ ⦦ ⦦ Hotdog Not hotdog 0 10 255 168 n ∑ i=1 wi xi + b x ⦦ ⦦ ⦦
  13. ACTIVATION FUNCTIONS ReLU 0 1,5 3 4,5 6 -6 -3

    0 3 6 Sigmoid 0 0,25 0,5 0,75 1 -6 -3 0 3 6 Tanh -1 -0,5 0 0,5 1 -6 -3 0 3 6
  14. HOW DO NEURAL NETWORKS WORK? 1 1 1 ⦦ ⦦

    ⦦ ⦦ ⦦ ⦦ Hotdog Not hotdog 0 10 255 168 ReLU ( n ∑ i=1 wi xi + b )
  15. HOW DO NEURAL NETWORKS WORK? 1 1 1 ⦦ ⦦

    ⦦ ⦦ ⦦ ⦦ Hotdog Not hotdog 0 10 255 168
  16. HOW DO NEURAL NETWORKS WORK? 1 1 1 ⦦ ⦦

    ⦦ ⦦ ⦦ ⦦ Hotdog Not hotdog 0.10 0.90 Prediction 0 10 255 168
  17. HOW DO NEURAL NETWORKS LEARN?

  18. INFERENCE TRAINING

  19. HOW DO NEURAL NETWORKS LEARN? 1 1 1 ⦦ ⦦

    ⦦ ⦦ ⦦ ⦦ Hotdog Not hotdog 0.50 0.90 Prediction 0 10 255 168
  20. HOW DO NEURAL NETWORKS LEARN? 1 1 1 ⦦ ⦦

    ⦦ ⦦ ⦦ ⦦ Hotdog Not hotdog 0.50 0 0.90 1 Prediction G round truth 0 10 255 168
  21. HOW DO NEURAL NETWORKS LEARN? 1 1 1 ⦦ ⦦

    ⦦ ⦦ ⦦ ⦦ Hotdog Not hotdog ⏐0.50 - 0⏐ = 0.50 ⏐0.90 - 1⏐ = 0.10 0.60 Prediction G round truth Loss 0 10 255 168
  22. HOW DO NEURAL NETWORKS LEARN? 1 1 1 ⦦ ⦦

    ⦦ ⦦ ⦦ ⦦ Hotdog Not hotdog ⏐0.55 - 0⏐ = 0.55 ⏐0.85 - 1⏐ = 0.15 0.70 Prediction G round truth Loss 0 10 255 168
  23. HOW DO NEURAL NETWORKS LEARN? 1 1 1 ⦦ ⦦

    ⦦ ⦦ ⦦ ⦦ Hotdog Not hotdog ⏐0.45 - 0⏐ = 0.45 ⏐0.95 - 1⏐ = 0.05 0.50 Prediction G round truth Loss 0 10 255 168
  24. HOW DO NEURAL NETWORKS LEARN? • Backpropagation - calculate the

    gradients using derivatives • Stochastic gradient descent - use gradients to adjust weights
  25. BACKPROPAGATION 1 1 1 ⦦ ⦦ ⦦ ⦦ ⦦ ⦦

    Hotdog Not hotdog ⏐0.45 - 0⏐ = 0.45 ⏐0.95 - 1⏐ = 0.05 0.50 Prediction G round truth Loss 0 10 255 168 ∇W3 = dLoss dW3 ∇W2 = dLoss dW2 ∇W1 = dLoss dW1
  26. BACKPROPAGATION 1 1 1 ⦦ ⦦ ⦦ ⦦ ⦦ ⦦

    Hotdog Not hotdog ⏐0.45 - 0⏐ = 0.45 ⏐0.95 - 1⏐ = 0.05 0.50 Prediction G round truth Loss 0 10 255 168 ∇W3 ∇W2 ∇W1
  27. STOCHASTIC GRADIENT DESCENT one step per mini-batch

  28. CONVOLUTIONAL NEURAL NETWORKS

  29. WHAT IS A NEURAL NETWORK? 1 1 1 ⦦ ⦦

    ⦦ ⦦ ⦦ ⦦ ⦦ 1 Input Layer Hidden Layer Bias Output Layer
  30. CONVOLUTIONAL NEURAL NETWORK 1 1 1 ⦦ ⦦ ⦦ ⦦

    1 Input Layer Fully-connected Bias Output Layer Convolutional layer
  31. CONVOLUTION OPERATION Input image Weights ∗ = Output image

  32. DETECT VERTICAL EDGES 5 5 5 0 0 0 5

    5 5 0 0 0 5 5 5 0 0 0 5 5 5 0 0 0 5 5 5 0 0 0 5 5 5 0 0 0 1 0 -1 2 0 -2 1 0 -1 Input data Sobel filter ∗ =
  33. DETECT VERTICAL EDGES 5 5 5 0 0 0 5

    5 5 0 0 0 5 5 5 0 0 0 5 5 5 0 0 0 5 5 5 0 0 0 5 5 5 0 0 0 1 0 -1 2 0 -2 1 0 -1 0 Input data Sobel filter ∗ = ∑ m ∑ n I(m,n) K(m,n)
  34. DETECT VERTICAL EDGES 5 5 5 0 0 0 5

    5 5 0 0 0 5 5 5 0 0 0 5 5 5 0 0 0 5 5 5 0 0 0 5 5 5 0 0 0 1 0 -1 2 0 -2 1 0 -1 0 20 Input data Sobel filter ∗ =
  35. DETECT VERTICAL EDGES 5 5 5 0 0 0 5

    5 5 0 0 0 5 5 5 0 0 0 5 5 5 0 0 0 5 5 5 0 0 0 5 5 5 0 0 0 1 0 -1 2 0 -2 1 0 -1 0 20 20 Input data Sobel filter ∗ =
  36. DETECT VERTICAL EDGES 5 5 5 0 0 0 5

    5 5 0 0 0 5 5 5 0 0 0 5 5 5 0 0 0 5 5 5 0 0 0 5 5 5 0 0 0 1 0 -1 2 0 -2 1 0 -1 0 20 20 0 Input data Sobel filter ∗ =
  37. DETECT VERTICAL EDGES 5 5 5 0 0 0 5

    5 5 0 0 0 5 5 5 0 0 0 5 5 5 0 0 0 5 5 5 0 0 0 5 5 5 0 0 0 1 0 -1 2 0 -2 1 0 -1 0 20 20 0 0 ... Input data Sobel filter ∗ =
  38. DETECT VERTICAL EDGES 5 5 5 0 0 0 5

    5 5 0 0 0 5 5 5 0 0 0 5 5 5 0 0 0 5 5 5 0 0 0 5 5 5 0 0 0 1 0 -1 2 0 -2 1 0 -1 0 20 20 0 0 20 20 0 0 20 20 0 0 20 20 0 Input data Sobel filter Feature map ∗ =
  39. USING MULTIPLE FILTERS 5 5 5 0 0 0 5

    5 5 0 0 0 5 5 5 0 0 0 5 5 5 0 0 0 5 5 5 0 0 0 5 5 5 0 0 0 Feature map 1 ∗ = Feature map 2
  40. LEARNING MULTIPLE FILTERS [Krizhevsky et al., NIPS'12]

  41. CONVOLUTIONAL NETWORK: VGG-16 224x224x3 Convolution & ReLU Max Pooling Fully

    Connected & ReLU FC + SoftMax 224x224x64 112x112x128 56x56x256 28x28x512 14x14x512 7x7x512 4096 1000 112x112x64 56x56x128 28x28x256 14x14x512
  42. VISUALIZATION OF CONVOLUTIONAL NETWORKS [Zeiler and Fergus, ECCV 2014]

  43. VISUALIZATION OF CONVOLUTIONAL NETWORKS [Zeiler and Fergus, ECCV 2014]

  44. [Zeiler and Fergus, ECCV 2014]

  45. APPLICATIONS OF CNNS • Image recognition • Facial recognition •

    Detecting objects in an image • Handwriting recognition • Video analysis • Drug discovery • Medical diagnostics
  46. None
  47. RECURRENT NEURAL NETWORKS

  48. WHAT IS A NEURAL NETWORK? 1 1 1 ⦦ ⦦

    ⦦ ⦦ ⦦ ⦦ ⦦ 1 Input Layer Hidden Layer Bias Output Layer b W1 x W2 W3 ^ y
  49. WHAT IS A NEURAL NETWORK? Input Layer Hidden Layer Output

    Layer W1 x W2 W3 ^ y
  50. RECURRENT NEURAL NETWORKS (WORKING WITH SEQUENCES) xt WI ^ yt

    Wy WR xt WI ^ yt Wy xt+1 WI ^ yt+1 Wy xt+2 WI ^ yt+2 Wy WR WR
  51. RECURRENT NEURAL NETWORKS 1 0 0 0 "e" 0 0

    1 0 "l" 0 0 1 0 "l" e 0 h 1 l 0 o 0 "h" = "h" 0 1 0 0 0.7 0.1 0.1 0.1 "e" 0.2 0.2 0.4 0.2 "l" 0.1 0.0 0.9 0.0 "l" 0.1 0.1 0.1 0.7 "o"
  52. RECURRENT NEURAL NETWORKS Naturalism and decision for the majority of

    Arab countries' capitalide was grounded by the Irish language by [[John Clair]], [[An Imperial Japanese Revolt]], associated with Guangzham's sovereignty. His generals were the powerful ruler of the Portugal in the [[Protestant Immineners]], which could be said to be directly in Cantonese Communication, which followed a ceremony and set inspired prison, training. The emperor travelled back to [[Antioch, Perth, October 25|21]] to note, the Kingdom of Costa Rica, unsuccessful fashioned the [[Thrales]], [[Cynth's Dajoard]], known in western [[Scotland]], near Italy to the conquest of India with the conflict. [http://karpathy.github.io/2015/05/21/rnn-effectiveness/]
  53. RECURRENT NEURAL NETWORKS this movie was really good <eos> positive

    0.81 negative 0.19
  54. RECURRENT NEURAL NETWORKS he learned about neural networks<eos> uczył się

    o sieciach neuronowych
  55. APPLICATIONS OF RNNS • Language modelling • Machine translation •

    Speech recognition • Music composition • Time series prediction • Human action recognition • Protein homology detection
  56. None
  57. HIGHER-ORDER NETWORKS

  58. VIDEO CAPTIONING NETWORK Conv Conv Conv Conv people racing to

    the top of sand dunes
  59. GENERATIVE ADVERSARIAL NETWORKS Conv Random noise Deconvolution real 0.81 fake

    0.19 Fake image Real images Generator Discriminator
  60. this person does not exist.com

  61. this person does not exist.com

  62. None
  63. DEEP LEARNING SPECIALIZATION ON COURSERA

  64. DEEP LEARNING SOFTWARE

  65. DEEP LEARNING FRAMEWORK ARCHITECTURE Python API Graph Intermediate Representation CPU

    GPU Accelerator Execution backends
  66. ngraph.ai

  67. THANK YOU michal.karzynski.pl References ImageNet Classification with Deep Convolutional Neural

    Networks (NIPS, 2012)
 Alex Krizhevsky, Ilya Sutskever, Geoffrey Hinton
 https://papers.nips.cc/paper/4824-imagenet-classification-with-deep-convolutional-neural-networks Visualizing and Understanding Convolutional Networks (ECCV, 2014)
 Matthew Zeiler, Rob Fergus
 https://arxiv.org/abs/1311.2901 The Unreasonable Effectiveness of Recurrent Neural Networks
 Andrej Karpathy
 http://karpathy.github.io/2015/05/21/rnn-effectiveness/ Star Trek images via TrekCore.com. 
 All Star Trek trademarks and images are owned by CBS Studios Inc. Hot-dog image by TheCulinaryGeek on Flickr
 https://www.flickr.com/photos/preppybyday/ 5076302395 All other images from pexels.com
 https://www.pexels.com/ * Other names and brands may be claimed as the property of others