Teaching Machines to Understand What They See

Humans rely a lot on their visual perception to understand the world. Can machines do the same? In this talk we will briefly go over the basics of machine learning with biologically inspired neural networks. We will then discuss the details of convolutional neural networks and how to use them to understand images.

Transcript

1. Teaching Machines to Understand What They See Semih Yagcioglu 1
2. Part I: Background 2
3. Can Machines Think? 3
4. Can Machines Think? 4 Alan M. Turing (1950)
5. What is Intelligence? Intelligence is the computational part of the ability to achieve goals in the world. Varying kinds and degrees of intelligence occur in people, many animals and some machines. 5 Credit: John McCarthy
6. What is Artificial Intelligence? “It is the science and engineering of making intelligent machines, especially intelligent computer programs. It is related to the similar task of using computers to understand human intelligence, but AI does not have to confine itself to methods that are biologically observable.” 6 Credit: John McCarthy
7. Dartmouth AI Project Proposal (1955) “We propose that a 2 month, 10 man study of artificial intelligence be carried out during the summer of 1956 at Dartmouth College in Hanover, New Hampshire.” The study is to proceed on the basis of the conjecture that every aspect of learning or any other feature of intelligence can in principle be so precisely described that a machine can be made to simulate it. An attempt will be made to find how to make machines use language, form abstractions and concepts, solve kinds of problems now reserved for humans, and improve themselves. We think that a significant advance can be made in one or more of these problems if a carefully selected group of scientists work on it together for a summer.” 7 Slide Credit: Dhruv Batra
8. Timeline • 1940’s — Interest in neurons, neural networks and their relationship to mathematics and learning • 1943 — McCulloch & Pitts: “A logical calculus of the ideas immanent in nervous activity” • 1950 — Turing’s paper “Computing Machinery and Intelligence”, A.M. Turing introduced “The Turing Test”. • 1956 — Dartmouth conference • 1950’s and 1960’s — enthusiasm and optimism; big promises • Late 1960’s and 1970’s — Realization that further progress was really hard; disillusionment • 1980’s — Expert Systems, neural networks, etc.; AI now a little different; quiet successes • 1990’s to present — Intelligent agents • 2000’s — robot pets, self-driving cars 8 Credit: Peter Norvig
9. Discovery of Neurons 9 Ramón y Cajal, 1900
10. Structure of a Neuron 10 Inputs arrive at dendrites, and axons serve as output channel
11. Artificial Neurons 11 McCulloch and Pitts, 1947
12. Artificial Neurons 12 McCulloch and Pitts, 1947
13. Implementing Logic Gates 13 Logic AND gate with McCulloch and Pitts neuron
14. What’s Missing? This type of mechanism is very useful for modelling a simple function but there is no learning at all. 14
15. What is Learning? “the acquisition of knowledge or skills through experience, study, or by being taught.” 15 Slide Credit: Dhruv Batra
16. How Do We Learn? 16 By seeing examples, many many of them
17. How Do We Learn? 17 By making decisions and seeing their outcomes
18. How Do Machines Learn? 18 How about mimicking biological learning?
19. What is Machine Learning? “algorithms that improve their performance (P) at some task (T) with experience (E)” - Tom Mitchell 19
20. A Challenge Let’s say you want to solve Character Recognition 20 Slide Credit: Dhruv Batra
21. A Challenge Hard way: Understand handwriting/characters 21 Slide Credit: Dhruv Batra
22. A Challenge Lazy way: Throw data 22 Slide Credit: Dhruv Batra
23. Comparison 23 Should we code every detail, or let the computer decide?
24. Machine Learning in a Nutshell • Tens of thousands of machine learning algorithms – Hundreds new every year • Decades of ML research oversimplified: – All of Machine Learning: – Learn a mapping from input to output f: X → Y – X: emails, Y: {spam, notspam} 24 Slide Credit: Pedro Domingos
25. Machine Learning in a Nutshell • Input: x (images, text, emails…) • Output: y (spam or non-spam…) • (Unknown) Target Function – f: X → Y (the “true” mapping / reality) • Data – (x 1 ,y 1 ), (x 2 ,y 2 ), …, (x N ,y N ) • Model / Hypothesis Class – g: X → Y – y = g(x) = sign(wTx) 25 Slide Credit: Dhruv Batra
26. Machine Learning in a Nutshell • Every machine learning algorithm has three components: – Representation / Model Class – Evaluation / Objective Function – Optimization 26 Slide Credit: Pedro Domingos
27. Perceptron 27 Frank Rosenblatt (1957)
28. Mark I Perceptron Machine 28 First implementation of the perceptron algorithm
29. Perceptron 29 Frank Rosenblatt (1957)
30. Perceptron Learning Rule • initialize random weights wi, set learning rate a = 0.1 • while termination condition is satisfied ⚬ for each training example ( x, y ) ⚬ calculate the output: o = fthreshold( wi*xi ) ⚬ if the Perceptron does not respond correctly update the weights • wi = wi + a ( y - o ) xi • where y is the desired output, o is the output generated by the Perceptron, wi is the weight associated with the i-th connection. 30
31. Perceptron Learning Rule Suppose an example of perceptron which accepts two inputs x1 = 2 and x2 = 1, with weights w1 = 0.5 and w2 = 0.3 and w0 = -1. The output of the perceptron is : O = 2 * 0.5 + 1 * 0.3 - 1 = 0.3 Therefore the output is 1. If the correct output however is 0, the weights will be adjusted according to the Perceptron rule as follows: w 1 = 0.5 + ( 0 - 1 ) * 2 = - 1.5 w 2 = 0.3 + ( 0 - 1 ) * 1 = - 0.7 w 0 = - 1 + ( 0 - 1 ) * 1 = - 2 31
32. Linear Classification We have a set of points to classify 32
33. Linear Classification Perfect! 33
34. Linear Classification But we have more data! 34
35. Linear Classification It’s not working any more 35
36. Separability 36
37. XOR Problem 37
38. Multilayer Perceptron 38
39. Multilayer Perceptron 39 Slide Credit: Fei-Fei Li & Andrej Karpathy & Justin Johnson “2-layer Neural Net”, or “1-hidden-layer Neural Net”
40. Multilayer Perceptron 40 Slide Credit: Fei-Fei Li & Andrej Karpathy & Justin Johnson “3-layer Neural Net”, or “2-hidden-layer Neural Net”
41. Multilayer Perceptron 41 Slide Credit: Luis Serrano
42. Multilayer Perceptron 42 Slide Credit: Luis Serrano
43. Multilayer Perceptron 43 Slide Credit: Luis Serrano
44. Multilayer Perceptron 44 Slide Credit: Luis Serrano
45. Gradient Descent 45 Slide Credit: Fei-Fei Li & Andrej Karpathy & Justin Johnson
46. Backpropagation 46 Slide Credit: Siraj Raval
47. Chain Rule 47 Slide Credit: Fei-Fei Li & Andrej Karpathy & Justin Johnson e.g. x = -2, y = 5, z = -4 Want: Chain rule:
48. Playground playground.tensorflow.org 48
49. Part II: Understanding Images 49
50. Understanding Images Is Hard 50 Slide Credit: Fei-Fei Li & Andrej Karpathy & Justin Johnson
51. Image Classification 51 Slide Credit: Fei-Fei Li & Andrej Karpathy & Justin Johnson cat (assume given set of discrete labels) {dog, cat, truck, plane, ...}
52. Problem: Semantic Gap 52 Slide Credit: Fei-Fei Li & Andrej Karpathy & Justin Johnson
53. Challenge: Viewpoint Variation 53 Slide Credit: Fei-Fei Li & Andrej Karpathy & Justin Johnson
54. Challenge: Illumination 54 Slide Credit: Fei-Fei Li & Andrej Karpathy & Justin Johnson
55. Challenge: Deformation 55 Slide Credit: Fei-Fei Li & Andrej Karpathy & Justin Johnson
56. Challenge: Occlusion 56 Slide Credit: Fei-Fei Li & Andrej Karpathy & Justin Johnson
57. Challenge: Background Clutter 57 Slide Credit: Fei-Fei Li & Andrej Karpathy & Justin Johnson
58. Challenge: Intraclass Variance 58 Slide Credit: Fei-Fei Li & Andrej Karpathy & Justin Johnson
59. How Our Visual System Works? 59
60. Hubel and Wiesel’s Cat Experiment 60 Neurons fire when there is certain type of stimulus
61. Neocognitron 61 Kunihiko Fukushima (1980)
62. Neocognitron 62 Input interconnections to the cells within a single cell plane
63. Neocognitron 63 Learning is done greedily in each layer
64. Neocognitron 64
65. Neocognitron 65
66. Neocognitron 66
67. Convolutional Neural Networks 67 Image Credit: Vojtech Pavlovsky
68. Convolution 68 Slide Credit: Vicky Kalogeiton
69. Convolution 69 Slide Credit: Vicky Kalogeiton
70. Reasoning Behind CNNs 70 Slide Credit: Marc'Aurelio Ranzato
71. Reasoning Behind CNNs 71 Slide Credit: Marc'Aurelio Ranzato Reduce connection to local regions
72. Reasoning Behind CNNs 72 Slide Credit: Marc'Aurelio Ranzato Reuse the same kernel everywhere
73. Reasoning Behind CNNs 73 Slide Credit: Marc'Aurelio Ranzato Learn multiple filters
74. Reasoning Behind CNNs 74 Slide Credit: Marc'Aurelio Ranzato Build translation invariance via spatial pooling
75. CNN Components 75 Slide Credit: A. Vedaldi
76. 32 32 3 Convolution Layer 32x32x3 image width height depth 76 Slide Credit: Fei-Fei Li & Andrej Karpathy & Justin Johnson
77. 32 32 3 Convolution Layer 5x5x3 filter 32x32x3 image Convolve the filter with the image i.e. “slide over the image spatially, computing dot products” 77 Slide Credit: Fei-Fei Li & Andrej Karpathy & Justin Johnson
78. 32 32 3 Convolution Layer 5x5x3 filter 32x32x3 image Convolve the filter with the image i.e. “slide over the image spatially, computing dot products” Filters always extend the full depth of the input volume 78 Slide Credit: Fei-Fei Li & Andrej Karpathy & Justin Johnson
79. 32 32 3 Convolution Layer 32x32x3 image 5x5x3 filter 1 number: the result of taking a dot product between the filter and a small 5x5x3 chunk of the image (i.e. 5*5*3 = 75-dimensional dot product + bias) 79 Slide Credit: Fei-Fei Li & Andrej Karpathy & Justin Johnson
80. 32 32 3 Convolution Layer 32x32x3 image 5x5x3 filter convolve (slide) over all spatial locations activation map 1 28 28 80 Slide Credit: Fei-Fei Li & Andrej Karpathy & Justin Johnson
81. 32 32 3 Convolution Layer 32x32x3 image 5x5x3 filter convolve (slide) over all spatial locations activation maps 1 28 28 consider a second, green filter 81 Slide Credit: Fei-Fei Li & Andrej Karpathy & Justin Johnson
82. 32 32 3 Convolution Layer activation maps 6 28 28 For example, if we had 6 5x5 filters, we’ll get 6 separate activation maps: We stack these up to get a “new image” of size 28x28x6! 82 Slide Credit: Fei-Fei Li & Andrej Karpathy & Justin Johnson
83. Preview: ConvNet is a sequence of Convolution Layers, interspersed with activation functions 32 32 3 28 28 6 CONV, ReLU e.g. 6 5x5x3 filters 83 Slide Credit: Fei-Fei Li & Andrej Karpathy & Justin Johnson
84. Preview: ConvNet is a sequence of Convolutional Layers, interspersed with activation functions 32 32 3 CONV, ReLU e.g. 6 5x5x3 filters 28 28 6 CONV, ReLU e.g. 10 5x5x6 filters CONV, ReLU …. 10 24 24 84 Slide Credit: Fei-Fei Li & Andrej Karpathy & Justin Johnson
85. Successive Layers Learns Higher-Level Features 85 Slide Credit: David W. Corne
86. Featural Hierarchy 86
87. Feature Visualization 87 Slide Credit: Yann Lecun
88. A closer look at spatial dimensions: 32 32 3 32x32x3 image 5x5x3 filter convolve (slide) over all spatial locations activation map 1 28 28 88 Slide Credit: Fei-Fei Li & Andrej Karpathy & Justin Johnson
89. 7x7 input (spatially) assume 3x3 filter 7 7 A closer look at spatial dimensions: 89 Slide Credit: Fei-Fei Li & Andrej Karpathy & Justin Johnson
90. 7x7 input (spatially) assume 3x3 filter 7 7 A closer look at spatial dimensions: 90 Slide Credit: Fei-Fei Li & Andrej Karpathy & Justin Johnson
91. 7x7 input (spatially) assume 3x3 filter 7 7 A closer look at spatial dimensions: 91 Slide Credit: Fei-Fei Li & Andrej Karpathy & Justin Johnson
92. 7x7 input (spatially) assume 3x3 filter 7 7 A closer look at spatial dimensions: 92 Slide Credit: Fei-Fei Li & Andrej Karpathy & Justin Johnson
93. 7x7 input (spatially) assume 3x3 filter => 5x5 output 7 7 A closer look at spatial dimensions: 93 Slide Credit: Fei-Fei Li & Andrej Karpathy & Justin Johnson
94. 7x7 input (spatially) assume 3x3 filter applied with stride 2 7 7 A closer look at spatial dimensions: 94 Slide Credit: Fei-Fei Li & Andrej Karpathy & Justin Johnson
95. 7x7 input (spatially) assume 3x3 filter applied with stride 2 7 7 A closer look at spatial dimensions: 95 Slide Credit: Fei-Fei Li & Andrej Karpathy & Justin Johnson
96. 7x7 input (spatially) assume 3x3 filter applied with stride 2 => 3x3 output! 7 7 A closer look at spatial dimensions: 96 Slide Credit: Fei-Fei Li & Andrej Karpathy & Justin Johnson
97. 7x7 input (spatially) assume 3x3 filter applied with stride 3? 7 7 A closer look at spatial dimensions: 97 Slide Credit: Fei-Fei Li & Andrej Karpathy & Justin Johnson
98. 7x7 input (spatially) assume 3x3 filter applied with stride 3? 7 7 A closer look at spatial dimensions: doesn’t fit! cannot apply 3x3 filter on 7x7 input with stride 3. 98 Slide Credit: Fei-Fei Li & Andrej Karpathy & Justin Johnson
99. N N F F Output size: (N - F) / stride + 1 e.g. N = 7, F = 3: stride 1 => (7 - 3)/1 + 1 = 5 stride 2 => (7 - 3)/2 + 1 = 3 stride 3 => (7 - 3)/3 + 1 = 2.33 :\ 99 Slide Credit: Fei-Fei Li & Andrej Karpathy & Justin Johnson
100. In practice: Common to zero pad the border 0 0 0 0 0 0 0 0 0 0 e.g. input 7x7 3x3 filter, applied with stride 1 pad with 1 pixel border => what is the output? (recall:) (N - F) / stride + 1 100 Slide Credit: Fei-Fei Li & Andrej Karpathy & Justin Johnson
101. In practice: Common to zero pad the border e.g. input 7x7 3x3 filter, applied with stride 1 pad with 1 pixel border => what is the output? 7x7 output! 0 0 0 0 0 0 0 0 0 0 101 Slide Credit: Fei-Fei Li & Andrej Karpathy & Justin Johnson
102. 102 In practice: Common to zero pad the border e.g. input 7x7 3x3 filter, applied with stride 1 pad with 1 pixel border => what is the output? 7x7 output! in general, common to see CONV layers with stride 1, filters of size FxF, and zero-padding with (F-1)/2. (will preserve size spatially) e.g. F = 3 => zero pad with 1 F = 5 => zero pad with 2 F = 7 => zero pad with 3 0 0 0 0 0 0 0 0 0 0 Slide Credit: Fei-Fei Li & Andrej Karpathy & Justin Johnson
103. 32 32 3 CONV, ReLU e.g. 6 5x5x3 filters 28 28 6 CONV, ReLU e.g. 10 5x5x6 filters CONV, ReLU …. 10 24 24 Convolving 103 Slide Credit: Fei-Fei Li & Andrej Karpathy & Justin Johnson shrinks volumes spatially! (32 -> 28 -> 24 ...)
104. Input volume: 32x32x3 10 5x5 filters with stride 1, pad 2 Output volume size: ? Example 104 Slide Credit: Fei-Fei Li & Andrej Karpathy & Justin Johnson
105. Input volume: 32x32x3 10 5x5 filters with stride 1, pad 2 Output volume size: (32+2*2-5)/1+1 = 32 spatially, so 32x32x10 Example 105 Slide Credit: Fei-Fei Li & Andrej Karpathy & Justin Johnson
106. Input volume: 32x32x3 10 5x5 filters with stride 1, pad 2 Number of parameters in this layer? Example 106 Slide Credit: Fei-Fei Li & Andrej Karpathy & Justin Johnson
107. Input volume: 32x32x3 10 5x5 filters with stride 1, pad 2 Number of parameters in this layer? each filter has 5*5*3 + 1 = 76 params (+1 for bias) => 76*10 = 760 Example 107 Slide Credit: Fei-Fei Li & Andrej Karpathy & Justin Johnson
108. 108 32 32 3 32x32x3 image 5x5x3 filter 1 number: the result of taking a dot product between the filter and this part of the image (i.e. 5*5*3 = 75-dimensional dot product) The brain/neuron view of CONV Layer Slide Credit: Fei-Fei Li & Andrej Karpathy & Justin Johnson
109. 32 32 3 32x32x3 image 5x5x3 filter 1 number: the result of taking a dot product between the filter and this part of the image (i.e. 5*5*3 = 75-dimensional dot product) It’s just a neuron with local connectivity... The brain/neuron view of CONV Layer 109 Slide Credit: Fei-Fei Li & Andrej Karpathy & Justin Johnson
110. 32 32 3 An activation map is a 28x28 sheet of neuron outputs: 1. Each is connected to a small region in the input 2. All of them share parameters “5x5 filter” -> “5x5 receptive field for each neuron” 28 28 The brain/neuron view of CONV Layer 110 Slide Credit: Fei-Fei Li & Andrej Karpathy & Justin Johnson
111. 32 32 3 28 28 E.g. with 5 filters, CONV layer consists of neurons arranged in a 3D grid (28x28x5) There will be 5 different neurons all looking at the same region in the input volume 5 The brain/neuron view of CONV Layer 111 Slide Credit: Fei-Fei Li & Andrej Karpathy & Justin Johnson
112. Pooling Layer 112 Slide Credit: Fei-Fei Li & Andrej Karpathy & Justin Johnson Smaller representations, adds translation invariance
113. 1 1 2 4 5 6 7 8 3 2 1 0 1 2 3 4 Single depth slice x y max pool with 2x2 filters and stride 2 6 8 3 4 Max Pooling 113 Slide Credit: Fei-Fei Li & Andrej Karpathy & Justin Johnson
114. Hierarchical Representation 114 Slide Credit: Greg Shakhnarovich trade off between localization and abstraction
115. Patching Things Up 115 Slide Credit: Fei-Fei Li & Andrej Karpathy & Justin Johnson
116. Part III: Building a CNN from Scratch 116
117. Resources Tutorial code and accompanying notebook is available at: semihyagcioglu.com/talks/teaching-machines-to-see 117