An Introduction to Deep Learning

Deep Learning for computer vision. First presented at the School of Computing Sciences, University of East Anglia, Norwich, UK.
This version was presented at PyData London, January 2016.

Transcript

1. Deep Learning An Introductory Tutorial G. French University of East Anglia & Kings College London Image montages from http://www.image-net.org
2. ImageNet
3. Image classification dataset
4. ~1,000,000 images ~1,000 classes Ground truths prepared manually through Amazon Mechanical Turk
5. ImageNet Top-5 challenge: You score if ground truth class is one your top 5 predictions
6. ImageNet in 2012 Best approaches used hand-crafted features (SIFT, HOGs, Fisher vectors, etc) + classifier Top-5 error rate: ~25%
7. Then the game changed.
8. Krizhevsky, Sutskever and Hinton; ImageNet Classification with Deep Convolutional Neural networks [Krizhevsky12] Top-5 error rate of ~15%
9. In the last few years, more modern networks have achieved better results still [Simonyan14, He15] Top-5 error rates of ~5-7%
10. I hope this talk will give you an idea of how!
11. What is a neural network?
12. Multiple layers Data propagates through layers Transformed by each layer
13. Neural network Input layer Hidden layer 0 Hidden layer 1 Output layer ⋯ Inputs Outputs ⋯
14. Single layer of a neural network () Input vector Weighted connections Bias Activation function / non-linearity Layer activation
15. = input (M-element vector) = output (N-element vector) = network weights (NxM matrix) = bias (N-element vector) = activation function; tanh / sigmoid / ReLU = ( + )
16. Multiple layers Input vector ( + ) Hidden layer 0 activation ( + ) Hidden layer 1 activation ( + ) Final layer activation (output) ( + ) ⋯
17. In a nutshell: = ( + )
18. Repeat for each layer: 0 = (0 + 0 ) 1 = 1 0 + 1 ⋯ = ( −1 + )
19. As a classifier Input vector Hidden layer 0 activation Final layer activation with softmax (output) ⋯ Image pixels = 0.25 = 0.5 = 0.1 = 0.15 Class probabilities
20. Training a neural network
21. Learn values for parameters; and (for each layer) Use back-propagation
22. Initialise weights randomly (more on this later) Initialise biases to 0
23. Iteratively modify parameters with gradient descent
24. Take example from training set, evaluate network given the training input; =
25. The cost (sometimes called loss), is a measure of the difference between network output and ground truth output
26. Compute the derivative of cost w.r.t. params ( and )
27. Update parameters: 0 ′ = 0 − 0 0 ′ = 0 − 0 γ = learning rate
28. In practice this is done on a mini-batch of examples (e.g. 128) in parallel per pass Compute cost for each example, then average. Compute derivative of average cost w.r.t. params.
29. Using mini-batches works well when using a GPU since computations can be parallelised
30. Cost function
31. Regression Final layer: no activation function / identity. Cost: Sum of squared differences
32. Classification Just like logistic regression
33. Final layer: softmax as activation function ; output vector of class probabilities Cost: negative-log-likelihood / categorical cross-entropy
34. Fully connected neural networks
35. Simplest model Each unit in each layer is connected too all units in previous layer All we have considered so far
36. How well does this perform on image classification?
37. MNIST hand-written digit dataset 28x28 images, 10 classes 60K training examples, 10K validation, 10K test Examples from MNIST
38. Network: 1 hidden layer of 64 units after 300 iterations over training set: 2.85% validation error Hidden layer weights visualised as 28x28 images Input Hidden Output 784 (28x28 images) 64 10
39. Network: 2 hidden layers, both 256 units after 300 iterations over training set: 1.83% validation error Input Hidden 784 (28x28 images) 256 Hidden Output 256 10
40. MNIST is quite a special case Digits nicely centred within image Scaled to approx. same size
41. The fully connected networks so far have a weakness: No translation invariance; learned features are position dependent
42. For more general imagery: requires a training set large enough to see all features in all possible positions… Requires network with enough units to represent this…
43. Convolutional networks
44. Convolution Slide a convolution kernel over an image Multiply image pixels by kernel pixels and sum
45. Convolution Convolutions are often used for feature detection
46. A brief detour…
47. Gabor filters ∗
48. Used for texture classification Bares similarity to low levels of cat visual system [Jones87]
49. Back on track to… Convolutional networks
50. Recap: FC (fully-connected) layer () Input vector Weighted connections Bias Activation function / non-linearity Layer activation
51. Convolutional layer Each unit only connected to units in its neighbourhood
52. Convolutional layer Weights are shared Red weights have same value As do greens… And yellows
53. The values of the weights form a convolution kernel For practical computer vision, more an one kernel must be used to extract a variety of features
54. Convolutional layer Different weight-kernels: Output is vector/image with multiple channels
55. Still = ( + ) As convolution can be expressed as multiplication by weight matrix
56. Note In subsequent layers, each kernel connects to pixels in ALL channels in previous layer
57. Max-pooling ‘layer’ [Ciresan12] Take maximum value from each (, ) pooling region Down-samples image by factor Operates on channels independently
58. These are the models that have been getting excellent ImageNet results
59. How about an example?
60. A Simplified LeNet [LeCun95] for MNIST digits
61. Simplified LeNet for MNIST digits 28 28 24 24 Input Output 1 20 Conv: 20 5x5 kernels Maxpool 2x2 12 8 8 20 50 4 4 50 Conv: 50 5x5 kernels Maxpool 2x2 256 10 Fully connected (flatten and) fully connected 12
62. after 300 iterations over training set: 99.21% validation accuracy Model Error FC64 2.85% FC256--FC256 1.83% 20C5--MP2--50C5--MP2--FC256 0.79%
63. What about the learned kernels? Image taken from paper [Krizhevsky12] Gabor filters
64. Neural networks – recent developments
65. GOOD NEWS Training neural networks is more practical now
66. More processing power Less of a black art
67. (most important) IMPROVEMENT Processing power
68. Image processing requires large networks with perhaps millions of parameters Lots of training examples need to train Easily results in billions or even trillions of FLOPS
69. Neural networks are ‘embarrassingly parallelisable’ therefore ideally suited to GPUs Use GPUs for all but the smallest of networks
70. As of now, nVidia is the most popular make of GPU. Cheaper gaming cards perfectly adequate Only use Tesla in production
71. IMPROVEMENT New popular activation function: ReLU - Rectified Linear Unit
72. ReLU - Rectified Linear Unit = max(, 0)
73. ReLU works better than tanh / sigmoid in many cases I don’t really understand the reasons (to be honest! -) See [Glorot11] [Glorot10]; written by people who do!
74. IMPROVEMENT Random weight initialisation
75. Previously: rules of thumb often used, e.g. normal distribution with = 0.01 Problems arise when training deep networks with > 8 layers [Simonyan14], [He15]
76. More recent approaches choose initial weights to maintain unit variance (as much as possible) throughout layers Otherwise layers can reduce or magnify magnitudes of signals exponentially
77. Recent approach by He et. Al. [He15]: = 1 Where is the fan-in; the number of incoming connections, and is the gain (for ReLU activation function use = 2)
78. For FC layer: = + = size of / width of ( is a -element vector, is a Q, P matrix)
79. For convolutional layer: = product of kernel width, kernel height and number of channels incoming from previous layer
80. This will ensure that ≈ ()
81. Reducing Over-fitting
82. Over-fitting always a problem in ML Model over-fits when it is very good at matching samples in training set but not those in validation/test
83. Neural networks are very prone to over- fitting
84. Two techniques DropOut Dataset augmentation
85. DropOut [Hinton12] During training, randomly choose units to ‘drop out’ by setting their output to 0, with probability , usually around 0.5 (compensate by multiplying values by 1 1− )
86. During test/predict: Run as normal (no DropOut)
87. Normally applied to later, fully connected layers
88. Dropout OFF Input layer Hidden layer 0 Output layer
89. Dropout ON (1) Input layer Hidden layer 0 Output layer
90. Dropout ON (2) Input layer Hidden layer 0 Output layer
91. Sampling a different subset of the network for each training example Kind of like model averaging with only one model -
92. What effect does it have? (approx. replication of [Hinton12])
93. Dataset: MNIST Digits Network: Single hidden layer, fully connected, 256 units, = 0.4 5000 iterations over training set
94. DropOut OFF Train loss: 0.0003 Validation loss: 0.094 Validation error: 1.9% DropOut ON Train loss: 0.0034 Validation loss: 0.077 Validation error: 1.56%
95. Loss plots Epoch 100 onwards for scale
96. Dataset augmentation Take existing dataset and expand by adding transformed version of existing samples
97. Dataset augmentation for images [Krizhevsky12] Cropping and translation Scaling Rotation Lighting/colour modifications
98. Neural network software
99. Two categories of software: Neural network toolkit (normally faster) Expression compilers
100. Neural network toolkit Most popular is CAFFE (from Berkeley) http://caffe.berkeleyvision.org/
101. Specify network architecture in terms of layers
102. Layers usually described using custom config/language CAFFE uses Google Protocol Buffers for base syntax (YAML/JSON like) and for data (since GPB is binary)
103. CAFFE can be used from: command line MATLAB Python
104. Expression compilers Theano (from University of Montreal) Torch 7 Tensorflow (more recent)
105. Describe network architecture in terms of mathematical expressions Expressions compiled to CUDA code and executed on GPU
106. Theano, Torch 7 and Tensorflow provide automatic symbolic differentiation: Big win; less bugs and less manual work
107. In comparison Advantages Disadvantages Network toolkit (e.g. CAFFE) • CAFFE is fast • Most likely easer to get going • Bindings for MATLAB, Python, command line access • Less flexible; harder to extend (need to learn architecture, manual differentiation) Expression compiler (e.g. Theano) • Extensible; new layer type or cost function: no problem • See what goes on under the hood • Being adventurous is easier! • Slower (Theano) • Debugging can be tricky (compiled expressions are a step away from your code) • Typically only work with one language (e.g. Python for Theano)
108. Resources and tutorials to get you going
109. http://cs.stanford.edu/people/karpathy /convnetjs/ Neural networks running in your web browser Excellent demos that shows how they work and what they can do
110. https://github.com/Newmu/Theano- Tutorials Very simple Python code examples proceeding through logistic regression, fully connected and convolutional models. Shows complete mathematical expressions and training procedures
111. http://deeplearning.net/tutorial/ More Theano tutorials More complete; explains mathematics behind them Code is longer than previous examples
112. CAFFE: http://caffe.berkeleyvision.org/ Plenty of documentation and tutorials
113. Some cool work in the field that might be of interest
114. Visualizing and understanding convolutional networks [Zeiler14] Visualisations of responses of layers to images
115. Visualizing and understanding convolutional networks [Zeiler14] Image taken from [Zeiler14]
116. Visualizing and understanding convolutional networks [Zeiler14] Image taken from [Zeiler14]
117. Deep Neural Networks are Easily Fooled: High Confidence Predictions in Recognizable Images [Nguyen15] Generate images that are unrecognizable to human eyes but are recognized by the network
118. Deep Neural Networks are Easily Fooled: High Confidence Predictions in Recognizable Images [Nguyen15] Image taken from [Nguyen15]
119. Learning to generate chairs with convolutional neural networks [Dosovitskiy15] Network in reverse; orientation, design colour, etc parameters as input, rendered images as output training images
120. Learning to generate chairs with convolutional neural networks [Dosovitskiy15] Image taken from [Dosovitskiy15]
121. A Neural Algorithm of Artistic Style [Gatys15] Take an OxfordNet model [Simonyan14] and extract texture features from one of the convolutional layers, given a target style / painting as input Use gradient descent to iterate photo – not weights – so that its texture features match those of the target image.
122. A Neural Algorithm of Artistic Style [Gatys15] Image taken from [Gatys15]
123. Unsupervised representation Learning with Deep Convolutional Generative Adversarial Nets [Radford 15] Train two networks; one given random parameters to generate an image, another to discriminate between a generated image and one from the training set
124. Generative Adversarial Nets [Radford15] Images of bedrooms generated using neural net Image taken from [Radford15]
125. Generative Adversarial Nets [Radford15] Image taken from [Radford15]
126. Finishing words
127. Deep learning is a fascinating field with lots going on Very flexible, wide range of techniques and applications
128. Deep neural networks have proved to be highly effective* for computer vision, speech recognition and other areas *like with every other shiny new toy, see the small-print!
129. SMALL-PRINT Sufficient training data required (curse of dimensionality) Dataset augmentation advisable
130. SMALL-PRINT Model will only represent training examples; may not probably wont generalise
131. SMALL-PRINT Choose architecture carefully Use a GPU
132. I hope this has proved to be a good introduction to the topic!
133. Thank you!
134. References
135. [Ciresan12] Ciresan, Meier and Schmidhuber; Multi-column deep neural networks for image classification, Computer vision and Pattern Recognition (CVPR), 2012
136. [Dosovitskiy15] Dosovitskiy, Springenberg and Box; Learning to generate chairs with convolutional neural networks, arXiv preprint, 2015
137. [Gatys15] Gatys, Echer, Bethge; A Neural Algorithm of Artistic Style, arXiv: 1508.06576, 2015
138. [Glorot10] Glorot, Bengio; Understanding the difficulty of training deep feedforward neural networks, International conference on artificial intelligence and statistics, 2010
139. [Glorot11] Glorot, Bordes, Bengio; Deep Sparse Rectifier Neural Networks, JMLR 2011
140. [He15] He, Zhang, Ren and Sun; Delving Deep into Rectifiers: Surpassing Human- Level Performance on ImageNet Classification, arXiv 2015
141. [Hinton12] G.E. Hinton, N. Srivastava, A. Krizhevsky, I. Sutskever and R. R. Salakhutdinov; Improving neural networks by preventing co-adaptation of feature detectors. arXiv preprint arXiv:1207.0580, 2012.
142. [Jones87] Jones, J.P.; Palmer, L.A. (1987). "An evaluation of the two-dimensional gabor filter model of simple receptive fields in cat striate cortex". J. Neurophysiol 58 (6): 1233–1258
143. [Krizhevsky12] Krizhevsky, Sutskever and Hinton; ImageNet Classification with Deep Convolutional Neural networks, NIPS 2012
144. [LeCun95] LeCun, Yann et. al.; Comparison of learning algorithms for handwritten digit recognition, International conference on artificial neural networks, 1995
145. [Nguyen15] Nguyen, Yosinski and Clune; Deep Neural Networks are Easily Fooled: High Confidence Predictions for Unrecognizable Images, Computer Vision and Pattern Recognition (CVPR) 2015
146. [Radford15] Radford, Metz, Chintala; Unsupervised Representation Learning with Deep Convolutional Generative Adversarial Networks, arXiv:1511.06434, 2015
147. [Simonyan14] K. Simonyan and Zisserman; Very deep convolutional networks for large-scale image recognition, arXiv:1409.1556, 2014
148. [Zeiler14] Zeiler and Fergus; Visualizing and understanding convolutional networks, Computer Vision - ECCV 2014