Introduction to Deep Learning - Cambridge Python User Group

An introduction to deep learning, given at Cambridge Python User Group, 02/Feb/2016

Transcript

1. Deep Learning An Introductory Tutorial G. French Kings College London & University of East Anglia Image montages from http://www.image-net.org
2. ImageNet
3. Image classification dataset
4. ~1,000,000 images ~1,000 classes Ground truths prepared manually through Amazon Mechanical Turk
5. ImageNet Top-5 challenge: You score if ground truth class is one your top 5 predictions
6. ImageNet in 2012 Best approaches used hand-crafted features (SIFT, HOGs, Fisher vectors, etc) + classifier Top-5 error rate: ~25%
7. Then the game changed.
8. Krizhevsky, Sutskever and Hinton; ImageNet Classification with Deep Convolutional Neural networks [Krizhevsky12] Top-5 error rate of ~15%
9. In the last few years, more modern networks have achieved better results still [Simonyan14, He15] Top-5 error rates of ~5-7%
10. I hope this talk will give you an idea of how!
11. What is a neural network?
12. Multiple layers Data propagates through layers Transformed by each layer
13. Neural network Input layer Hidden layer 0 Hidden layer 1 Output layer ⋯ Inputs Outputs ⋯
14. Single layer of a neural network () Input vector Weighted connections Bias Activation function / non-linearity Layer activation
15. = input (M-element vector) = output (N-element vector) = network weights (NxM matrix) = bias (N-element vector) = activation function; tanh / sigmoid / ReLU = ( + )
16. Multiple layers Input vector ( + ) Hidden layer 0 activation ( + ) Hidden layer 1 activation ( + ) Final layer activation (output) ( + ) ⋯
17. In a nutshell: = ( + )
18. Repeat for each layer: 0 = (0 + 0 ) 1 = 1 0 + 1 ⋯ = ( −1 + )
19. As a classifier Input vector Hidden layer 0 activation Final layer activation with softmax (output) ⋯ Image pixels = 0.25 = 0.5 = 0.1 = 0.15 Class probabilities
20. Training a neural network
21. Learn values for parameters; and (for each layer) Use back-propagation
22. Initialise weights randomly (more on this later) Initialise biases to 0
23. Iteratively modify parameters with gradient descent
24. Take example from training set, evaluate network given the training input; =
25. The cost (sometimes called loss), is a measure of the difference between network output and ground truth output
26. Compute the derivative of cost w.r.t. params ( and )
27. Update parameters: 0 ′ = 0 − 0 0 ′ = 0 − 0 γ = learning rate
28. In practice this is done on a mini-batch of examples (e.g. 128) in parallel per pass Compute cost for each example, then average. Compute derivative of average cost w.r.t. params.
29. Using mini-batches works well when using a GPU since computations can be parallelised
30. Cost function
31. Regression Final layer: no activation function / identity. Cost: Sum of squared differences
32. Classification Just like logistic regression
33. Final layer: softmax as activation function ; output vector of class probabilities Cost: negative-log-likelihood / categorical cross-entropy
34. Fully connected neural networks
35. Simplest model Each unit in each layer is connected too all units in previous layer All we have considered so far
36. How well does this perform on image classification?
37. MNIST hand-written digit dataset 28x28 images, 10 classes 60K training examples, 10K validation, 10K test Examples from MNIST
38. Network: 1 hidden layer of 64 units after 300 iterations over training set: 2.85% validation error Hidden layer weights visualised as 28x28 images Input Hidden Output 784 (28x28 images) 64 10
39. Network: 2 hidden layers, both 256 units after 300 iterations over training set: 1.83% validation error Input Hidden 784 (28x28 images) 256 Hidden Output 256 10
40. MNIST is quite a special case Digits nicely centred within image Scaled to approx. same size
41. The fully connected networks so far have a weakness: No translation invariance; learned features are position dependent
42. For more general imagery: requires a training set large enough to see all features in all possible positions… Requires network with enough units to represent this…
43. Convolutional networks
44. Convolution Slide a convolution kernel over an image Multiply image pixels by kernel pixels and sum
45. Convolution Convolutions are often used for feature detection
46. A brief detour…
47. Gabor filters ∗
48. Used for texture classification Bares similarity to low levels of cat visual system [Jones87]
49. Back on track to… Convolutional networks
50. Recap: FC (fully-connected) layer () Input vector Weighted connections Bias Activation function / non-linearity Layer activation
51. Convolutional layer Each unit only connected to units in its neighbourhood
52. Convolutional layer Weights are shared Red weights have same value As do greens… And yellows
53. The values of the weights form a convolution kernel For practical computer vision, more an one kernel must be used to extract a variety of features
54. Convolutional layer Different weight-kernels: Output is vector/image with multiple channels
55. Still = ( + ) As convolution can be expressed as multiplication by weight matrix
56. Note In subsequent layers, each kernel connects to pixels in ALL channels in previous layer
57. Max-pooling ‘layer’ [Ciresan12] Take maximum value from each (, ) pooling region Down-samples image by factor Operates on channels independently
58. These are the models that have been getting excellent ImageNet results
59. How about an example?
60. A Simplified LeNet [LeCun95] for MNIST digits
61. Simplified LeNet for MNIST digits 28 28 24 24 Input Output 1 20 Conv: 20 5x5 kernels Maxpool 2x2 12 8 8 20 50 4 4 50 Conv: 50 5x5 kernels Maxpool 2x2 256 10 Fully connected (flatten and) fully connected 12
62. after 300 iterations over training set: 99.21% validation accuracy Model Error FC64 2.85% FC256--FC256 1.83% 20C5--MP2--50C5--MP2--FC256 0.79%
63. What about the learned kernels? Image taken from paper [Krizhevsky12] Gabor filters
64. Neural networks – recent developments
65. GOOD NEWS Training neural networks is more practical now
66. More processing power Less of a black art
67. IMPROVEMENTS Processing power ReLU activation function Batch normalisation DropOut
68. (most important) IMPROVEMENT Processing power
69. Image processing requires large networks with perhaps millions of parameters Lots of training examples need to train Easily results in billions or even trillions of FLOPS
70. Neural networks are ‘embarrassingly parallelisable’ therefore ideally suited to GPUs Use GPUs for all but the smallest of networks
71. As of now, nVidia is the most popular make of GPU. Cheaper gaming cards perfectly adequate Only use Tesla in production
72. IMPROVEMENT New popular activation function: ReLU - Rectified Linear Unit
73. ReLU - Rectified Linear Unit = max(, 0)
74. ReLU works better than tanh / sigmoid in many cases I don’t really understand the reasons (to be honest! ) See [Glorot11] [Glorot10]; written by people who do!
75. IMPROVEMENT Batch normalisation
76. PROBLEM: Magnitudes of activations can vary considerably, layer to layer If each layer ‘multiplies’ magnitude by some factor, they explode or vanish
77. SOLUTIONS Initally: careful weight initialisation Now: replaced by batch normalisation
78. y = ( + ) Assume: σ = 1 σ depends on distribution of W; normal, uniform, std-dev, etc
79. In the past: initialise W, rules of thumb often used, e.g. normal distribution with = 0.01 Problems arise when training deep networks with > 8 layers [Simonyan14], [He15]
80. Previously: carefully choose distribution of W so that ≈ ()
81. E.g. approach by He et. Al. [He15]: = 1 Where is the fan-in; the number of incoming connections, and is the gain
82. New approach: BATCH NORMALISATION [Ioffe15] Keep distribution of activations sane as part of the network architecture
83. BATCH NORMALISATION For each mini-batch of examples during training Normalise using mean and standard deviation
84. Layer equation becomes: (scale, not needed if is ReLU) and (bias) are learned parameters σ = () μ = () = ( − + )
85. Note: For a fully connected layer, each unit/output should have its own mean and std-dev; aggregate across examples in the mini-batch
86. For a convolutional layer, each channel should have its own mean and std-dev; aggregate across examples in the mini- batch and across image rows and columns
87. During training, keep a running exponential moving average of mean and std-dev During test time, use the averaged mean and std-dev
88. Reducing Over-fitting
89. Over-fitting always a problem in ML Model over-fits when it is very good at matching samples in training set but not those in validation/test
90. Neural networks are very prone to over- fitting
91. Two techniques DropOut (quite a lot of people use batch normalisation instead) Dataset augmentation
92. DropOut [Hinton12] During training, randomly choose units to ‘drop out’ by setting their output to 0, with probability , usually around 0.5 (compensate by multiplying values by 1 1− )
93. During test/predict: Run as normal (no DropOut)
94. Normally applied to later, fully connected layers
95. Dropout OFF Input layer Hidden layer 0 Output layer
96. Dropout ON (1) Input layer Hidden layer 0 Output layer
97. Dropout ON (2) Input layer Hidden layer 0 Output layer
98. Sampling a different subset of the network for each training example Kind of like model averaging with only one model 
99. What effect does it have? (approx. replication of [Hinton12])
100. Dataset: MNIST Digits Network: Single hidden layer, fully connected, 256 units, = 0.4 5000 iterations over training set
101. DropOut OFF Train loss: 0.0003 Validation loss: 0.094 Validation error: 1.9% DropOut ON Train loss: 0.0034 Validation loss: 0.077 Validation error: 1.56%
102. Dataset augmentation Take existing dataset and expand by adding transformed version of existing samples
103. Dataset augmentation for images [Krizhevsky12] Cropping and translation Scaling Rotation Lighting/colour modifications
104. Neural network software
105. Two categories of software: Neural network toolkit (normally faster) Expression compilers
106. Neural network toolkit Most popular is CAFFE (from Berkeley) http://caffe.berkeleyvision.org/
107. Specify network architecture in terms of layers
108. Layers usually described using custom config/language CAFFE uses Google Protocol Buffers for base syntax (YAML/JSON like) and for data (since GPB is binary)
109. CAFFE can be used from: command line MATLAB Python
110. Expression compilers Theano (from University of Montreal) Torch 7 Tensorflow (more recent)
111. Describe network architecture in terms of mathematical expressions Expressions compiled to CUDA code and executed on GPU
112. Theano, Torch 7 and Tensorflow provide automatic symbolic differentiation: Big win; less bugs and less manual work
113. In comparison Advantages Disadvantages Network toolkit (e.g. CAFFE) • CAFFE is fast • Most likely easer to get going • Bindings for MATLAB, Python, command line access • Less flexible; harder to extend (need to learn architecture, manual differentiation) Expression compiler (e.g. Theano) • Extensible; new layer type or cost function: no problem • See what goes on under the hood • Being adventurous is easier! • Slower (Theano) • Debugging can be tricky (compiled expressions are a step away from your code) • Typically only work with one language (e.g. Python for Theano)
114. Resources and tutorials to get you going
115. http://cs.stanford.edu/people/karpathy /convnetjs/ Neural networks running in your web browser Excellent demos that shows how they work and what they can do
116. https://github.com/Newmu/Theano- Tutorials Very simple Python code examples proceeding through logistic regression, fully connected and convolutional models. Shows complete mathematical expressions and training procedures
117. http://deeplearning.net/tutorial/ More Theano tutorials More complete; explains mathematics behind them Code is longer than previous examples
118. CAFFE: http://caffe.berkeleyvision.org/ Plenty of documentation and tutorials
119. Our work
120. CCTV for Fisheries Project involving Dr. M. Fisher, Dr. M. Mackiewicz Funded by Marine Scotland
121. Automatically quantify the amount of fish discarded by fishing trawlers (preferably by species) Process surveillance footage of discard belt
123. STEPS: Segment fish from background Separate fish from one another Classify individual fish (TODO) Measure individual fish to estimate mass (TODO)
124. Approach Use 4-Fields approach [Ganin14]
125. -Fields Use network to transform input image patch into 16-element codeword vector Codeword vector used to look up a target (foreground or edge) patch that most closely matches, in dictionary of words
126. Use 4-Fields to transform input image to foreground map Use 4-Fields to transform input image to edge map Use Watershed algorithm to separate image into regions
130. Videos
131. Has problems with shadows Need more training data Much to do yet!
132. Some cool work in the field that might be of interest
133. Visualizing and understanding convolutional networks [Zeiler14] Visualisations of responses of layers to images
134. Visualizing and understanding convolutional networks [Zeiler14] Image taken from [Zeiler14]
135. Visualizing and understanding convolutional networks [Zeiler14] Image taken from [Zeiler14]
136. Deep Neural Networks are Easily Fooled: High Confidence Predictions in Recognizable Images [Nguyen15] Generate images that are unrecognizable to human eyes but are recognized by the network
137. Deep Neural Networks are Easily Fooled: High Confidence Predictions in Recognizable Images [Nguyen15] Image taken from [Nguyen15]
138. Learning to generate chairs with convolutional neural networks [Dosovitskiy15] Network in reverse; orientation, design colour, etc parameters as input, rendered images as output training images
139. Learning to generate chairs with convolutional neural networks [Dosovitskiy15] Image taken from [Dosovitskiy15]
140. A Neural Algorithm of Artistic Style [Gatys15] Take an OxfordNet model [Simonyan14] and extract texture features from one of the convolutional layers, given a target style / painting as input Use gradient descent to iterate photo – not weights – so that its texture features match those of the target image.
141. A Neural Algorithm of Artistic Style [Gatys15] Image taken from [Gatys15]
142. Unsupervised representation Learning with Deep Convolutional Generative Adversarial Nets [Radford 15] Train two networks; one given random parameters to generate an image, another to discriminate between a generated image and one from the training set
143. Generative Adversarial Nets [Radford15] Images of bedrooms generated using neural net Image taken from [Radford15]
144. Generative Adversarial Nets [Radford15] Image taken from [Radford15]
145. Finishing words
146. Deep learning is a fascinating field with lots going on Very flexible, wide range of techniques and applications
147. Deep neural networks have proved to be highly effective* for computer vision, speech recognition and other areas *like with every other shiny new toy, see the small-print!
148. SMALL-PRINT Sufficient training data required (curse of dimensionality) Dataset augmentation advisable
149. SMALL-PRINT Model will only represent training examples; may not probably wont generalise
150. SMALL-PRINT Choose architecture carefully Use a GPU
151. I hope this has proved to be a good introduction to the topic!
152. Thank you!
153. References
154. [Ciresan12] Ciresan, Meier and Schmidhuber; Multi-column deep neural networks for image classification, Computer vision and Pattern Recognition (CVPR), 2012
155. [Dosovitskiy15] Dosovitskiy, Springenberg and Box; Learning to generate chairs with convolutional neural networks, arXiv preprint, 2015
156. [Ganin14] Ganin, Lempitsky; 4-Fields: Neural Network Nearest Neighbor Fields for Image Transforms, 12th Asian Conference on Computer Vision, 2014
157. [Gatys15] Gatys, Echer, Bethge; A Neural Algorithm of Artistic Style, arXiv: 1508.06576, 2015
158. [Glorot10] Glorot, Bengio; Understanding the difficulty of training deep feedforward neural networks, International conference on artificial intelligence and statistics, 2010
159. [Glorot11] Glorot, Bordes, Bengio; Deep Sparse Rectifier Neural Networks, JMLR 2011
160. [He15] He, Zhang, Ren and Sun; Delving Deep into Rectifiers: Surpassing Human- Level Performance on ImageNet Classification, arXiv 2015
161. [Hinton12] G.E. Hinton, N. Srivastava, A. Krizhevsky, I. Sutskever and R. R. Salakhutdinov; Improving neural networks by preventing co-adaptation of feature detectors. arXiv preprint arXiv:1207.0580, 2012.
162. [Ioffe15] Ioffe, S.; Szegedy C.. (2015). “Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift". ICML 2015, arXiv:1502.03167
163. [Jones87] Jones, J.P.; Palmer, L.A. (1987). "An evaluation of the two-dimensional gabor filter model of simple receptive fields in cat striate cortex". J. Neurophysiol 58 (6): 1233–1258
164. [Krizhevsky12] Krizhevsky, Sutskever and Hinton; ImageNet Classification with Deep Convolutional Neural networks, NIPS 2012
165. [LeCun95] LeCun, Yann et. al.; Comparison of learning algorithms for handwritten digit recognition, International conference on artificial neural networks, 1995
166. [Nguyen15] Nguyen, Yosinski and Clune; Deep Neural Networks are Easily Fooled: High Confidence Predictions for Unrecognizable Images, Computer Vision and Pattern Recognition (CVPR) 2015
167. [Radford15] Radford, Metz, Chintala; Unsupervised Representation Learning with Deep Convolutional Generative Adversarial Networks, arXiv:1511.06434, 2015
168. [Simonyan14] K. Simonyan and Zisserman; Very deep convolutional networks for large-scale image recognition, arXiv:1409.1556, 2014
169. [Zeiler14] Zeiler and Fergus; Visualizing and understanding convolutional networks, Computer Vision - ECCV 2014