Deep Learning - Advanced Techniques Tutorial - PyData Amsterdam 2017

Slides from my PyData Amsterdam 2017 tutorial on deep learning.

Transcript

1. Deep Learning Tutorial Advanced Techniques PyData Amsterdam 2017 G. French – University of East Anglia Image montages from http://www.image-net.org
2. Focus
3. Image processing Using Theano1 and Lasagne2 1http://deeplearning.net/software/theano/ 2 https://github.com/Lasagne/Lasagne
4. What we’ll cover
5. Theano What it is and how it works Review: Multi-layer perceptron The basic model Convolutional networks Neural networks for computer vision
6. Lasagne and VGG-16 Explain Lasagne and use it with a convolutional network trained by the VGG group at Oxford University Transfer learning Re-using pre-trained networks Deep learning tricks of the trade tips to save you some time
7. Tutorial materials
8. Github Repo: https://github.com/Britefury/deep-learning-tutorial-pydata The notebooks are viewable on Github
9. Intro to Theano and Lasagne slides: https://speakerdeck.com/britefury https://speakerdeck.com/britefury/intro-to-theano-and-lasagne-for-deep-learning
10. Amazon AMI (Use GPU machine) AMI ID: ami-5f789e32 AMI Name: PyData London 2016 deep learning adv tutorial - Ubuntu-14.04 Anaconda2-4.0.0 Cuda-7.5 cuDNN-5 Theano-0.8 Lasagne Fuel
11. Theano
12. Neural network software exists on a spectrum Higher level - Specify network in terms of layers Flexible and powerful - Specify network in terms of mathematical expressions Neural network toolkits Expression compilers API style
13. Neural network software exists on a spectrum Less to debug so easier Depends on toolkit - (can end up with confusing errors in compiled code e.g. Theano) Neural network toolkits Expression compilers Debugging
14. Neural network software exists on a spectrum CAFFE Theano Tensorflow Torch Neural network toolkits Expression compilers Examples
15. Theano An expression compiler
16. Write numpy style expressions Compiles to either C (CPU) or CUDA (nVidia GPU)
17. Notebook: Theano basics Expressions Modify shared variables Variables and functions Gradient and updates
18. There is much more to Theano For more information: http://deeplearning.net/
19. Review: MLP (multi-layer perceptron)
20. = input (M-element vector) = output (N-element vector) = weights parameter (NxM matrix) = bias parameter (N-element vector) = activation function; normally ReLU but can be tanh or sigmoid = ( + )
21. In a nutshell: = ( + )
22. Repeat for each layer Input vector ( + ) Hidden layer 0 activation ( + ) Hidden layer 1 activation ( + ) Final layer activation (output) ( + ) ⋯
23. To train the network: Compute the derivative of cost w.r.t. parameters ( and ) (More on the cost/loss later)
24. Update parameters: + , = + − /0 /12 + , = + − /0 /32 γ = learning rate
25. Theano takes care of the differentiation for you!
26. MNIST Multi-layer Perceptron
27. (Obligatory) MNIST example: 2 hidden layers, both 256 units after 300 iterations over training set: 1.83% validation error Input Hidden 784 (28x28 images) 256 Hidden Output 256 10
28. MNIST is quite a special case Digits nicely centred within image Scaled to approx. same size
29. The fully connected networks so far have a weakness: No translation invariance; learned features are position dependent
30. For more general imagery: requires a training set large enough to see all features in all possible positions… Requires network with enough units to represent this…
31. Non-linearities and loss functions
32. The final non-linearity and the corresponding loss function is important Their choice depends on the desired output of the network
33. Classification Final non-linearity: softmax Softmax produces predicted probability vector from logits : = = => ∑ => @ AB+
34. Classification Loss: negative log probability (categorical cross-entropy); = prediction, = true value = − E ln A A
35. Lets dig a little deeper and do an experiment.
36. Lets take a neural network: Input vector ( + ) Hidden layer 0 activation ( + ) Hidden layer 1 activation ( + ) Final layer activation (output) ( + ) ⋯
37. Remove all but the final layer: Final layer activation (output) ( + ) Final layer
38. Drive its activation function directly (no matrix multiplication) Final layer activation (output) () Final layer We are going to ‘simulate’ the network up to that point by providing values for the logits directly
39. Create logit values: 0.0 -5.0 0.0 -4.0 0.0 -3.0 0.0 -2.0 0.0 -1.0 0.0 0.0 0.0 1.0 0.0 2.0 0.0 3.0 0.0 4.0 0.0 5.0 keep logit for class 0 constant vary logit for class 1
40. Lets add the predicted probabilities generated by softmax: 0.0 -5.0 0.9933072 0.0066929 0.0 -4.0 0.9820138 0.0179862 0.0 -3.0 0.9525741 0.0474259 0.0 -2.0 0.8807970 0.1192029 0.0 -1.0 0.7310586 0.2689414 0.0 0.0 0.5000000 0.5000000 0.0 1.0 0.2689414 0.7310586 0.0 2.0 0.1192029 0.8807971 0.0 3.0 0.0474259 0.9525741 0.0 4.0 0.0179862 0.9820138 0.0 5.0 0.0066929 0.9933072 => ∑ => @ AB+
41. Plot the predicted probability N from softmax and the true probability N (assume a value of 1)
42. Add negative log-loss: Note: loss is high when N is negative, tends to 0 when it is positive
43. Add gradient of negative log-loss:
44. Learning occurs via gradient descent Note: gradient in range [-1, 0], negative when N is negative, when N is positive the gradient is close to 0 (correct answer, not much learning to do)
45. Probability regression Use for predicting single value in [0, 1] range Can use for binary classification Could use to generate mask for an image
46. Probability regression Final non-linearity: sigmoid Sigmoid outputs values in range [0, 1]: = = 1 1 + ST
47. Probability regression Loss: binary cross-entropy; = prediction, = true value = − E ln A A + ln 1 − A 1 − A
48. Experiment Drive sigmoid directly Sigmoid takes a scalar per sample, rather than a vector
49. Probability regression plot Just like classification with softmax & NLL J (maths are similar)
50. In general When network gives correct answer: Gradient of the loss will be near 0 (no more learning)
51. In general When network gives incorrect answer: Gradient of loss will be of magnitude 1 pushing the network to learn
52. A gradient of 1 The gradient of the final non-linearity + loss function having a magnitude of 1 is worth consideration
53. A gradient of 1 They won’t scale up or down the gradient that is back-propagated throughout the earlier parts of the network
54. A gradient of 1 Keeping the gradient magnitudes sane is important (*) Hence the success of weight initialisation schemes and batch normalisation * Particularly for GANs
55. Regression with squared error loss Use for predicting real values NO final non-linearity (linear)
56. Regression with squared error loss Loss: squared error; U = prediction, = true value = − E − U V
57. Regression with squared error loss plot The gradient can have a large magnitude when U and differ greatly
58. Regression with Huber loss Uses squared error when difference between prediction and target is small, absolute error otherwise Used in [Girshick15]
59. Regression with Huber loss NO final non-linearity (linear)
60. Regression with Huber loss Loss: Huber loss; U = prediction, = true value ℎ , U = 1 2 U − V U − ≤ 1 U − − 1 2 ℎ
61. Regression with Huber loss plot Keeps the gradient magnitude ≤ 1
62. In summary: Final non-linearity and loss function depend on: What you want your network to generate Effects on gradients can be worth considering
63. Convolutional networks
64. Convolution Slide a convolution kernel over an image Multiply image pixels by kernel pixels and sum
65. Convolution Convolutions are often used for feature detection
66. A brief detour…
67. Gabor filters ∗
68. Back on track to… Convolutional networks
69. Recap: FC (fully-connected) layer () Input vector Weighted connections Bias Activation function (non-linearity) Layer activation
70. Convolutional layer Each unit only connected to units in its neighbourhood
71. Convolutional layer Weights are shared Red weights have same value As do greens… And yellows
72. The values of the weights form a convolution kernel For practical computer vision, more an one kernel must be used to extract a variety of features
73. Convolutional layer Different weight-kernels: Output is image with multiple channels
74. Still = ( + ) As convolution can be expressed as multiplication by weight matrix
75. Note In subsequent layers, each kernel connects to pixels in ALL channels in previous layer
76. Another way of looking at it: A single kernel of an e.g. 5x5 convolutional layer is a bit like…
77. fully-connected layer with 5x5 input image repeated across the whole image a new ‘fully-connected layer’ for each filter
78. Down-sampling Max-pooling or Striding
79. Down-sampling: max-pooling ‘layer’ [Ciresan12] Take maximum value from each 2 x 2 pooling region ( x ) in the general case Down-samples image by factor Operates on channels independently
80. Down-sampling: striding Only retain 1 pixel in every ; skip the rest Often fast; is built into the convolution operation of many neural network libraries
81. Example: A Simplified LeNet [LeCun95] for MNIST digits
82. Simplified LeNet for MNIST digits 28 28 24 24 Input Output 1 20 Conv: 20 5x5 kernels Maxpool 2x2 12 8 8 20 50 4 4 50 Conv: 50 5x5 kernels Maxpool 2x2 256 10 Fully connected (flatten and) fully connected 12
83. after 300 iterations over training set: 99.21% validation accuracy Model Error FC64 2.85% FC256--FC256 1.83% 20C5--MP2--50C5--MP2--FC256 0.79%
84. Lasagne and VGG-16
85. Lasagne is a neural network library built on Theano
86. Provides API for: constructing layers of a network getting Theano expressions representing output, loss, etc.
87. Lasagne is quite a thin layer on top of Theano, so understanding Theano is helpful On the plus side, implementing custom layers, loss functions, etc is quite doable.
88. An aside note about other libraries Definitely check out Keras http://keras.io Works on both Theano and Tensorflow
89. An aside note about other libraries Keras API can be simpler to get to grips with than Lasagne Lots of cool examples
90. Notebook: Lasagne basics Build network: modified LeNet for MNIST Train the network
91. Using a pre-trained VGG-16 Conv- net
92. Use VGG-16; the 16-layer model 1000-class image classifier, trained on ImageNet
93. VGG models are simple but effective Consist of: 3x3 convolutions 2x2 max pooling fully connected
94. Notation: 64C3 convolutional layer with 64 3x3 filters MP2 max-pooling, 2x2 # Layer Input: 3 x 224 x 224 (RGB image, zero-mean) 1 64C3 2 64C3 MP2 3 128C3 4 128C3 MP2 5 256C3 6 256C3 7 256C3 MP2
95. Notation: FC4096 fully-connected layer 4096 channels drop 50% With 50% drop-out during training # Layer 8 512C3 9 512C3 10 512C3 MP2 11 512C3 12 512C3 13 512C3 MP2 14 FC4096 (drop 50%) 15 FC4096 (drop 50%) 16 FC1000 soft-max
96. # Layer Input: 3 x 224 x 224 (RGB image, zero-mean) 1 64C3 2 64C3 MP2 3 128C3 4 128C3 MP2 5 256C3 6 256C3 7 256C3 MP2 # Layer 8 512C3 9 512C3 10 512C3 MP2 11 512C3 12 512C3 13 512C3 MP2 14 FC4096 (drop 50%) 15 FC4096 (drop 50%) 16 FC1000 soft-max
97. These kinds of architectures tend to work well: Small convolution kernels (3x3) Interspersed with max-pooling
98. Good first start when choosing a network architecture
99. Exercise / Demo Classifying an image with VGG-16
100. What about using VGG-16 to find a peacock in a photo?
101. Can extract square patches from image in sliding window fashion and classify:
102. Exercise / Demo Finding a peacock with VGG-16: part 1
103. Inefficient
104. Using convolutions to your advantage
105. Adjacent windows share majority of their pixels
106. For lower levels, this involves repeating many of the same computations, getting the same result ∗
107. If we could apply the first convolutional layer across the whole image rather than many 224x224 blocks we could re- use those computations….
108. Then we could also do this for the rest of the convolutional layers further down…
109. In fact we can use the whole network in a convolutional fashion; we just need to convert the fully-connected layers to convolutional layers.
110. Exercise / Demo Finding a peacock with VGG-16: part 2
111. This is a trick used when doing image segmentation, when we want to determine which parts of an image belong to which class At both training time and prediction time
112. Transfer learning (network re-use)
113. Training a neural network is notoriously data-hungry Preparing training data with ground truths is expensive and time consuming
114. What if we don’t have enough training data to get good results?
115. The ImageNet dataset is huge; millions of images with ground truths What if we could somehow use it to help us with a different task?
116. Good news: we can!
117. Transfer learning Re-use part (often most) of a pre-trained network for a new task
118. Example; can re-use part of VGG-16 net for: Classifying images with classes that weren’t part of the original ImageNet dataset
119. Example; can re-use part of VGG-16 net for: Localisation (find location of object in image) Segmentation (find exact boundary around object in image)
120. Transfer learning: how to Take existing network such as VGG-16
121. # Layer Input: 3 x 224 x 224 (RGB image, zero-mean) 1 64C3 2 64C3 MP2 3 128C3 4 128C3 MP2 5 256C3 6 256C3 7 256C3 MP2 # Layer 8 512C3 9 512C3 10 512C3 MP2 11 512C3 12 512C3 13 512C3 MP2 14 FC4096 (drop 50%) 15 FC4096 (drop 50%) 16 FC1000 soft-max
122. Remove last layers e.g. the fully- connected ones (just 14,15,16; those in the left box are hidden here for brevity!) # Layer 8 512C3 9 512C3 10 512C3 MP2 11 512C3 12 512C3 13 512C3 MP2
123. Build new randomly initialise layers to replace them (the number of layers created their size is only for illustration here) # Layer 8 512C3 9 512C3 10 512C3 MP2 11 512C3 12 512C3 13 512C3 MP2 FC1024 (drop 50%) FC21 soft-max
124. Training when using transfer learning Two approaches: Train new layers only Fine-tuning
125. Lets start with some code to get the pre- trained and new layer parameters separately
126. # Get all parameters in network; `get_all_params` works backward # from the final layer through the network all_params = lasagne.layers.get_all_params(final_layer, trainable=True) # Get parameters from pre-trained layers; give the top pre-trained layer pretrained_params = lasagne.layers.get_all_params( vgg16.network[‘pool5’], trainable=True) new_params = [p for p in all_params if p not in pretrained_params]
127. Transfer learning: train new layers Train the network with your training data, only learning parameters for the new layers
128. # Update new layers only new_updates = lasagne.updates.adam( training_loss, new_params, learning_rate=lr)
129. Transfer learning: fine-tuning Learn parameters for new layers Fine-tuning: learn parameters for pre- trained layers using a lower learning rate; normally 1/10th
130. # Update new layers with standard learning rate new_updates = lasagne.updates.adam( training_loss, new_params, learning_rate=lr) # Update new layers with standard learning rate pretrained_updates = lasagne.updates.adam( training_loss, pretrained_params, learning_rate=lr * 0.1) # Combine updates updates = new_updates.copy() updates.update(pretrained_updates)
131. Result Nice shiny network with good performance that was trained with much less of our training data
132. Exercise / Demo Solving Dogs vs Cats with Transfer Learning
133. Our work Quantifying fish discards on-board fishing trawlers We use VGG-16 based image segmentation
135. Foreground segmentation
137. Contour detection
139. Deep learning tricks of the trade
140. Early stopping
141. Early stopping Can get better accuracy Allows you to terminate training earlier, saving time
142. Split data into: training set validation set test set
143. During training, regularly – e.g. after each epoch – evaluate network performance on validation set
144. Validation score does not decrease monotonically, so the final score is not necessarily the best Dogs vs cats, with transfer learning and data augmentation
145. Each time validation score improves: Save network state Restore saved state once training is finished
146. Sometimes validation score can start getting gradually worse as the network overfits Once you reach this point, there’s no point training any more
147. Fixed patience Stop training if no improvement detected after a fixed number of epochs e.g. 50
148. Geometric patience Set the patience (index of epoch that you are prepared to wait until) to be the number of epochs elapsed so far times some multiple e.g. 2
149. Geometric patience An example can be seen in the Logistic Regression and MLP Theano tutorials: http://deeplearning.net/tutorial/mlp.html
150. Choosing: mini-batch size
151. Small mini-batches Maybe around ~8 Good but slower training Small mini-batch results in regularization (due to noise), reaching lower error rates in the end [Goodfellow16]. When using very small mini- batches, need to compensate with lower learning rate and more epochs. Slow due to low parallelism Does not use all cores of GPU Low memory usage Less neuron activations kept in RAM
152. Large mini-batches 1000s Ineffective training Won’t reach the same error rate as with smaller batches and may not learn at all. Can be fast due to high parallelism Uses GPU parallelism (there are limits; gains only achievable if there are unused CUDA cores) High memory usage Lots of neuron activations kept around; can run out of RAM on large networks
153. Happy medium (where you want to be) Maybe around 64-256, lots of experiments use ~100 Effective training Learns reasonably quickly – in terms of improvement per epoch – and reaches acceptable error rate or loss Medium performance Acceptable in many cases Medium memory usage Fine for modest sized networks
154. ~100 seems to work well; gets good results
155. Increasing mini-batch size will improve performance up to the point where all GPU units are in use Increasing it further will not improve performance; it will reduce accuracy
156. Caveat When working in a convolutional fashion - like the example of using VGG- net to find the peacock – or when doing image segmentation
157. In such cases, pushing large patches of an image through as a single batch along with a correspondingly large output patch re-uses data due to convolutions and results in substantial savings
158. My experience: Use patches that are as large as possible Although it’s a tricky balance with accuracy of the final result
159. Batch normalization
160. Batch normalization [Ioffe15] is recommended in most cases Speeds up training Loss and error drop faster per-epoch
161. Although epochs take longer (around 2x in my experience) Can (ultimately) reach lower error rates Lets you build deeper networks
162. Standardise activations (zero-mean, unit variance) per-channel between network layers Solves problems caused by exponential growth or shrinkage of layer activations
163. + = 1 N = 2 A`N = 2 A Assume that a layer – grey square – produces activations whose std-dev are twice that of the input:
164. = 1 = 2@ @ = 2@ + When layers are stacked together: ⋯
165. The magnitude of activations and therefore gradients either explode or vanish (if the layers reduce the magnitude of activations rather than magnify them)
166. Can be partially addressed with careful weight initialization [He15]. Batch normalization between layers keeps things sane; can train networks with hundreds of layers [He15b].
167. After convolutional, fully-connected or network-in-network* layers, before the non-linearity (*) kind of like a 1x1 convolutional layer [Lin13]
168. Lasagne batch normalization inserts itself into a layer before the non- linearity, so its nice and easy to use: l = lasagne.layers.batch_norm(l)
169. Data standardisation
170. Standardise your data Ensure zero-mean and unit standard deviation
171. Standardise input data In case of regression, standardise output data too (don’t forget to invert the standardisation of network predictions!)
172. Autoencoder; image reconstruction (regression), PCA whitening
173. Autoencoder; image reconstruction (regression), no standardisation
174. Still a good idea to do this, even when using batch normalization
175. Autoencoder; edge map reconstruction (regression), standardisation
176. Autoencoder; edge map reconstruction (regression), no standardisation
177. Standardisation Extract samples (pixels in the case of images) into an array Compute distribution and standardise
178. Either: Zero the mean and scale std-dev to 1, per channel (RGB for images) , = −
179. CIFAR10 RGB distribution
180. CIFAR10 RGB – standardised
181. Or better still: Use PCA whitening (retain all channels – we don’t want to reduce dimensionality)
182. CIFAR10 RGB – with principal components
183. CIFAR10 RGB - principal components aligned with standard basis
184. CIFAR10 RGB – PCA whitened
185. PCA whitening From Scikit-learn use PCA or IncrementalPCA
186. ZCA whitening is very similar Notice that PCA whitening rotated the RGB distribution ZCA whitening doesn’t; it only scales along the axes found by PCA
187. Could batch normalisation make standardisation unnecessary?
188. The previous fish based examples all used batch normalisation and still benefited from data standardisation, so no.
189. Data augmentation
190. Reduce over-fitting by artificially enlarging training set Modify existing training samples to make new ones
191. For images: Transformations: move, scale, rotate, reflect, etc. Consider the domain: horizontally flipping characters for character recognition would be a bad idea
192. For images: Lighting: Compute principal components of RGB pixel values Add normally distributed random multiples of 10% of the std-dev in each principal component
193. See [Krizhevsky12]; their paper discusses their techniques, that have since been used by many otehrs
194. Exercise / Demo Solving Dogs vs Cats with Transfer Learning and Data Augmentation
195. When training goes wrong and what to look for
196. Loss becomes NaN
197. Classification error rate equivalent of random guess (its not learning)
198. Learns to predict constant value; optimises constant value for best loss A constant value is a local minimum that the network won’t get out of (neural networks ‘cheat’ like crazy!)
199. Debugging your network
200. Neural networks (most) often DON’T learn what you want or expect them to
201. Local minima will be the bane of your existence
202. So, what has your network learnt? This is often a good question.
203. Saliency maps Determine which parts of an image the network is using to make its prediction Tells you what the network is ‘looking at’
204. Two approaches
205. 1. Region-level saliency Blank out different regions of the image and compute the difference in prediction
206. Exercise / Demo Block-level image saliency
207. 2. Pixel-level saliency Compute the gradient of the prediction of a specific class w.r.t. the image pixels
208. Exercise / Demo Pixel-level image saliency
209. Designing a computer vision pipeline
210. Simple problems may be solved with just a neural network
211. Not sufficient for more complex problems
212. Theoretically possible to use a single network, with enough training data (where enough is an impractical amount)
213. For more complex problems, the problem should be broken down
214. Example Identifying right whales, by Felix Lau 2nd place in Kaggle competition http://felixlaumon.github.io/2015/01/0 8/kaggle-right-whale.html
215. Identifying right whales, by Felix Lau The first naïve solution – training a classifier to identify individuals – did not work well
216. Region-based saliency map revealed that the network had ‘locked on’ to features in the ocean shape rather than the whales
217. Lau’s solution: Train a localiser to locate the whale in the image
218. Lau’s solution: Train a keypoint finder to locate two keypoints on the whale’s head to identify its orientation
219. Lau’s solution: Train classifier on oriented and cropped whale head images
220. Just for fun
221. Deep Dreams
222. When training a network, we use gradient descent to iteratively modify weights given images and ground truths
223. We just as easily use gradient descent to modify an image given weights
224. Deep Dreams: Take an image to hallucinate from Choose a layer, e.g. ‘pool4’ of VGG-19; choice depends on scale and level of features desired
225. Deep Dreams: Compute gradient of V-norm of layer w.r.t. image Use gradient ascent to increase V-norm
226. Exercise / Demo Deep Dreams
227. For other cool examples Look at the Keras library, they have loads http://keras.io
228. Some cool work in the field that might be of interest
229. Visualizing and understanding convolutional networks [Zeiler14] Visualisations of responses of layers to images
230. Visualizing and understanding convolutional networks [Zeiler14] Image taken from [Zeiler14]
231. Visualizing and understanding convolutional networks [Zeiler14] Image taken from [Zeiler14]
232. Plug & Play Generative Networks: Conditional Iterative Generation of Images in Latent Space [Nguyen17] Image taken from http://www.evolvingai.org/ppgn
233. Deep Neural Networks are Easily Fooled: High Confidence Predictions in Recognizable Images [Nguyen15] Generate images that are unrecognizable to human eyes but are recognized by the network
234. Deep Neural Networks are Easily Fooled: High Confidence Predictions in Recognizable Images [Nguyen15] Image taken from [Nguyen15]
235. Learning to generate chairs with convolutional neural networks [Dosovitskiy15] Network in reverse; orientation, design colour, etc parameters as input, rendered images as output training images
236. Learning to generate chairs with convolutional neural networks [Dosovitskiy15] Image taken from [Dosovitskiy15]
237. Unsupervised representation Learning with Deep Convolutional Generative Adversarial Nets [Radford 15] Train two networks; one given random parameters to generate an image, another to discriminate between a generated image and one from the training set
238. Generative Adversarial Nets [Radford15] Images of bedrooms generated using neural net Image taken from [Radford15]
239. Generative Adversarial Nets [Radford15] Image taken from [Radford15]
240. There has been a lot of work on GANs lately; mainly focused on improving their output quality
241. BEGAN:BoundaryEquilibriumGenerative AdversarialNetworks [Berthelot17] Image taken from [Berthelot17]
242. A Neural Algorithm of Artistic Style [Gatys15] Take an OxfordNet model [Simonyan14] and extract texture features from one of the convolutional layers, given a target style / painting as input Use gradient descent to iterate photo – not weights – so that its texture features match those of the target image.
243. A Neural Algorithm of Artistic Style [Gatys15] Image taken from [Gatys15]
244. Much work on style transfer has focused on either improving quality or performance Original algorithm used gradient descent; like Deep Dream, so its quite slow
245. Deep Photo Style Transfer [Luan17] (also iterative, but just awesome) Image taken from https://github.com/luanfujun/deep-photo-styletransfer
246. Hope you’ve found it helpful!
247. Thank you!
248. References
249. [Berthelot17] Berthelot D., Schumm T., Metz L.; “BEGAN: Boundary Equilibrium Generative Adversarial Networks”, arXiv 1703.10717 (2017).
250. [Girshick15] Girshick, Ross; “Fast R- CNN”, Proceedings of the IEEE International Conference on Computer Vision, 2015
251. [He15a] He, Zhang, Ren and Sun; “Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classification”, arXiv 2015
252. [He15b] He, Kaiming, et al. "Deep Residual Learning for Image Recognition." arXiv preprint arXiv:1512.03385 (2015).
253. [Hinton12] G.E. Hinton, N. Srivastava, A. Krizhevsky, I. Sutskever and R. R. Salakhutdinov; “Improving neural networks by preventing co-adaptation of feature detectors.” arXiv preprint arXiv:1207.0580, 2012.
254. [Ioffe15] Ioffe, S.; Szegedy C.. (2015). “Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift". ICML 2015, arXiv:1502.03167
255. [Jones87] Jones, J.P.; Palmer, L.A. (1987). "An evaluation of the two-dimensional gabor filter model of simple receptive fields in cat striate cortex". J. Neurophysiol 58 (6): 1233–1258
256. [Lin13] Lin, Min, Qiang Chen, and Shuicheng Yan. "Network in network." arXiv preprint arXiv:1312.4400 (2013).
257. [Luan17] Luan F., Paris S., Shechtman E., Bala K. “Deep Photo Style Transfer" arXiv:1703.07511 (2017).
258. [Krizhevsky12] Krizhevsky, A., Sutskever, I. and Hinton, G. E. "ImageNet Classification with Deep Convolutional Neural Networks." NIPS 2012.
259. [Nesterov83] Nesterov, Y. A method of solving a convex programming problem with convergence rate O(1/sqr(k)). Soviet Mathematics Doklady, 27:372–376 (1983).
260. [Nguyen17] Nguyen A., Yosinski J., Bengio Y., Dosovitskiy A., Clune J. “Plug & Play Generative Networks: Conditional Iterative Generation of Images in Latent Space”. CVPR 2017.
261. [Sutskever13] Sutskever, Ilya, et al. On the importance of initialization and momentum in deep learning. Proceedings of the 30th international conference on machine learning (ICML-13). 2013.
262. [Simonyan14] K. Simonyan and Zisserman; “Very deep convolutional networks for large-scale image recognition”, arXiv:1409.1556, 2014
263. [Wang14] Wang, Dan, and Yi Shang. "A new active labeling method for deep learning."Neural Networks (IJCNN), 2014 International Joint Conference on. IEEE, 2014.