Current_Methods_and_Models_of_Visual_Prediction.pdf

A short overview of techniques and models for visual prediction tasks.

Presented April 5th, 2017

Transcript

1. Current Methods and Models for Visual Prediction Evan Cushing University of British Columbia, Computer Science, Science Co-op duunec@gmail.com Oord et al., PixelCNN. Fig 9: Left: source image. Right: new portraits generated from high-level latent representation
2. What is “visual prediction”? • • Images generated from a predicted distribution • Generative models (VAE, GAN, Pixel) Left: van den Oord et al., PixelCNN 2016 Right: Lotter et al., 2016 • Images predicted given previous frame in sequence • Recurrent models (R-CNN, ConvLSTM) Generated prediction Sequential prediction
3. Autoencoder Neural Network Encoder : X→H Decoder: H→X’ • The encoder maps X to feature vector z • The decoder maps z to X’ • Optimize squared difference (X-X’) AE diagram: https://en.wikipedia.org/wiki/Autoencoder MNISThttps://blog.keras.io/building-autoencoders-in-keras.h
4. Generative Adversarial Networks (GAN) • GANs consist of two networks trained together as a min-max game ◦ The Discriminator, D, decides whether a given image is a generated image or true data ◦ The Generator, G, attempts to match the distribution of the true data to produce images that can fool D. Top: https://wiki.tum.de/pages/viewpage.action?pageId=23 562510#GenerativeAdversarialNetworks(GANs)-We Bottom: https://blog.openai.com/generative-models/
5. GAN: An Adversarial Game
6. GAN: Competing Gradients Left: Goodfellow et al., 2014.
7. DCGAN (https://arxiv.org/abs/1511.06434) • Deep-Convolutional GAN uses series of fractional-strided convolutions to generate an image and normal convolutions to discriminate the image Top: https://cdn-images-1.medium.com/max/1000/1* 39Nnni_nhPDaLu9AnTLoWw.png
8. Vector Arithmetic for Visual Concepts in Representation Space Radford et al., 2015.
9. Fun with DCGAN, https://github.com/mattya/chainer-DCGAN 30分後 2時間後 1日後 Image plots: http://qiita.com/mattya/items/e5bfe5e04b9d2f0bbd47
10. Recent Developments with GANs • WGAN (https://arxiv.org/abs/1701.07875) • LSGAN (https://arxiv.org/abs/1611.04076v2) • InfoGAN (https://arxiv.org/abs/1606.03657) • IntrospectiveAN (https://arxiv.org/abs/1609.07093) Curated collection of GAN papers: https://github.com/zhangqianhui/AdversarialNetsPapers
11. Pixel Networks: PixelRNN • Model estimates a distribution over natural images • Computes likelihood of images as a product of conditional probabilities Above: van den Oord et al., PixelRNN 2016.
12. Pixel Networks: PixelRNN • Layers in PixelRNN use LSTM units in their state and adopt a convolution to compute states along spatial dimensions • The network scans the image one row at a time and one pixel at a time for each row Center: van den Oord et al., PixelRNN 2016. p(x) is modeled as a discrete distribution, with every conditional being a multinomial modeled with a softmax layer
13. Pixel Networks: PixelCNN • Joint distribution of pixels in an image still computed as a product of conditionals • But layers are now composed of two gated convolutional stacks that capture context instead of LSTM units • Convolutions are masked to only capture relevant field of context
14. Conditional PixelCNN generations from ImageNet
15. Linear Interpolation on Embedding Space, PixelCNN
16. Pixel Networks, What’s the Big Deal? • High quality generated samples from directly optimizing log likelihood • State-of-the-art performance on various benchmarks (binarized MNIST, CIFAR-10), as well as a new benchmark for generative image modeling on ImageNet • PixelCNN can be used as a powerful decoder on latent space learned by other models
17. Predicting the Future with PredNet • Deep convolutional recurrent neural network trained for next-frame video prediction • Each layer in the network makes local predictions and only forwards deviations, the error E, from predictions to the next layer Left: Lotter et al., 2016.
18. Prednet Algorithm Left, Right: Lotter et al., 2016. Updating of states occurs through two passes: 1. Top-down pass to compute recurrent states 2. Forward pass to calculate predictions, errors, and higher level targets
19. PredNet, t+1 predictions
20. PredNet, t+n extrapolated predictions
21. Prednet, t+n extrapolated predictions
22. Ambiguity in Motion Dynamics Walker et al., 2016.
23. Motion Dynamics Video https://youtu.be/3PYgrmoG-OA
24. Why Bother with Visual Prediction? • Cutting-edge development • Unsupervised learning is the future • It’s fun working with generated samples • Many applications: ◦ Generated images, deblurring/denoising, text-to-image, domain-synthesis, future frame prediction, latent space modeling
25. Works Cited • Ian J. Goodfellow, et al. “Generative Adversarial Networks” 2014: https://arxiv.org/abs/1406.2661 • Alec Radford, et al., “Unsupervised Representation Learning with Deep Convolutional Generative Adversarial Networks” 2015: https://arxiv.org/abs/1511.06434 • Aaron van den Oord, et al. “Pixel Recurrent Neural Networks” 2016: https://arxiv.org/abs/1601.06759 • Aaron van den Oord, et al. “Conditional Image Generation with PixelCNN Decoders” 2016: https://arxiv.org/abs/1606.05328 • William Lotter, et al. “Deep Predictive Coding Networks for Video Prediction and Unsupervised Learning” 2016: https://arxiv.org/abs/1605.08104 • Jacob Walker, et al. “An Uncertain Future: Forecasting from Static Images using Variational Autoencoders” 2016: https://arxiv.org/abs/1606.07873
26. Suggested Reading Starting point: http://neuralnetworksanddeeplearning.com/chap1.html (what’s a neural network?) http://colah.github.io/posts/2015-08-Understanding-LSTMs/ (LSTM flow) http://cs231n.stanford.edu/syllabus.html (your silver bullet) Topics: https://blog.openai.com/generative-models/ (good overview of generative models) http://web.mit.edu/vondrick/tinyvideo/ (excellent reading list in “Related Works”) https://github.com/zhangqianhui/AdversarialNetsPapers (GANs GANs GANs!)
27. Thanks for Listening!