OpenTalks.AI - Виктор Лемпицкий, Deep image prior: Why Convolutional Networks Are So Good at Generating Realistic Images?

Transcript

1. Deep image prior: Why Convolutional Networks Are So Good at Generating Realistic Images?
2. Convolutional Networks: designed to recognize images [LeCun 89, 98] [Krizhevsky 12] [Szegedy 14]
3. ConvNets: how they end up being used • Image recognition (all kinds of tasks) • Speech recognition • Catching up with Recurrent nets in Natural Language Processing • Understanding positions in Atari, Go, Chess • Image processing • Image generation
4. Image processing via ConvNets: superresolution [Kim et al. CVPR16]
5. Image generation via ConvNets [Radford et al. ICLR16] [Karras et al. ICLR18]
6. Diversif.AI: ConvNets for your personalized fashion [Joint work with Diana Sungatullina, Egor Zakharov, Roman Trusov]
7. ConvNets: how they end up being used • Image recognition (all kinds of tasks) • Speech recognition • Catching up with Recurrent nets in Natural Language Processing • Understanding positions in Go, Chess…. • Image processing • Image generation
8. Why do generative ConvNets work? It must be because of a lot of learning (and high network capacity)! Right? Indeed: • ConvNets are data hungry, have millions of params • Big GANs are trained for days on tens of thousands of images (or more) • Training process is often tricky with lots of caveats, important to get it right
9. Our claim: it is not just learning! Proof:
10. Deep Image Prior: toy example
11. Image restoration: standard approach • – Recovered image • – Corrupted image (observed) “Classical” MAP approach to inverse problems: Regularization / Prior term • Denoising: • Inpainting: • Super-Res.: • Feature inv.:
12. Image restoration with Deep Image Prior • – Recovered image • – Corrupted image (observed) “Classical” MAP approach to inverse problems: Prior term Convolutional network with parameters Fixed input Consider all images obtained from a random signal z via a convolutional network with a certain architecture : Perform the reconstruction by solving (optionally: use fixed number of iterations): “Find the most likely image that can be generated by a ConvNet from z”
13. Architecture Architecture (U-Net type) Building blocks
14. • – Corrupted image (observed) • Initialize ◦ For example fill it with uniform noise • Solve ◦ With your favorite gradient-based method • Get the solution Deep Image Prior step by step
15. Deep superresolution without training
17. Corrupted Deep image prior Denoising Early stopping after ~2000 iterations needed.
18. Corrupted Deep image prior JPEG Artifacts removal Early stopping after ~2000 iterations needed.
19. Corrupted Deep image prior Image restoration
20. Corrupted Deep image prior Text inpainting
21. Comparison to Shepard networks [Ren et al.NIPS15]
22. Deep inpainting without training
23. Comparing architectures using deep image prior 98% occluded 99.5% occluded
24. Corrupted Deep image prior Flash/No Flash photography Flash photo serves as an input instead of z:
25. Recap • Deep Image Prior: “Swiss knife” for image processing (but a very slow one) • Convolutional network architectures impose natural image priors • Intuition: generative ConvNets capture “Hierarchical self-similarity” of natural images by design • Learning is important, but so is the prior • Outlook: further convergence with computer graphics • Outlook: convergence with recognition networks Thank you!