Geoffrey Litt
May 07, 2017
690

ENHANCE!! - Upscaling Images with Neural Networks

When characters on a TV show “enhance!” a blurry image, you probably laugh and tell your friends that it’s impossible to do that in real life. But over the past year, deep learning research has actually made this kind of possible! How will you explain this to your friends!?

In this talk, you’ll get an intuitive introduction to generative adversarial networks, a new machine learning technique that’s surprisingly good at upscaling images. You’ll learn how these systems are inspired by human art forgers, and how you can use them to do other things like transform a horse into a zebra, convert a sentence into a photo, and so much more!

May 07, 2017

Transcript

1. ENHANCE!!
@geoffreylitt

2. Entertainment
Accuracy of
portraying
computers

4. Bicubic interpolation

5. Bicubic interpolation

6. DO BETTER

7. NEURAL
NETWORKS!!!

8. Neural network upscaling

9. Neural network training
{ }
, { }
,
Training
data

10. Neural network training
A: Neural
network’s
guess
Per-pixel
subtraction
pixels Total
per-pixel
difference
Minimize this!

11. Neural network training
Loss function:
“Minimize the total per-pixel
difference between the two
images”

12. Neural network
Code: https://github.com/david-gpu/srez

13. Neural network
Bicubic interpolation

14. MORE.
ENHANCE.

15. A: Neural
network’s
guess
Per-pixel
subtraction

16. A: Neural
network’s guess
Really high loss

17. A: Neural
network’s guess
Lower loss

18. A new loss function
???
“Minimize the total per-pixel difference
between your guess and the real
But also make your guess look sharp
and realistic”
Loss =
Total Per-pixel difference +

19. GENERATIVE
NETWORKS

I’ll develop better
counterfeit detection
techniques!
I’ll develop better
counterfeiting
techniques!

21. Generator
GAN structure
{
{
Training example
B: Real
A: Neural
network’s
guess

22. Discriminator
Generator
GAN structure
{
{
Training example
B: Real
Probability
of fake
Goal: differentiate fake vs.
real with 100% accuracy
Goal:
Minimize discriminator
accuracy
A: Neural
network’s
guess

23. Discriminator
Generator
GAN structure
{
{
Training example
B: Real
Probability
of fake
A: Neural
network’s
guess
“Here’s how you could make a
better fake next time”

24. low-res bicubic GAN Real answer

25. GAN-trained generator
Generator

26. GAN results
original
GAN
Per-pixel diff
neural network

27. GOOD
JOB

low-res

29. Unpaired Image-to-Image Translation using Cycle-Consistent Adversarial Networks
Jun-Yan Zhu*, Taesung Park*, Phillip Isola, Alexei A. Efros

30. Image-to-Image Translation with Conditional Adversarial Networks
Phillip Isola, Jun-Yan Zhu, Tinghui Zhou, Alexei A. Efros
https://affinelayer.com/pixsrv/

31. Image-to-Image Translation with Conditional Adversarial Networks
Phillip Isola, Jun-Yan Zhu, Tinghui Zhou, Alexei A. Efros
https://affinelayer.com/pixsrv/

32. StackGAN: Text to Photo-realistic Image Synthesis with