Deep Learning Image Manipulation

Slide 1

Slide 1 text

%FFQ-FBSOJOH *NBHF.BOJQVMBUJPO BOJMMVTUSBUFEHVJEF .-,JUDIFO

Slide 2

Slide 2 text

"CPVUNF w -FT[FL3ZCJDLJ w HJUIVC!MVOBSEPH w CPSOJO1PMBOE w .-3FTFBSDIFSBU$PPLQBE w *MJLFOBUUP

Slide 3

Slide 3 text

DBSFFST!DPPLQBEDPN 8BOUUPXPSLXJUIVT

Slide 4

Slide 4 text

$POWPMVUJPOBM "SJUINFUJD OCIKE

Slide 5

Slide 5 text

*NBHFTUPGFBUVSFT

Slide 6

Slide 6 text

$POWPMVUJPO http://deeplearning.net/software/theano/tutorial/conv_arithmetic.html input output input output kernel

Slide 7

Slide 7 text

4USJEF http://deeplearning.net/software/theano/tutorial/conv_arithmetic.html 2px 2px 2px 2px

Slide 8

Slide 8 text

1BEEJOH http://deeplearning.net/software/theano/tutorial/conv_arithmetic.html 2px 2px

Slide 9

Slide 9 text

4USJEFQBEEJOH http://deeplearning.net/software/theano/tutorial/conv_arithmetic.html

Slide 10

Slide 10 text

5SBOTQPTFE http://deeplearning.net/software/theano/tutorial/conv_arithmetic.html simulated here with padding also called “deconvolution” “fractional stride”

Slide 11

Slide 11 text

%PXOTBNQMJOH features or small resolution image convolutional layer or layers RGB image input output

Slide 12

Slide 12 text

6QTBNQMJOH upsampling CNN layer or layers RGB image features or small resolution image input output

Slide 13

Slide 13 text

&ODPEFS%FDPEFS D E image in Decoder Encoder image out feature space

Slide 14

Slide 14 text

'VMMZ$POOFDUFE $MBTTJpFS approve loan reject class data or features also called “Dense” layer

Slide 15

Slide 15 text

$//$MBTTJpFS food person plant other AlexNet, LeNet, VGG…

Slide 16

Slide 16 text

'PPE/FU ™ food not food

Slide 17

Slide 17 text

@teenybiscuit

Slide 18

Slide 18 text

No content

Slide 19

Slide 19 text

@teenybiscuit

Slide 20

Slide 20 text

@teenybiscuit

Slide 21

Slide 21 text

@teenybiscuit

Slide 22

Slide 22 text

@teenybiscuit

Slide 23

Slide 23 text

@teenybiscuit

Slide 24

Slide 24 text

(FOFSBUJWF "EWFSTBSJBM /FUXPSLT

Slide 25

Slide 25 text

Generator Discriminator https://speakerdeck.com/lunardog/deep-convolutional-voight-kampf-test “Couple of bots studying for the Turing Test”

Slide 26

Slide 26 text

Unsupervised Representation Learning with Deep Convolutional Generative Adversarial Networks Alec Radford, Luke Metz, Soumith Chintala (Submitted on 19 Nov 2015 (v1), last revised 7 Jan 2016 (this version, v2)) https://arxiv.org/abs/1511.06434

Slide 27

Slide 27 text

Generator Discriminator G MPPLTMFHJU UPUBMMZTIPQQFE D

Slide 28

Slide 28 text

G SFBM GBLF D D(G(noise)) ˠ real (FOFSBUPSUSBJOJOH Discriminator acts as the teacher

Slide 29

Slide 29 text

G SFBM GBLF D SFBM GBLF D D(G(noise)) ˠ fake D(photo) ˠ real %JTDSJNJOBUPSUSBJOJOH Generator provides negative examples

Slide 30

Slide 30 text

No content

Slide 31

Slide 31 text

https://www.youtube.com/watch?v=rs3aI7bACGc ©Yota Ishida

Slide 32

Slide 32 text

Slide 33

Slide 33 text

$POEJUJPOBM ("/T

Slide 34

Slide 34 text

G NBMF GFNBMF DIJME FMEFSMZ G(noise | conditions) $POEJUJPOBM(FOFSBUPS

Slide 35

Slide 35 text

SJHIU XSPOH NBMF GFNBMF DIJME FMEFSMZ D $POEJUJPOBM%JTDSJNJOBUPS

Slide 36

Slide 36 text

SJHIU XSPOH NBMF GFNBMF DIJME FMEFSMZ D SJHIU XSPOH NBMF GFNBMF DIJME FMEFSMZ SJHIU XSPOH NBMF GFNBMF DIJME FMEFSMZ D D

Slide 37

Slide 37 text

SJHIU XSPOH D $POEJUJPOBM("/ https://arxiv.org/abs/1411.1784 Conditional Generative Adversarial Nets Mehdi Mirza, Simon Osindero (Submitted on 6 Nov 2014) Generator Discriminator NBMF GFNBMF DIJME FMEFSMZ G NBMF GFNBMF DIJME FMEFSMZ same condition

Slide 38

Slide 38 text

G NBMF GFNBMF DIJME FMEFSMZ SJHIU XSPOH NBMF GFNBMF DIJME FMEFSMZ D $POEJUJPOBM("/ Discriminator Generator

Slide 39

Slide 39 text

https://www.faceapp.com/ Disclaimer: FaceApp authors don’t disclose their method. This is only my guess. It may have nothing to do with GANs. original

Slide 40

Slide 40 text

original https://www.faceapp.com/

Slide 41

Slide 41 text

https://www.faceapp.com/ original

Slide 42

Slide 42 text

"SUJTUJD4UZMF5SBOTGFS Improved!

Slide 43

Slide 43 text

https://prisma-ai.com/

Slide 44

Slide 44 text

https://prisma-ai.com/ https://prisma-ai.com/

Slide 45

Slide 45 text

https://prisma-ai.com/ https://prisma-ai.com/

Slide 46

Slide 46 text

https://prisma-ai.com/ https://prisma-ai.com/

Slide 47

Slide 47 text

https://arxiv.org/abs/1603.08155 transformation network loss network Gram matrices in feature space pre-trained content image style image

Slide 48

Slide 48 text

“Gram matrices in feature space” https://en.wikipedia.org/wiki/Gramian_matrix

Slide 49

Slide 49 text

https://www.youtube.com/watch?v=xVJwwWQlQ1o

Slide 50

Slide 50 text

$ZDMF("/

Slide 51

Slide 51 text

https://github.com/junyanz/CycleGAN

Slide 52

Slide 52 text

https://github.com/junyanz/CycleGAN

Slide 53

Slide 53 text

https://github.com/junyanz/CycleGAN

Slide 54

Slide 54 text

(FOFSBUPS transformation network https://arxiv.org/abs/1603.08155

Slide 55

Slide 55 text

GBLF IPSTF GBLF IPSTF … %JTDSJNJOBUPS fully convolutional judges patches   of the input image https://arxiv.org/abs/1603.08155

Slide 56

Slide 56 text

"EWFSTBSJBM-PTT X F G Y GBLF [FCSB GBLF [FCSB … GBLF IPSTF GBLF IPSTF … X(F(horse)) ˠ classify as zebra Y(F(zebra)) ˠ classify as horse

Slide 57

Slide 57 text

$ZDMF-PTT G F G(F(image))ˠ the same image F G F(G(image))ˠ the same image

Slide 58

Slide 58 text

https://www.youtube.com/watch?v=9reHvktowLY

Slide 59

Slide 59 text

5IF&OE