Deep Learning for clothes and changing pose

%FFQ-FBSOJOHBOE'BTIJPO .,!DSDSQBS All ﬁgures are from papers referred in this
keynote.

50$ MJTUPGQBQFST PWFSWJFXGSPNUIFQFSTQFDUJWFPG HFOFSBUFETBNQMFT EBUBTFUT NPEFMT MPTTFT
EFUBJMT 2

("/T w *(PPEGFMMPXQSPQPTFE("/TJO w 5IFSFBSFQMFOUZPG("/T w 8("/ $SBNFS("/ 5SJBOHMF("/ 1SPHSFTTJWF(SPXJOH("/
w QJYQJY $ZDMF("/ #JDZDMF("/ 4UBS("/ 3

'BTIJPOJO$PNQVUFS7JTJPO w 'BTIJPOJTBUUSBDUJOHNPSFBOENPSFSFTFBSDIFST w %FFQ'BTIJPO EBUBTFU w DPNQBUJCJMJUZFTUJNBUJPO DPPSEJOBUFHSBEJOHDPNQMFNFOUBUJPO
w IVNBOQBSTJOH TFNBOUJDTFHNFOUBUJPO w GBTIJPOBSUJDMFBUUSJCVUFTQSFEJDUJPOHFOFSBUJPO w GBTIJPOBSUJDMFEFTJHOTZOUIFTJT 4

MJTUPGQBQFST Jetchev, N., & Bergmann, U. (2017). The Conditional
Analogy GAN: Swapping Fashion Articles on People Images. Retrieved from http://arxiv.org/abs/1709.04695 - ZalandoGAN Zhu, S., Fidler, S., Urtasun, R., Lin, D., & Loy, C. C. (2017). Be Your Own Prada: Fashion Synthesis with Structural Coherence. Retrieved from http://arxiv.org/abs/1710.07346 - FashionGAN Han, X., Wu, Z., Wu, Z., Yu, R., & Davis, L. S. (2017). VITON: An Image-based Virtual Try-on Network. Retrieved from http://arxiv.org/abs/1711.08447 - VITON Ma, L., Sun, Q., Georgoulis, S., Van Gool, L., Schiele, B., & Fritz, M. (2017). Disentangled Person Image Generation. Retrieved from http://arxiv.org/abs/1712.02621 - DPIG Siarohin, A., Sangineto, E., Lathuiliere, S., & Sebe, N. (n.d.). Deformable GANs for Pose-based Human Image Generation. Retrieved from https://arxiv.org/pdf/1801.00055.pdf - Deformable GAN GANs for Try-on Ma, L., Sun, Q., Jia, X., Schiele, B., Tuytelaars, T., & Gool, L. Van. (n.d.). Pose Guided Person Image Generation. Retrieved from https://arxiv.org/pdf/1705.09368.pdf - PG2

OPUBUJPOT w IVNBO w BSUJDMFBGBTIJPOJUFNXPSOCZBNPEFMJOʠIVNBOʡJNBHF w UBSHFUBGBTIJPOJUFNXIJDIBNPEFMXJMMXFBS w ПUSBJOFE7((VTJOH*NBHF/FU w
TVGpYAJANFBOTCMPDLJOEFYJO7((NPEFMT w Y Z DPODBUFOBUFEJNBHFXIPTFPGDIBOOFMTJTY Z 6

PWFSWJFX

2VBMJUBUJWFDPNQBSJTPO ZalandoGAN FashionGAN VITON DPIG Deformable GAN PG2

EBUBTFUDPNQBSJTPO w [BMBOEP w DSFBUFEGSPN[BMBOEPTF w "1*JT<IFSF> w UIFSFJTOPUBOZQVCMJDEBUBTFU w
%FFQ'BTIJPO w BWBJMBCMF<IFSF> 9 [BMBOEP %FFQ'BTIJPO ;BMBOEP("/ ✔ ✘ 1( ✘ ✔ 'BTIJPO("/ ✘ ✔ 7*50/ ✔ ✘ %1*( ✘ ✔ %FGPSNBCMF ("/ ✘ ✔

NPEFMDPNQBSJTPO w 1&1PTF&TUJNBUPS 10 TJOHMFTUFQ HFOFSBUJPO XJUI1& UFYU ;BMBOEP("/ ✔
✘ ✘ 1( ✘ ✔ ✘ 'BTIJPO("/ ✘ ✘ ✔ 7*50/ ✘ ✔̈ ✘ %1*( ✘ ✔ ✘ %FGPSNBCMF ("/ ✘ ✔ ✘ †: use human parser

MPTTDPNQBSJTPO 11 ;BMBOEP("/ $ZDMFMPTT -QOPSN "EWFSTBSJBMMPTT 1( NBTL- "EWMPTT
'BTIJPO("/ "EWMPTT 7*50/ -OPSN NBTL 1FSDFQUVBMMPTT %1*( -OPSN NBTL "EWMPTT 1PTFMPTT NBTLMPTT %FGPSNBCMF("/ "EWMPTT OFBSFTUOFJHICPSMPTT

EFUBJMT

;BMBOEP("/ w 5IFQJPOFFSPG/FVSBM/FUTXBQQJOHGBTIJPOBSUJDMFT w %BUBTFUJTDSFBUFEGSPNJNBHFTJO[BMBOEPEF w (FOFSBUPS w JOQVUUVQMFPGIVNBO JUFNBOEUBSHFUJUFN
w PVUQVUBSUJDMFTXBQQFEIVNBO JUFNBSFB NBTL JOIVNBO w %JTDSJNJOBUPS w JOQVUUVQMFPGIVNBOT 13

;BMBOEP("/ w PCKFDUJWF w "EWFSTBSJBMMPTTXIFUIFSJOQVUTBSFSFBMPSGBMTF w $ZDMFMPTTGPSTXBQQJOHUPIBWFDPOTJTUFODZ w -MPTTUPSFHVMBSJ[FNBTL 14

;BMBOEP("/ 15 Adversarial loss Cycle loss

1( OJQT w OPUGPSTXBQQJOHDMPUIFT w HFUVTFPG1PTF&TUJNBUPSUPHFUUIFFYBDUBSFBPGBSUJDMF w TUFQHFOFSBUJPO w 4UBHFDIBOHFTQPTFBOEPVUQVUDPBSTFJNBHF
w 4UBHFSFpOFJNBHF 16

1( OJQT w 6/FUMJLFNPEFMTBSFVTFE w 1PTF&TUJNBUPSFYUSBDULFZQPJOUT w HFOFSBUJOHqPX TUBHFUBLFT
DPOEJUJPOJNBHF UBSHFUQPTF BTJOQVU TUBHFUBLFT DPOEJUJPOJNBHF DPBSTFSFTVMU BTJOQVU w (FOFSBUPSTGPSTUBHFTBSFMJLF6/FU 17

1( w PCKFDUJWF w .BTLFE-MPTTˠ4UBHF w -OPSNPG DPBSTFUBSHFU Y.# w
"EWFSTBSJBMMPTTˠ4UBHF w .BTLFE-MPTTˠ4UBHF w -OPSNPG SFpOFEUBSHFU Y.# 18 MB

'BTIJPO("/ JDDW w TXBQQJOHDMPUIFTBDDPSEJOHUPHJWFOUFYU w OFXEBUBTFUBWBJMBCMFJO%FFQ'BTIJPO BDBEFNJDVTF w OPUSFRVJSFQBJSFEEBUBTFU
w SFRVJSFTFNBOUJDTFHNFOUBUJPO BCCSFWJBUJPO44 w 44NPEFMBSFUSBJOFEPO"53EBUBTFU w CBTF$//JT7(( 19

'BTIJPO("/ JDDW w FYUSBDUEFTJHODPEJOH B W BOE44NBQ4 w BWFDUPSPGCJOBSZBUUSJCVUFTPGJOQVUJNBHF  TPNFBUUSJCVUFTBSFQSFTFSWFEXIJMFHFOFSBUJPO
w WWFDUPSPGFODPEFEUFYU w 4QFSTPOʟTTFHNFOUBUJPONBQ w IBJS GBDF VQQFSDMPUIFT QBOUTTIPSUT FUD 20

'BTIJPO("/ w TIBQF(FOFSBUPSUBLFTʠEPXOTBNQMFE4ʡN 4 OPUSBX4 w N 4 XPSLTBTTQBUJBMDPOTUSBJOU w
HFOFSBUF44NBQ 4 XIJDIJTGPSUBSHFUJNBHF w JNBHF3FOEFSFSUBLFT4BOEEFTJHODPEJOHBTJOQVU 21

7*50/ w CFUUFSUIBO;BMBOEP("/ w BMTP 7*50/VTFFODPEFSEFDPEFSOFUTJOTUFBEPG("/T w SFRVJSF)VNBO1BSTFS OPU1PTF&TUJNBUPS w
SFTIBQFDMPUIFTNBTLUPpUUBSHFUDMPUIFTCZ514 w SFEVDFUIFBSFBUPESBXOFXMZBTQPTTJCMF w DPBSTFUPpOFHSBJOFEHFOFSBUJPONFUIPE w QFSDFQUVBMMPTTJOTUFBEPG- CFDBVTFUIFMBUUFSUFOEUPHFOFSBUFCMVSSZJNBHFTXIFOBQQMJFEUPDPMPS JNBHFT w 5PUBM7BSJBUJPO aHSBEaBMQIB w XBSQFE-OPSNPGaBMQIB 22 TPS

7*50/ w FODPEFSHFOFSBUPSOFUXPSLT w (D DPBSTF w JOQVU QFSTPOSFQSFTFOUBUJPO
UBSHFUDMPUIFT w PVUQVUDPBSTFSFTVMUBOENBTL w (S SFpOFNFOU w JOQVU XBSQFEUBSHFU DPBSTFSFTVMU w PVUQVUBMQIBNBTL 23

7*50/ 24 how to generate person representation 2-step generation

%1*( w ʠTUBHFGSBNFXPSLUPMFBSO  NBOJQVMBUBCMFFNCFEEJOHGFBUVSFTGPSBMMUISFFGBDUPSTʡ w GPSFHSPVOE w CBDLHSPVOE w QPTF
w MFBSONBQQJOH(BVTTJBOEJTUUPGFBUVSF 25

%1*( w NBOJQVMBUBCMFFNCFEEJOHGFBUVSFTMFBSOJOH FYUSBDUGFBUVSFTVTJOHNVMUJCSBODIFEBVUPFODPEFS  DPOEJUJPOBMJNBHFHFOFSBUJPOUBTL NBQQJOH(BVTTJBOUPGFBUVSFFNCFEEJOHEJTUSJCVUJPO  MFBSOFEUISPVHIBEWFSTBSJBMUSBJOJOH  TBNQMJOHUBTLT
26

%1*( w 4UBHF w FYUSBDUGPSFHSPVOECBDLHSPVOEGFBUVSFTCZBQQMZJOHNBTLNBEFGSPNQPTF JOGPSNBUJPODPNQVUFECZ1&UPGFBUVSF  FYUSBDUFEGSPN$// w
TQMJU'(JOUPSFHJPOT w XFʟWFHPU w QPTFGFBUVSF w '(#(GFBUVSF 27

%1*( w 4UBHF w FODPEFLJOETPGGFBUVSFTVTJOHDPSSFTQPOEJOHFODPEFST FODPEFEQPTF
'(T #( 28

%1*( w 4UBHF w QPTFEFDPEFSSFDPOTUSVDUQPTF JNBHFGSPNFODPEFEQPTFGFBUVSF w '(#(EFDPEFSSFDPOTUSVDUJOQVU
JNBHFGSPN'(T#(GFBUVSFT DPODBUFOBUFEXJUIQPTFJNBHF 29

%1*( w -PTTGPSTUBHF w QPTFSFDPOTUSVDUJPO- w '(#( w BEWFSTBSJBMMPTTXSUGFBUVSFT w
%JTDSJNJOBUFXIFUIFSJOQVUJTGSPNFODPEFSTPS(BVTTJBO w -OPSNXSUJOQVUBOESFDPOTUSVDUFEJNBHF 30

%1*( w 4UBHF w HFOFSBUFTBNQMFTUPEFDPEFOFXJNBHFT w NBQ(BVTTJBOUPFODPEFEQPTF'(#(GFBUVSFT 31

%FGPSNBCMF("/ w GPDVTPODIBOHJOHQPTF w QSPQPTF/FBSFTU/FJHICPS-PTT  DBOCFVTFEJOTUFBEPG--MPTT w UPMFSBUFUPTNBMMTQBUJBMNJTBMJHONFOUT w TJNJMBSUPQFSDFQUVBMMPTTJOUIBUVTJOH*NBHF/FUQSFUSBJOFE
7(( SFMV@VTFE 32 I haven’t understood this well.

%FGPSNBCMF("/ w SFRVJSFQBJSFEJNBHFEEBUBTFU w TLJQDPOOFDUJPO w BGpOFUSBOTGPSNBUJPOUPBQQSPYJNBUFEFGPSNBUJPO w -PTT w
BEWFSTBSJBMMPTT w //MPTT 33

Deep Learning for clothes and changing pose

Deep Learning for clothes and changing pose

Masaki Kozuki

More Decks by Masaki Kozuki

Other Decks in Research

Featured

Transcript

%FFQ-FBSOJOHBOE'BTIJPO .,!DSDSQBS All ﬁgures are from papers referred in this

50$ MJTUPGQBQFST PWFSWJFXGSPNUIFQFSTQFDUJWFPG HFOFSBUFETBNQMFT EBUBTFUT NPEFMT MPTTFT

("/T w *(PPEGFMMPXQSPQPTFE("/TJO w 5IFSFBSFQMFOUZPG("/T w 8("/ $SBNFS("/ 5SJBOHMF("/ 1SPHSFTTJWF(SPXJOH("/

'BTIJPOJO$PNQVUFS7JTJPO w 'BTIJPOJTBUUSBDUJOHNPSFBOENPSFSFTFBSDIFST w %FFQ'BTIJPO EBUBTFU w DPNQBUJCJMJUZFTUJNBUJPO DPPSEJOBUFHSBEJOHDPNQMFNFOUBUJPO

MJTUPGQBQFST Jetchev, N., & Bergmann, U. (2017). The Conditional

OPUBUJPOT w IVNBO w BSUJDMFBGBTIJPOJUFNXPSOCZBNPEFMJOʠIVNBOʡJNBHF w UBSHFUBGBTIJPOJUFNXIJDIBNPEFMXJMMXFBS w ПUSBJOFE7((VTJOH*NBHF/FU w

PWFSWJFX

2VBMJUBUJWFDPNQBSJTPO ZalandoGAN FashionGAN VITON DPIG Deformable GAN PG2

EBUBTFUDPNQBSJTPO w [BMBOEP w DSFBUFEGSPN[BMBOEPTF w "1*JT<IFSF> w UIFSFJTOPUBOZQVCMJDEBUBTFU w

NPEFMDPNQBSJTPO w 1&1PTF&TUJNBUPS 10 TJOHMFTUFQ HFOFSBUJPO XJUI1& UFYU ;BMBOEP("/ ✔

MPTTDPNQBSJTPO 11 ;BMBOEP("/ $ZDMFMPTT -QOPSN "EWFSTBSJBMMPTT 1( NBTL- "EWMPTT

EFUBJMT

;BMBOEP("/ w 5IFQJPOFFSPG/FVSBM/FUTXBQQJOHGBTIJPOBSUJDMFT w %BUBTFUJTDSFBUFEGSPNJNBHFTJO[BMBOEPEF w (FOFSBUPS w JOQVUUVQMFPGIVNBO JUFNBOEUBSHFUJUFN

;BMBOEP("/ w PCKFDUJWF w "EWFSTBSJBMMPTTXIFUIFSJOQVUTBSFSFBMPSGBMTF w $ZDMFMPTTGPSTXBQQJOHUPIBWFDPOTJTUFODZ w -MPTTUPSFHVMBSJ[FNBTL 14

;BMBOEP("/ 15 Adversarial loss Cycle loss

1( OJQT w OPUGPSTXBQQJOHDMPUIFT w HFUVTFPG1PTF&TUJNBUPSUPHFUUIFFYBDUBSFBPGBSUJDMF w TUFQHFOFSBUJPO w 4UBHFDIBOHFTQPTFBOEPVUQVUDPBSTFJNBHF

1( OJQT w 6/FUMJLFNPEFMTBSFVTFE w 1PTF&TUJNBUPSFYUSBDULFZQPJOUT w HFOFSBUJOHqPX TUBHFUBLFT

1( w PCKFDUJWF w .BTLFE-MPTTˠ4UBHF w -OPSNPG DPBSTFUBSHFU Y.# w

'BTIJPO("/ JDDW w TXBQQJOHDMPUIFTBDDPSEJOHUPHJWFOUFYU w OFXEBUBTFUBWBJMBCMFJO%FFQ'BTIJPO BDBEFNJDVTF w OPUSFRVJSFQBJSFEEBUBTFU

'BTIJPO("/ JDDW w FYUSBDUEFTJHODPEJOH B W BOE44NBQ4 w BWFDUPSPGCJOBSZBUUSJCVUFTPGJOQVUJNBHF  TPNFBUUSJCVUFTBSFQSFTFSWFEXIJMFHFOFSBUJPO

'BTIJPO("/ w TIBQF(FOFSBUPSUBLFTʠEPXOTBNQMFE4ʡN 4 OPUSBX4 w N 4 XPSLTBTTQBUJBMDPOTUSBJOU w

750/ w CFUUFSUIBO;BMBOEP("/ w BMTP 750/VTFFODPEFSEFDPEFSOFUTJOTUFBEPG("/T w SFRVJSF)VNBO1BSTFS OPU1PTF&TUJNBUPS w

7*50/ w FODPEFSHFOFSBUPSOFUXPSLT w (D DPBSTF w JOQVU QFSTPOSFQSFTFOUBUJPO

7*50/ 24 how to generate person representation 2-step generation

%1*( w ʠTUBHFGSBNFXPSLUPMFBSO  NBOJQVMBUBCMFFNCFEEJOHGFBUVSFTGPSBMMUISFFGBDUPSTʡ w GPSFHSPVOE w CBDLHSPVOE w QPTF

%1*( w NBOJQVMBUBCMFFNCFEEJOHGFBUVSFTMFBSOJOH FYUSBDUGFBUVSFTVTJOHNVMUJCSBODIFEBVUPFODPEFS  DPOEJUJPOBMJNBHFHFOFSBUJPOUBTL NBQQJOH(BVTTJBOUPGFBUVSFFNCFEEJOHEJTUSJCVUJPO  MFBSOFEUISPVHIBEWFSTBSJBMUSBJOJOH  TBNQMJOHUBTLT

%1*( w 4UBHF w FYUSBDUGPSFHSPVOECBDLHSPVOEGFBUVSFTCZBQQMZJOHNBTLNBEFGSPNQPTF JOGPSNBUJPODPNQVUFECZ1&UPGFBUVSF  FYUSBDUFEGSPN$// w

%1*( w 4UBHF w FODPEFLJOETPGGFBUVSFTVTJOHDPSSFTQPOEJOHFODPEFST FODPEFEQPTF

%1*( w 4UBHF w QPTFEFDPEFSSFDPOTUSVDUQPTF JNBHFGSPNFODPEFEQPTFGFBUVSF w '(#(EFDPEFSSFDPOTUSVDUJOQVU

%1*( w -PTTGPSTUBHF w QPTFSFDPOTUSVDUJPO- w '(#( w BEWFSTBSJBMMPTTXSUGFBUVSFT w

%1*( w 4UBHF w HFOFSBUFTBNQMFTUPEFDPEFOFXJNBHFT w NBQ(BVTTJBOUPFODPEFEQPTF'(#(GFBUVSFT 31

%FGPSNBCMF("/ w GPDVTPODIBOHJOHQPTF w QSPQPTF/FBSFTU/FJHICPS-PTT  DBOCFVTFEJOTUFBEPG--MPTT w UPMFSBUFUPTNBMMTQBUJBMNJTBMJHONFOUT w TJNJMBSUPQFSDFQUVBMMPTTJOUIBUVTJOH*NBHF/FUQSFUSBJOFE

%FGPSNBCMF("/ w SFRVJSFQBJSFEJNBHFEEBUBTFU w TLJQDPOOFDUJPO w BGpOFUSBOTGPSNBUJPOUPBQQSPYJNBUFEFGPSNBUJPO w -PTT w