Share
Embed
EdgeConnect: Generative Image Inpainting with Adversarial Edge Learning
[DL-paper] 2019年1月にarxivに投稿されたEdgeConnectというInpainting手法に関する論文.論文(https://arxiv.org/abs/1901.00212) , Code (https://github.com/knazeri/edge-connect)
tanimutomo
March 29, 2019
Transcript
33 slides
-
EdgeConnect: Generative Image Inpainting with Adversarial Edge Learning Tomoki
Tanimura, B4 Jin Nakazawa Lab, Keio University -
w λΠτϧ w &EHF$POOFDU(FOFSBUJWF*NBHF*OQBJOUJOHXJUI"EWFSTBSJBM&EHF-FBSOJOH w ஶऀ w ,BNZBS/B[FSJ &SJD/H 5POZ+PTFQI
'BJTBM;2VSFTIJ .ISBO&CSBIJNJ w $BOBEBͷ0OUBSJPେֶ w ߘ w ݄ʹ"SYJWʹߘ͞Εͨͷ ॻࢽใ !2 -
w લఏ w ਓؒը૾Λݟͨͱ͖ʹɼҰ෦͚ͩҟͳΔ෦͕͋ΔʢҰ؏ੑ͕อ͍ͯͯͳ͍෦͕͋Δʣͱɼ ͦ͜ʹҙ͕ࣝूதͯ͠͠·͏ w طଘख๏ͷ w *OQBJOUJOHͨ͠෦͚͕ͩɼաʹฏԽ͞Εͯɼ΅͚ͯ͠·͍ͬͯΔ w
ˠԿͱ͔͍ͨ͠ w Ծઆ w -JOFT'JSTU $PMPS/FYU w ը૾ͷ&EHF.BQΛར༻͢Εɼ͖ͬΓͨ͠ը૾͕࠶ߏͰ͖ΔͷͰͳ͍͔ʁ Ϟνϕʔγϣϯ !4 -
w ֦ࢄϕʔε %J⒎VTJPOCBTFE w ύονϕʔε 1BUDICBTFE w ֶशϕʔε
-FBSOJOHCBTFE w ωοτϫʔΫͷ طଘख๏ !7 -
w ಛ w पลϐΫηϧͷՃॏฏۉΛͱͬͨΓ͢Δख๏ w 0QFO$7ʹσϑΥϧτͰ࣮͞Ε͍ͯΔΑ͏ͳख๏ w ݶք w ը૾ͷҙຯతͳใΛߟྀͰ͖ͳ͍
w େ͖ͳൣғΛ࠶ߏ͢Δ͜ͱʹݶք͕͋Δ ֦ࢄϕʔε %J⒎VTJPOCBTFE !8 -
w ಛ w ϚεΫྖҬपลʹྨࣅ͍ͯ͠ΔྖҬΛಉ͡ը૾ͳ͍͔Βݕࡧ͠షΓ͚Δ w ϚεΫྖҬʹྨࣅ͢ΔྖҬ͕ը૾ʹଘࡏ͢Δ͜ͱ͕લఏ w ݶք w إը૾ʹ͓͍ͯޱΛ࠶ߏ͢ΔέʔεͷΑ͏ʹ
ը૾ʹͳ͍ͷΛ࠶ߏ͢Δ͜ͱͰ͖ͳ͍ w ग़ݱස͕͍ͷΛ࠶ߏ͕͍͠ w 1BUDI.BUDI w ྨࣅύονͷݕࡧΛϥϯμϜʹߦ͏ΞϧΰϦζϜΛఏҊ͠ɼ ܭࢉ࣌ؒΛѹతʹॖͨ͠ ύονϕʔε !9 PatchMatch -
w ಛ w ("/ͷ(FOFSBUPSΛ*NBHF*OQBJOUJOH/FUXPSLʹֶͯ͠श͢Δ w ೖྗݩը૾ͱϚεΫը૾ɼ࠶ߏͨ͠ͷͱݩը૾Λ%JTDSJNJOBUPSʹ͔͚Δ w ݶք w աʹฏԽ͞Εͯ΅͚ͨग़ྗʹͳͬͯ͠·͏
w (-$*$ w %JMBUFE$POWͭͷ(MPCBMͱ-PBDMͷ %JTDSJNJOBUPSʹΑͬͯߴਫ਼ͷ*OQBJOUJOHΛ࣮ݱ ֶशϕʔε !10 GLCIC -
w ಛ w جຊֶशϕʔεͷख๏Ͱɼ༷ʑͳ$7ܥͷλεΫͰΘΕΔςΫχοΫΛΓࠐΜͩ w ݶք w ͦͦ$PBSTFSFTVMU͕ਖ਼͍͠ߏΛ͍࣋ͬͯΔͱ ݶΒͳ͍ w
ϚεΫͷܗ͕ݶΒΕΔ w $" XJUI$POUFYUVBM"UUFOUJPO w $PBSTFUP'JOFߏͷಋೖ w $PBSTF3FTVMUͱݩʑͷೖྗը૾ͱͷ ྨࣅΛܭࢉ͢Δ"UUFOUJPOػߏΛಋೖ ωοτϫʔΫͷ !11 CA (Generative Image Inpainting with Contextual Attention) -
w ( &EHF(FOFSBUPS ͱ( *NBHF$PNQMFUJPO/FUXPSL Ͱߏ͞ΕΔ w &EHF(FOFSBUFS (
w ೖྗάϨʔεέʔϧը૾ɼ&EHFը૾ɼϚεΫը૾ w ग़ྗ࠶ߏ͞Εͨ&EHFը૾ w *NBHF$PNQMFUJPO/FUXPSL ( w ೖྗݩը૾ɼ࠶ߏ͞Εͨ&EHFը૾ w ग़ྗ࠶ߏը૾ &EHF$POOFDU !12 Edge Generater Image Completion Net ֶश • G1ͱG2ΛɼD1, D2Λ༻͍ͯ ͦΕͧΕผʑʹֶश • D1Λൈ͍ͯɼG1ͱG2Λ End-to-EndͰֶश -
w (FOFSBUFS w %JMBUFE3FTJEVBM#MPDLT w ֤$POWPMVUJPO4QFDUSBM/PSN *OTUBODF/PSN3F-6 w -PTT w
"EWFSTBSJBM-PTT 'FBUVSF.BUDIJOH-PTT w "EWFSTBSJBM-PTTʢ4/("/ w 'FBUVSF.BUDIJOH-PTT &EHF(FOFSBUPS !13 -
w (FOFSBUFS w ϊΠζ͔ΒԿΒ͔ͷը૾Λੜ͢Δ w %JTDSJNJOBUFS w ੜ͞Εͨը૾ͱσʔληοτʹ͋ΔϦΞϧ ͳը૾ΛݟͯɼͲͪΒ͕ຊͷը૾͔Λݟ ͚Δ
w Πϝʔδ w %JTDSJNJOBUFSͷݟ͚Δೳྗ͕ߴ·Δ w (FOFSBUFSݟ͚Δೳྗ͕ߴ·ͬͨ %JTDSJNJOBUFSΛͩ·͢Α͏ʹֶश͢Δ w ݁Ռతʹ(FOFSBUFS͕ຊͷΑ͏ͳը૾Λੜ Ͱ͖ΔΑ͏ʹͳΔ "EWFSTBSJBM-PTT !14 -
w ೖྗը૾ͱɼೖྗը૾ʹϚεΫΛ͔͚ͨը૾ Λ༻ҙ w (FOFSBUFS w ϚεΫΛؚΉೖྗը૾͔ΒϚεΫΛ࠶ߏ ͨ͠ը૾Λੜ w %JTDSJNJOBUFS
w ݩʑͷೖྗը૾ͱϚεΫ෦͕࠶ߏ͞Ε ͨೖྗը૾Λࣝผ͢ w తؔ w ɹ w $$PNQMFUJPO/FUXPSL (FOFSBUFS %%JTDSJNJOBUFS ..BTL *NBHF*OQBJOUJOHʹ͓͚Δఢରతֶश !15 -
w %JTDSJNJOBUFSͷதؒͷ'FBUVSF.BQൺֱ͢Δ w ࠶ߏը૾Λೖྗͨ࣌͠ͷ%ͷ'FBUVSF.BQ w ݩը૾Λೖྗͨ࣌͠ͷ%ͷ'FBUVSF.BQΛൺֱ w ֤'FBUVSF.BQͷ--PTTͷظΛ࠷খԽ͢Δ w 1FSDFQUVBM-PTTʹࣅ͍ͯΔ͕ɼࠓճ&EHFը૾Ͱ͋ΔͨΊɼ1FSDFQUVBM-PTTʹΑΔֶश
͍ͨ͠Ίɼ'FBUVSF.BUDIJOH-PTTʹͨ͠ 'FBUVSF.BUDIJOH-PTT !17 ੜը૾ͷ Feature Map ϦΞϧը૾ͷ Feature Map ֤Feature Mapͷ ࠩͷظ -
w (FOFSBUFS&EHF(FOFSBUFSͱಉ͡ w -PTT w --PTT "EWFSTBSJBM-PTT 1FSDFQUVBM-PTT 4UZMF-PTT w
1FSDFQUVBM-PTT w 4UZMF-PTT w "EWFSTBSJBM-PTT XJUIPVU4/ w -PTTؔʹΑΔ੍͕ଟ͍ͨΊɼ4/ΛೖΕΔͱֶश͕ۃʹ͘ͳΔͨΊɼൈ͍ͨ *NBHF$PNQMFUJPO/FUXPSL !18 -
w ୯७ͳੜ݁ՌͱϦΞϧը૾ͷ݁ՌͷࠩΛଌΔͷ Ͱͳ͘ɼੜ݁ՌͱϦΞϧը૾͕ྨࣅͨ͠ಛྔ Λ࣋ͭΑ͏ʹֶश͢Δ ը૾ͷίϯςϯπ͚ۙͮΔʣ w ੜը૾ͱϦΞϧը૾Λ-PTT/FUXPSL 7(( QFSUBJOFECZ*NBHF/FUͳͲʣʹೖྗͨ͠ࡍͷ 'FBUVSF.BQͷࠩ
w ɹ 1FSDFQUVBM-PTT !19 9 Perceptual Lossͷจͷਤ ੜը૾ͷ Feature Map ϦΞϧը૾ͷ Feature Map ֤Feature Mapͷ ࠩͷظ -
w ੜը૾ͱϦΞϧը૾ͷελΠϧʢ৭ɼςΫενϟɼ ࢹ֮తͳύλʔϯʣ͕ࣅΔΑ͏ʹֶश w 1FSDFQUVBM-PTTͰར༻͢Δ'FBUVSF.BQͷϖΞʢੜ ͱϦΞϧʣͷάϥϜϚτϦοΫεͷࠩ w 4UZMF-PTT !20 9
Style Lossͷจͷਤ ੜը૾ͷ Feature Mapͷ Gram Matrix ϦΞϧը૾ͷ Feature Mapͷ Gram Matirx Gram Matrix νϟϯωϧcͱc’ʹ͓͚ΔFeature Mapͷཁૉ੮ͷ -
w &EHFͷݕग़ w ਖ਼ղσʔλͷ࡞ؚΊɼ$BOOZΤοδݕग़Λར༻ͨ͠ w ଞͷ&EHFݕग़ख๏Λ༻ͨ࣌͠ͷਫ਼ൺֱߦͳ͍ͬͯΔ w .BTLը૾ w 3FHVMBS)PMFͱ*SSFHVMBSIPMFͲͪΒ༻࣮ͯ͠ݧͨ͠
w ը૾σʔληοτ w $FMFC" 1MBDFT 1BSJT4USFFU7JFX w αΠζY ࣮ݧ !21 Regular Hole Irregular Hole CelebA Places2 Paris Street View -
w ఆੑධՁ w *OQBJOUJOHͷ݁Ռͱ&EHFը૾ͷ*OQBJOUJOHͷ݁Ռ w (-$*$ $"ͷ݁Ռͱͷൺֱ w ఆྔධՁ w
- 14/3 44*. '*% w ϢʔβʔධՁ w ը૾ΛݟͤͯϦΞϧ͔൱͔Λ͑Δςετ w ϦΞϧը૾ͱ࠶ߏ݁Ռͷը૾ΛݟͤͯͲͪΒ͕ϦΞϧ͔Λ͑Δςετ w &EHFΛར༻͢Δ͜ͱʹΑΔਫ਼্ʹؔ͢Δݕূ ࣮ݧ༰ !22 -
w ը૾ͷൣғʹϚεΫ͕͔͔͍ͬͯͯϦΞϧͳ࠶ߏ͕Մೳ w 4UZMF-PTTʹΑͬͯ$IFDLFSCPBSE༷͕ͳ͘ɼ៉ྷʹੜ͞Ε͍ͯΔɽ ఆੑධՁ !23 Original Input (Inpainted) Edge
Inpainted Original Input (Inpainted) Edge Inpainted -
w 14/3ʢϐʔΫ৴߸ϊΠζൺʣ w ͭͷը૾ؒͷϊΠζൺΛද͢ɽ w 44*.ʢ4USVDUVBM4JNJMBSJUZʣ w ըૉͷมԽɼίϯτϥετͷมԽɼߏͷมԽΛද͢ ఆྔධՁࢦඪ !25
PSNR = 10 ⇥ log10 MAX2 I MSE = 20 ⇥ log10 MAXI p MSE <latexit sha1_base64="6cdW3QK+V7H5Ue5NeKZnMNCX958=">AAACU3ichVFNS8MwGE7r93Q69eglOBRPo52CXoSpCHpQ5sd0sM6RZukWl36YvBVG6X8UwYN/xIsHTesO6gRfCDw8H+TNEzcSXIFlvRrmxOTU9MzsXGF+obi4VFpeuVFhLClr0FCEsukSxQQPWAM4CNaMJCO+K9itOzjK9NtHJhUPg2sYRqztk17APU4JaKpTuq9fnV/iTbyPbQs7wH2mcOJIH4uwl+acJwlNzg6ad9XOaZqcXR2n2HEKWaL6TyLzO+pBQp5KO6WyVbHywePAHoEyGk29U3p2uiGNfRYAFUSplm1F0E6IBE4FSwtOrFhE6ID0WEvDgOhV2kneSYo3NNPFXij1CQDn7PdEQnylhr6rnT6BvvqtZeRfWisGb6+d8CCKgQX06yIvFhhCnBWMu1wyCmKoAaGS610x7RNdCehvKOgS7N9PHgc31Yq9Xale7JRrh6M6ZtEaWkdbyEa7qIZOUB01EEVP6A19GMh4Md5N05z8sprGKLOKfoxZ/AS+e69x</latexit> SSIM = (2µxµy + c1)(2 xy + c2) (µ2 x + µ2 y + c1)( 2 x + 2 y + c2) <latexit sha1_base64="te08wqiCIK9nNox9fuCmoQRUS/U=">AAACW3icbVFdS+swGE7r15xfU/HKmxeHUBFGWwW9OTD05ngheNCpsM6SZukMJm1J0sNK2Z/0ynNx/srBdN3g+PFCypPngyRPo4wzpV33j2UvLC4trzRWm2vrG5tbre2de5XmktAeSXkqHyOsKGcJ7WmmOX3MJMUi4vQherms9IffVCqWJne6yOhA4FHCYkawNlTYkre3V9fwA4JYYlI6fiDycAzVt4BjIKF3BA74ECg2Ejgsx8Wkov2jSekY05NvzMdQo3nAqc1zbb6pZZMMW223404HvgJvBtpoNjdh6zUYpiQXNNGEY6X6npvpQYmlZoTTSTPIFc0wecEj2jcwwYKqQTntZgKHhhlCnEqzEg1T9v9EiYVShYiMU2D9rD5rFfmd1s91fD4oWZLlmiakPijOOegUqqJhyCQlmhcGYCKZuSuQZ2xa1uZ3NE0J3ucnfwX3fsc76fi/Ttvdi1kdDbSPDpCDPHSGuugnukE9RNAb+metWA3rr71gN+312mpbs8wu+jD23js2Ya63</latexit> -
w '*% 'SFDIFU*ODFQUJPO%JTUBODF w ը૾ؒͷಛΛൺֱ͢ΔείΞʢϐΫηϧґଘ͡Όͳ͍ʣ w *ODFQUJPO7ʹϦΞϧը૾Λೖྗͨ࣌͠ͱੜ͞Εͨը૾Λೖྗͨ࣌͠ͷதؒͷϕΫτϧIΛ ൺֱ w
֤ϕΫτϧI͕ଟ࣍ݩͷΨεʹै͍ͬͯΔͱԾఆ͠ɼؒͷ8BTTFSTUFJOڑΛଌΔ ఆྔධՁࢦඪ !26