EdgeConnect: 
 Generative Image Inpainting with Adversarial Edge Learning Tomoki Tanimura, B4 Jin Nakazawa Lab, Keio University

w λΠτϧ w &EHF$POOFDU(FOFSBUJWF*NBHF*OQBJOUJOHXJUI"EWFSTBSJBM&EHF-FBSOJOH w ஶऀ w ,BNZBS/B[FSJ &SJD/H 5POZ+PTFQI 'BJTBM;2VSFTIJ .ISBO&CSBIJNJ w $BOBEBͷ0OUBSJPେֶ w ౤ߘ w ݄ʹ"SYJWʹ౤ߘ͞Εͨ΋ͷ ॻࢽ৘ใ !2

w طଘͷ*OQBJOUJOHख๏͸ߏ଄తͳ৘ใΛա౓ͳεϜʔδϯάͰࣦͬͯ͠·͍ͬͯΔ w ը૾ͷ&EHF.BQΛ*NBHF*OQBJOUJOHʹར༻ͨ͠ख๏ΛఏҊ w $FMFC" 1MBDFT 1BSJT4USFF7JFXEBUBTFUʹ͓͍ͯɼ4P5" ֓ཁ !3

w લఏ w ਓؒ͸ը૾Λݟͨͱ͖ʹɼҰ෦͚ͩҟͳΔ෦෼͕͋ΔʢҰ؏ੑ͕อ͍ͯͯͳ͍෦෼͕͋Δʣͱɼ ͦ͜ʹҙ͕ࣝूதͯ͠͠·͏ w طଘख๏ͷ໰୊఺ w *OQBJOUJOHͨ͠෦෼͚͕ͩɼա౓ʹฏ׈Խ͞Εͯɼ΅΍͚ͯ͠·͍ͬͯΔ w ˠԿͱ͔͍ͨ͠ w Ծઆ w -JOFT'JSTU $PMPS/FYU w ը૾ͷ&EHF.BQΛར༻͢Ε͹ɼ͸͖ͬΓͨ͠ը૾͕࠶ߏ੒Ͱ͖ΔͷͰ͸ͳ͍͔ʁ Ϟνϕʔγϣϯ !4

w &EHF.BQΛར༻ͨ͠*NBHF*OQBJOUJOHख๏ΛఏҊ w ख๏ͷ֓ཁ w &EHF(FOFSBUPS&EHF.BQʹ͓͚ΔϚεΫ෦෼Λ*OQBJOUJOH w *NBHF$PNQMFUJPO/FUXPSL*OQBJOUJOH͞Εͨ&EHF.BQΛར༻ͯ͠ɼݩը૾Λ*OQBJOUJOH ఏҊख๏ !5 ΋ͬͱΘ͔Γ΍͍͢ը૾ʹมߋ͢Δ

w ը૾ͷܽଛ෦෼ΛࣗવʹຒΊΔλεΫ *NBHF*OQBJOUJOHͱ͸ !6

w ֦ࢄϕʔε %J⒎VTJPOCBTFE w ύονϕʔε 1BUDICBTFE w ֶशϕʔε -FBSOJOHCBTFE w ωοτϫʔΫͷ޻෉ طଘख๏ !7

w ಛ௃ w पลϐΫηϧͷՃॏฏۉΛͱͬͨΓ͢Δख๏ w 0QFO$7ʹσϑΥϧτͰ࣮૷͞Ε͍ͯΔΑ͏ͳख๏ w ݶք w ը૾ͷҙຯతͳ৘ใΛߟྀͰ͖ͳ͍ w େ͖ͳൣғΛ࠶ߏ੒͢Δ͜ͱʹ͸ݶք͕͋Δ ֦ࢄϕʔε %J⒎VTJPOCBTFE !8

w ಛ௃ w ϚεΫྖҬपลʹྨࣅ͍ͯ͠ΔྖҬΛಉ͡ը૾ͳ͍͔Βݕࡧ͠షΓ෇͚Δ w ϚεΫྖҬʹྨࣅ͢ΔྖҬ͕ը૾಺ʹଘࡏ͢Δ͜ͱ͕લఏ w ݶք w إը૾ʹ͓͍ͯޱΛ࠶ߏ੒͢ΔέʔεͷΑ͏ʹ
 ը૾ʹͳ͍΋ͷΛ࠶ߏ੒͢Δ͜ͱ͸Ͱ͖ͳ͍ w ग़ݱස౓͕௿͍΋ͷΛ࠶ߏ੒͕೉͍͠ w 1BUDI.BUDI w ྨࣅύονͷݕࡧΛϥϯμϜʹߦ͏ΞϧΰϦζϜΛఏҊ͠ɼ
 ܭࢉ࣌ؒΛѹ౗తʹ୹ॖͨ͠ ύονϕʔε !9 PatchMatch

w ಛ௃ w ("/ͷ(FOFSBUPSΛ*NBHF*OQBJOUJOH/FUXPSLʹֶͯ͠श͢Δ w ೖྗ͸ݩը૾ͱϚεΫը૾ɼ࠶ߏ੒ͨ͠΋ͷͱݩը૾Λ%JTDSJNJOBUPSʹ͔͚Δ w ݶք w ա౓ʹฏ׈Խ͞Εͯ΅΍͚ͨग़ྗʹͳͬͯ͠·͏ w (-$*$ w %JMBUFE$POWͭͷ(MPCBMͱ-PBDMͷ
 %JTDSJNJOBUPSʹΑͬͯߴਫ਼౓ͷ*OQBJOUJOHΛ࣮ݱ ֶशϕʔε !10 GLCIC

w ಛ௃ w جຊ͸ֶशϕʔεͷख๏Ͱɼ༷ʑͳ$7ܥͷλεΫͰ࢖ΘΕΔςΫχοΫΛ੝ΓࠐΜͩ w ݶք w ͦ΋ͦ΋$PBSTFSFTVMU͕ਖ਼͍͠ߏ଄Λ͍࣋ͬͯΔͱ͸
 ݶΒͳ͍ w ϚεΫͷܗ͕ݶΒΕΔ w $" XJUI$POUFYUVBM"UUFOUJPO w $PBSTFUP'JOFߏ଄ͷಋೖ w $PBSTF3FTVMUͱݩʑͷೖྗը૾ͱͷ
 ྨࣅ౓Λܭࢉ͢Δ"UUFOUJPOػߏΛಋೖ ωοτϫʔΫͷ޻෉ !11 CA (Generative Image Inpainting with Contextual Attention)

w ( &EHF(FOFSBUPS ͱ( *NBHF$PNQMFUJPO/FUXPSL Ͱߏ੒͞ΕΔ w &EHF(FOFSBUFS ( w ೖྗάϨʔεέʔϧը૾ɼ&EHFը૾ɼϚεΫը૾ w ग़ྗ࠶ߏ੒͞Εͨ&EHFը૾ w *NBHF$PNQMFUJPO/FUXPSL ( w ೖྗݩը૾ɼ࠶ߏ੒͞Εͨ&EHFը૾ w ग़ྗ࠶ߏ੒ը૾ &EHF$POOFDU !12 Edge Generater Image Completion Net ֶश • G1ͱG2ΛɼD1, D2Λ༻͍ͯ
 ͦΕͧΕผʑʹֶश • D1Λൈ͍ͯɼG1ͱG2Λ
 End-to-EndͰֶश

w (FOFSBUFS w %JMBUFE3FTJEVBM#MPDLT w ֤૚$POWPMVUJPO4QFDUSBM/PSN
 *OTUBODF/PSN3F-6 w -PTT w "EWFSTBSJBM-PTT'FBUVSF.BUDIJOH-PTT w "EWFSTBSJBM-PTTʢ4/("/ w 'FBUVSF.BUDIJOH-PTT &EHF(FOFSBUPS !13

w (FOFSBUFS w ϊΠζ͔ΒԿΒ͔ͷը૾Λੜ੒͢Δ w %JTDSJNJOBUFS w ੜ੒͞Εͨը૾ͱσʔληοτʹ͋ΔϦΞϧ ͳը૾ΛݟͯɼͲͪΒ͕ຊ෺ͷը૾͔Λݟ෼ ͚Δ w Πϝʔδ w %JTDSJNJOBUFSͷݟ෼͚Δೳྗ͕ߴ·Δ w (FOFSBUFS͸ݟ෼͚Δೳྗ͕ߴ·ͬͨ %JTDSJNJOBUFSΛͩ·͢Α͏ʹֶश͢Δ w ݁Ռతʹ(FOFSBUFS͕ຊ෺ͷΑ͏ͳը૾Λੜ ੒Ͱ͖ΔΑ͏ʹͳΔ "EWFSTBSJBM-PTT !14

w ೖྗը૾ͱɼೖྗը૾ʹϚεΫΛ͔͚ͨը૾ Λ༻ҙ w (FOFSBUFS w ϚεΫΛؚΉೖྗը૾͔ΒϚεΫΛ࠶ߏ੒ ͨ͠ը૾Λੜ੒ w %JTDSJNJOBUFS w ݩʑͷೖྗը૾ͱϚεΫ෦෼͕࠶ߏ੒͞Ε ͨೖྗը૾Λࣝผ͢ w ໨తؔ਺ w ɹ w $$PNQMFUJPO/FUXPSL (FOFSBUFS 
 %%JTDSJNJOBUFS ..BTL *NBHF*OQBJOUJOHʹ͓͚Δఢରతֶश !15

&EHF(FOFSBUFSͷ"EWFSTBSJBM-PTT !16 ϚεΫ͕͔͔Δલͷനࠇը૾ ࠶ߏ੒͞ΕͨΤοδը૾ ݩը૾͔Β࡞੒ͨ͠Τοδը૾

w %JTDSJNJOBUFSͷதؒ૚ͷ'FBUVSF.BQൺֱ͢Δ w ࠶ߏ੒ը૾Λೖྗͨ࣌͠ͷ%ͷ'FBUVSF.BQ w ݩը૾Λೖྗͨ࣌͠ͷ%ͷ'FBUVSF.BQΛൺֱ w ֤'FBUVSF.BQͷ--PTTͷظ଴஋Λ࠷খԽ͢Δ w 1FSDFQUVBM-PTTʹࣅ͍ͯΔ͕ɼࠓճ͸&EHFը૾Ͱ͋ΔͨΊɼ1FSDFQUVBM-PTTʹΑΔֶश͸೉ ͍ͨ͠Ίɼ'FBUVSF.BUDIJOH-PTTʹͨ͠ 'FBUVSF.BUDIJOH-PTT !17 ੜ੒ը૾ͷ Feature Map ϦΞϧը૾ͷ Feature Map ֤Feature Mapͷ ࠩ෼ͷظ଴஋

w (FOFSBUFS͸&EHF(FOFSBUFSͱಉ͡ w -PTT w --PTT"EWFSTBSJBM-PTT1FSDFQUVBM-PTT4UZMF-PTT w 1FSDFQUVBM-PTT w 4UZMF-PTT w "EWFSTBSJBM-PTT XJUIPVU4/ w -PTTؔ਺ʹΑΔ੍໿͕ଟ͍ͨΊɼ4/ΛೖΕΔͱֶश͕ۃ୺ʹ஗͘ͳΔͨΊɼൈ͍ͨ *NBHF$PNQMFUJPO/FUXPSL !18

w ୯७ͳੜ੒݁ՌͱϦΞϧը૾ͷ݁Ռͷࠩ෼ΛଌΔͷ Ͱ͸ͳ͘ɼੜ੒݁ՌͱϦΞϧը૾͕ྨࣅͨ͠ಛ௃ྔ Λ࣋ͭΑ͏ʹֶश͢Δ ը૾ͷίϯςϯπ͚ۙͮΔʣ w ੜ੒ը૾ͱϦΞϧը૾Λ-PTT/FUXPSL 7(( QFSUBJOFECZ*NBHF/FUͳͲʣʹೖྗͨ͠ࡍͷ 'FBUVSF.BQͷࠩ෼ w ɹ 1FSDFQUVBM-PTT !19 9 Perceptual Lossͷ࿦จͷਤ ੜ੒ը૾ͷ Feature Map ϦΞϧը૾ͷ Feature Map ֤Feature Mapͷ ࠩ෼ͷظ଴஋

w ੜ੒ը૾ͱϦΞϧը૾ͷελΠϧʢ৭ɼςΫενϟɼ ࢹ֮తͳύλʔϯʣ͕ࣅΔΑ͏ʹֶश w 1FSDFQUVBM-PTTͰར༻͢Δ'FBUVSF.BQͷϖΞʢੜ ੒ͱϦΞϧʣͷάϥϜϚτϦοΫεͷࠩ෼ w 4UZMF-PTT !20 9 Style Lossͷ࿦จͷਤ ੜ੒ը૾ͷ Feature Mapͷ Gram Matrix ϦΞϧը૾ͷ Feature Mapͷ Gram Matirx Gram Matrix νϟϯωϧcͱc’ʹ͓͚ΔFeature Mapͷཁૉ੮ͷ࿨

w &EHFͷݕग़ w ਖ਼ղσʔλͷ࡞੒΋ؚΊɼ$BOOZΤοδݕग़Λར༻ͨ͠ w ଞͷ&EHFݕग़ख๏Λ࢖༻ͨ࣌͠ͷਫ਼౓ൺֱ΋ߦͳ͍ͬͯΔ w .BTLը૾ w 3FHVMBS)PMFͱ*SSFHVMBSIPMFͲͪΒ΋࢖༻࣮ͯ͠ݧͨ͠ w ը૾σʔληοτ w $FMFC" 1MBDFT 1BSJT4USFFU7JFX w αΠζY ࣮ݧ !21 Regular Hole Irregular Hole CelebA Places2 Paris Street View

w ఆੑධՁ w *OQBJOUJOHͷ݁Ռͱ&EHFը૾ͷ*OQBJOUJOHͷ݁Ռ w (-$*$ $"ͷ݁Ռͱͷൺֱ w ఆྔධՁ w - 14/3 44*. '*% w ϢʔβʔධՁ w ը૾ΛݟͤͯϦΞϧ͔൱͔Λ౴͑Δςετ w ϦΞϧը૾ͱ࠶ߏ੒݁Ռͷը૾ΛݟͤͯͲͪΒ͕ϦΞϧ͔Λ౴͑Δςετ w &EHFΛར༻͢Δ͜ͱʹΑΔਫ਼౓޲্ʹؔ͢Δݕূ ࣮ݧ಺༰ !22

w ը૾ͷ޿ൣғʹϚεΫ͕͔͔͍ͬͯͯ΋ϦΞϧͳ࠶ߏ੒͕Մೳ w 4UZMF-PTTʹΑͬͯ$IFDLFSCPBSE໛༷͕ͳ͘ɼ៉ྷʹੜ੒͞Ε͍ͯΔɽ ఆੑධՁ !23 Original Input (Inpainted) Edge Inpainted Original Input (Inpainted) Edge Inpainted

w ଞͷख๏ΑΓ΋(5ʹ͍ۙ݁ՌΛಘΒΕͨɽ w &EHFͷ৘ใΛར༻͢Δ͜ͱͰɼ*OQBJOUJOHͷࡍʹ৭ͷ෼෍͚ͩΛֶश͢Ε͹ྑ͍ͨΊɼਫ਼౓͕ ߴ͍Β͍͠ ଞͷख๏ͱͷൺֱ !24 CA GLCIC EdgeͷGTΛ࢖༻ͯ͠ Inpainting

w 14/3ʢϐʔΫ৴߸ϊΠζൺʣ w ͭͷը૾ؒͷϊΠζൺ཰Λද͢ɽ w 44*.ʢ4USVDUVBM4JNJMBSJUZʣ w ըૉ஋ͷมԽɼίϯτϥετͷมԽɼߏ଄ͷมԽΛද͢ ఆྔධՁࢦඪ !25 PSNR = 10 ⇥ log10 MAX2 I MSE = 20 ⇥ log10 MAXI p MSE AAACU3ichVFNS8MwGE7r93Q69eglOBRPo52CXoSpCHpQ5sd0sM6RZukWl36YvBVG6X8UwYN/xIsHTesO6gRfCDw8H+TNEzcSXIFlvRrmxOTU9MzsXGF+obi4VFpeuVFhLClr0FCEsukSxQQPWAM4CNaMJCO+K9itOzjK9NtHJhUPg2sYRqztk17APU4JaKpTuq9fnV/iTbyPbQs7wH2mcOJIH4uwl+acJwlNzg6ad9XOaZqcXR2n2HEKWaL6TyLzO+pBQp5KO6WyVbHywePAHoEyGk29U3p2uiGNfRYAFUSplm1F0E6IBE4FSwtOrFhE6ID0WEvDgOhV2kneSYo3NNPFXij1CQDn7PdEQnylhr6rnT6BvvqtZeRfWisGb6+d8CCKgQX06yIvFhhCnBWMu1wyCmKoAaGS610x7RNdCehvKOgS7N9PHgc31Yq9Xale7JRrh6M6ZtEaWkdbyEa7qIZOUB01EEVP6A19GMh4Md5N05z8sprGKLOKfoxZ/AS+e69x SSIM = (2µxµy + c1)(2 xy + c2) (µ2 x + µ2 y + c1)( 2 x + 2 y + c2) AAACW3icbVFdS+swGE7r15xfU/HKmxeHUBFGWwW9OTD05ngheNCpsM6SZukMJm1J0sNK2Z/0ynNx/srBdN3g+PFCypPngyRPo4wzpV33j2UvLC4trzRWm2vrG5tbre2de5XmktAeSXkqHyOsKGcJ7WmmOX3MJMUi4vQherms9IffVCqWJne6yOhA4FHCYkawNlTYkre3V9fwA4JYYlI6fiDycAzVt4BjIKF3BA74ECg2Ejgsx8Wkov2jSekY05NvzMdQo3nAqc1zbb6pZZMMW223404HvgJvBtpoNjdh6zUYpiQXNNGEY6X6npvpQYmlZoTTSTPIFc0wecEj2jcwwYKqQTntZgKHhhlCnEqzEg1T9v9EiYVShYiMU2D9rD5rFfmd1s91fD4oWZLlmiakPijOOegUqqJhyCQlmhcGYCKZuSuQZ2xa1uZ3NE0J3ucnfwX3fsc76fi/Ttvdi1kdDbSPDpCDPHSGuugnukE9RNAb+metWA3rr71gN+312mpbs8wu+jD23js2Ya63

w '*% 'SFDIFU*ODFQUJPO%JTUBODF w ը૾ؒͷಛ௃Λൺֱ͢ΔείΞʢϐΫηϧґଘ͡Όͳ͍ʣ w *ODFQUJPO7ʹϦΞϧը૾Λೖྗͨ࣌͠ͱੜ੒͞Εͨը૾Λೖྗͨ࣌͠ͷதؒ૚ͷϕΫτϧIΛ ൺֱ w ֤ϕΫτϧI͕ଟ࣍ݩͷΨ΢ε෼෍ʹै͍ͬͯΔͱԾఆ͠ɼ෼෍ؒͷ8BTTFSTUFJOڑ཭ΛଌΔ ఆྔධՁࢦඪ !26

w ը૾ʹର͢ΔϚεΫͷൺ཰Λม͑ͳ͕Βɼ֤ ࢦඪΛଌఆ w ͍͍ͩͨউͬͯΔ w --PTT͕1DPOWʹෛ͚͍ͯΔͷ͸ଌఆʹެฏ ͕͞ͳ͍ͨΊͩͱओு w 1DPOW͸ஶऀ࣮૷͕ެ։͞Ε͍ͯͳ͔ͬͨͬͨ Ίɼ࿦จ಺ʹ͋Δ஋ͷΈ ఆྔධՁͷ݁Ռ !27

w "'$ϥϯμϜͰબ୒͞Εͨը૾ΛݟͤͯɼϦΞϧ͔ϦΞϧ͡Όͳ͍͔౴͑Δ w +/%ϦΞϧը૾ͱੜ੒ը૾ΛݟͤͯɼͲ͕ͬͪϦΞϧ͔౴͑ͤ͞Δ w ݁Ռ w ैདྷख๏ʹൺ΂ͯɼ͔ͳΓείΞ͸ߴ͘ͳ͍ͬͯΔ w ɹɹɹɹɹɹɹɹɹɹɹɹɹɹɹɹɹɹɹΛߟ͑Δͱɼ݁ߏߴ͍ ϢʔβʔධՁ !28

w &EHF͋Γͱͳ͠Ͱಉ͡ωοτϫʔΫͰֶशͨ݁͠Ռ w &EHFແ͠ͷ৔߹͸ɼ*NBHF$PNQMFUJPO/FUXPSLͷํ͚ͩΛֶशͨ͠ w &EHF͕ޮ͍͍ͯΔ͜ͱ͕Θ͔ͬͨ &EHFͷޮՌ !29

w $BOOZ&EHFݕग़ͷΨ΢γΞϯϑΟϧλʔͷඪ४ภࠩ М Λ͍ΖΜͳ஋ʹมԽͤͨ͞ w М͕௿͍ͱ͋·ΓTNPPUIJOH͞Εͳ͍ͨΊࡉ͔͍&EHF·Ͱݕग़͞ΕΔ w ߴ͍ͱٯʹݕग़͞Εͳ͍ &EHFͷݕग़ํ๏ʹΑΔ*OQBJOUJOH΁ͷӨڹ !30

w $BOOZʹΑΔΤοδݕग़ͱ//ϕʔεͷ)&%ͱ͍͏ख๏Λ૊Έ߹Θͤͨ w )&%͸&EHF͝ͱʹڧ౓΍ଠ͞΋ग़Δ w Ծઆ w $BOOZͱͷཁૉ੮ΛͱΕ͹ɼΑΓखॻ͖ͬΆ͍&EHFը૾͕ੜ੒Ͱ͖Δ w खॻ͖ͬΆ͍΄Ͳ͏·͘*OQBJOUͰ͖ΔͷͰ͸ w ݁Ռແཧͩͬͨ ଞͷݕग़ख๏Λࢼͨ͠ !31

ΞϓϦέʔγϣϯ !32 ยํΛผͷը૾ͷEdgeʹͯ͠ੜ੒ ෆཁͳΦϒδΣΫτΛআڈ͢ΔͨΊʹ࢖༻

w ը૾ͷ&EHF৘ใΛར༻͢Δ͜ͱͰɼࣗવͳ࠶ߏ੒Λ࣮ݱͨ͠ w ैདྷख๏ΑΓ΋ߴ͍ਫ਼౓Λୡ੒ͨ͠ w &EHF͕*OQBJOUJOHʹޮ͍͍ͯΔ͜ͱ͕Θ͔ͬͨ w &EHF(FOFSBUFS͕͏·͘࠶ߏ੒Ͱ͖ͳ͍ͱ͍͚ͳ͍ͷͰɼͦͷվྑ͕ඞཁ w ߴղ૾౓ͷը૾ʹରͯ͠͸&EHFͷ࠶ߏ੒͕೉͍͠ ·ͱΊ !33