Upgrade to Pro — share decks privately, control downloads, hide ads and more …

[Paper Summary] Image Inpainting for Irregular ...

[Paper Summary] Image Inpainting for Irregular Holes Using Partial Convolution

This slide is the summary of "Image Inpainting for Irregular Holes Using Partial Convolution".
Original Paper is http://openaccess.thecvf.com/content_ECCV_2018/papers/Guilin_Liu_Image_Inpainting_for_ECCV_2018_paper.pdf .
My Re-implementation is here ( https://github.com/tanimutomo/partialconv ).

tanimutomo

May 21, 2019
Tweet

More Decks by tanimutomo

Other Decks in Research

Transcript

  1. 2019/05/21 Image Inpainting 
 for Irregular Holes 
 using Partial

    Convolution Tomoki Tanimura, B4 d-hacks, Jin Nakazawa Lab, Keio University
  2. 2019/05/21 Information • Title: • Image Inpainting for Irregular Holes

    Using Partial Convolution • Authors: • Guilin Liu Fitsum A. Reda Kevin J. Shih Ting-Chun Wang Andrew Tao Bryan Catanzaro • Affiliation: • NVIDIA Corporation • Accepted: • ECCV2018 !2
  3. 2019/05/21 Overview • Propose a new image inpainting method •

    Enable to deal with the irregular holes • Predict the hole pixels based on valid (no-hole) pixels • Outperforms other methods for irregular holes !3
  4. 2019/05/21 Background • Existing Methods • Initialization • Hole pixels

    are initialized by the mean of the entire pixels • => blurriness and color discrepancy • Rely on Post-process • Reduce such blurriness • But, Expensive or may fail • Our Method (Inpainting with Partial Conv) • In partial convolution, the hole pixels is inpainted based on the valid (non hole) pixels • Hole pixels are inpainted gradually as well as mask become small gradually • Initialization and Post-process are NOT needed !4
  5. 2019/05/21 Inpainting Network • Base Network is the U-Net-like architecture

    • Typical convolution is replaced with the Partial convolution !5
  6. 2019/05/21 Partial Convolutional Layer • Partial Convolution + Mask Update

    function • Partial Convolution • • Mask Update function • !6
  7. 2019/05/21 Loss function • Definition • Iin: Input Image /

    M(0 for holes): mask image / Iout: predicted image • Igt: Grand Truth Image / Icomp: Iout with non-hole’GT • Total Loss function • • Valid Loss • • Hole Loss • • Total Variation Loss • !8
  8. 2019/05/21 Perceptual Loss • L1 distance between the following image’s

    feature maps • the generated image • the ground truth image • Feature maps is created by VGG16 !10 VGG16
  9. 2019/05/21 Style Loss • L1 distance between the gram matrices

    of the feature maps • The feature maps is same as the one used in perceptual loss • Gram Matrix !11
  10. 2019/05/21 User Study • Which result is more realistic than

    the other? • 50% means two methods are equal !14
  11. 2019/05/21 Conclusion • Propose the Partial Convolution (in other paper)

    and Mask update mechanism • Enable to deal with the mask which is any shape, size, location • This method is weak • when the image has sparse structure • when the mask has large hole (as well as all other methods) !15
  12. 2019/05/21 Difficulty in Implementation • Data is too huge (Places2

    - 512px is 0.5TB) • Training time is too crazy long (10 days described in the paper) • Whether the gradient of VGG network’s parameters should be turned on • Where the VGG’s normalization should be applied • How to calculate the Total Variation Loss • How to update the mask for partial con • How to create the irregular mask !16