2019/05/21 Image Inpainting 
 for Irregular Holes 
 using Partial Convolution Tomoki Tanimura, B4 d-hacks, Jin Nakazawa Lab, Keio University

2019/05/21 Information • Title: • Image Inpainting for Irregular Holes Using Partial Convolution • Authors: • Guilin Liu Fitsum A. Reda Kevin J. Shih Ting-Chun Wang Andrew Tao Bryan Catanzaro • Affiliation: • NVIDIA Corporation • Accepted: • ECCV2018 !2

2019/05/21 Overview • Propose a new image inpainting method • Enable to deal with the irregular holes • Predict the hole pixels based on valid (no-hole) pixels • Outperforms other methods for irregular holes !3

2019/05/21 Background • Existing Methods • Initialization • Hole pixels are initialized by the mean of the entire pixels • => blurriness and color discrepancy • Rely on Post-process • Reduce such blurriness • But, Expensive or may fail • Our Method (Inpainting with Partial Conv) • In partial convolution, the hole pixels is inpainted based on the valid (non hole) pixels • Hole pixels are inpainted gradually as well as mask become small gradually • Initialization and Post-process are NOT needed !4

2019/05/21 Inpainting Network • Base Network is the U-Net-like architecture • Typical convolution is replaced with the Partial convolution !5

2019/05/21 Partial Convolutional Layer • Partial Convolution + Mask Update function • Partial Convolution • • Mask Update function • !6

2019/05/21 Partial Convolutional Layer !7 Next Input Renormalization

2019/05/21 Loss function • Definition • Iin: Input Image / M(0 for holes): mask image / Iout: predicted image • Igt: Grand Truth Image / Icomp: Iout with non-hole’GT • Total Loss function • • Valid Loss • • Hole Loss • • Total Variation Loss • !8

2019/05/21 Loss function • Perceptual Loss • • Style Loss • !9 VGG16

2019/05/21 Perceptual Loss • L1 distance between the following image’s feature maps • the generated image • the ground truth image • Feature maps is created by VGG16 !10 VGG16

2019/05/21 Style Loss • L1 distance between the gram matrices of the feature maps • The feature maps is same as the one used in perceptual loss • Gram Matrix !11

2019/05/21 Qualitative Comparison !12

2019/05/21 Quantitative Comparison !13 Score (Method) Hole ratio

2019/05/21 User Study • Which result is more realistic than the other? • 50% means two methods are equal !14

2019/05/21 Conclusion • Propose the Partial Convolution (in other paper) and Mask update mechanism • Enable to deal with the mask which is any shape, size, location • This method is weak • when the image has sparse structure • when the mask has large hole (as well as all other methods) !15

2019/05/21 Difficulty in Implementation • Data is too huge (Places2 - 512px is 0.5TB) • Training time is too crazy long (10 days described in the paper) • Whether the gradient of VGG network’s parameters should be turned on • Where the VGG’s normalization should be applied • How to calculate the Total Variation Loss • How to update the mask for partial con • How to create the irregular mask !16