CollaGAN: Collaborative GAN for Missing Image Data Imputation

Oral Presentation at CVPR 2019, Long Beach, CA, June 11th, 2019.

Transcript

1. CollaGAN: Collaborative GAN for Missing Image Data Imputation Dongwook Lee and Junyoung Kim, Won-Jin Moon, Jong Chul Ye Bio Imaging Signal Processing and Learning lab. KAIST KONKUK UNIVERSITY MEDICAL CENTER
2. Motivation : Missing data problem • Widespread problem in research fields (dealing with big data) Poster #135 - CollaGAN: Collaborative GAN for Missing Image Data Imputation (Tues, June 18, 15:20-18:00, Poster1.2, Synthesis) [1] C. Chen et al. 2014 CVPR [2] C. Shang et al. 2017, IEEE Int Conf Big Data. Pose View Example of Missing data problems in Computer vision[1] and Medical data[2] Age Gender Heart rate Subjects Obesity …
3. Motivation : Missing data problem Poster #135 - CollaGAN: Collaborative GAN for Missing Image Data Imputation (Tues, June 18, 15:20-18:00, Poster1.2, Synthesis) • Consequences of Missing data • Introduce bias/error because of incomplete data • What is ‘imputation’? • Replace missing data with substituted values è Algorithms for the numerical data Extend the problem to Missing ‘Image’ Imputation on multi-domain. Matrix Completion (Mazumder et al., 2010) Denoising Auto-Encoder (Vincent et al., 2008, Gondora et al. 2017) GAIN, Generative Adversarial Imputation Nets (Yoon et al., 2018)
4. Motivation : Missing ‘image’ imputation • Needs #1 – medical imaging • Multiple contrast images for cancer diagnosis[1] • Missing Contrast problem • Needs #2 – illumination images • Facial images from various angles of illumination • Finding the specific illumination (standard) Figure. Example for missing images in medical imaging Poster #135 - CollaGAN: Collaborative GAN for Missing Image Data Imputation (Tues, June 18, 15:20-18:00, Poster1.2, Synthesis) [1] A. Drevelegas et al. 2011, Imaging modalities in brain tumors Figure. Example for missing images in illumination database T1w T2w T1-FLAIR T2-FLAIR -90º -45º 0º 45º 90º
5. Motivation : Missing ‘image’ imputation • Proposed Missing image imputation technique called Collaborative Generative Adversarial Network (CollaGAN) Poster #135 - CollaGAN: Collaborative GAN for Missing Image Data Imputation (Tues, June 18, 15:20-18:00, Poster1.2, Synthesis) Figure. Example for missing illumination imputation Figure. Example for missing MR contrast imputation
6. Related work • Background knowledge for CollaGAN • Generative Adversarial Networks (GAN) • CycleGAN : two-domain image-to-image translation • StarGAN : multi-domain image-to-image translation Poster #135 - CollaGAN: Collaborative GAN for Missing Image Data Imputation (Tues, June 18, 15:20-18:00, Poster1.2, Synthesis)
7. Related work • Generative Adversarial Networks[1] Database Generator, G Discriminator, D Loss Real Sampler, z Fake min $ max ' ( ), + = -.~01232(.) log ) 9 + -;~0<(;) [log(1 − ) + @ )] [1] I. Goodfellow et al. 2014, NIPS Poster #135 - CollaGAN: Collaborative GAN for Missing Image Data Imputation (Tues, June 18, 15:20-18:00, Poster1.2, Synthesis)
8. Related work • CycleGAN[1] , DiscoGAN[2] [1] Zhu, JY et al. 2017, ICCV [2] Kim, T et al. 2017, ICML G_cat G_dog Real D_cat Loss D_dog Loss Real Real Real Fake Fake min $ max ' ( )* , ), , -* , -, = /0~23454(0) log )* ;* + /=~23454(=) [log(1 − )* -* A )] +/=~23454(=) log ), A, + /0~23454(0) [log(1 − ), -, ; )] Poster #135 - CollaGAN: Collaborative GAN for Missing Image Data Imputation (Tues, June 18, 15:20-18:00, Poster1.2, Synthesis)
9. Related work • CycleGAN[1] , DiscoGAN[2] à Cycle Consistency [1] Zhu, JY et al. 2017, ICCV [2] Kim, T et al. 2017, ICML G_cat Fake G_dog Rec. Fake Real Real G_dog G_cat Rec. !"#"$ !"#"% !"#" &' , &) = +,~./010(,) &) &' 4 − 4 6 ++#~./010(#) &' &) 7 − 7 6 Poster #135 - CollaGAN: Collaborative GAN for Missing Image Data Imputation (Tues, June 18, 15:20-18:00, Poster1.2, Synthesis)
10. Related work: Expansion on Multi-domain • CycleGAN[1], DiscoGAN[2] • Problem on scalability • Not shared feature learning • StarGAN[3] & RadialGAN[4] • Deals with multiple domain • Shared learning on latent space • Single input to single output • Cannot handle the multiple inputs [1] Zhu, JY et al. 2017, ICCV [2] Kim, T et al. 2017, ICML, [3] Choi, Y et al. 2017, CVPR [4] Yoon, J et al. 2018, ICML #Domain #G #D 2 2 2 3 6 3 ! "# 4 ⋮ ⋮ ⋮ N N(N-1) N Figure. Example of 4-distinct Cross-domain model[3] and numbers of G and D for N cross-domain models. Figure. Concepts of StarGAN[3] and RadialGAN[4] Poster #135 - CollaGAN: Collaborative GAN for Missing Image Data Imputation (Tues, June 18, 15:20-18:00, Poster1.2, Synthesis)
11. Related work: Expansion on Multi-domain Poster #135 - CollaGAN: Collaborative GAN for Missing Image Data Imputation (Tues, June 18, 15:20-18:00, Poster1.2, Synthesis) • CycleGAN & StarGAN • Try to approach the problem by image-to-image translation way. • Limitations: Scalability, Single input to single output CycleGAN[1] StarGAN[2] [1] Zhu, JY et al. 2017, ICCV [2] Choi, Y et al. 2017, CVPR
12. Related work: Expansion on Multi-domain Poster #135 - CollaGAN: Collaborative GAN for Missing Image Data Imputation (Tues, June 18, 15:20-18:00, Poster1.2, Synthesis) • CycleGAN & StarGAN • Try to approach the problem by image-to-image translation way. • Limitations: Scalability, Single input to single output [1] Zhu, JY et al. 2017, ICCV [2] Choi, Y et al. 2017, CVPR CollaGAN
13. Proposed method Poster #135 - CollaGAN: Collaborative GAN for Missing Image Data Imputation (Tues, June 18, 15:20-18:00, Poster1.2, Synthesis) works on Various number of inputs (with a help of input dropout) # Inputs = 2 # Inputs = 1 CollaGAN
14. Proposed method •CollaGAN • Multi domain Image imputation method • Using Multiple inputs • Using Single Generator • Using Adversarial Model • Using Single Discriminator • Using Multiple Cycle Consistency Multiple Inputs Single Generator Single Discriminator Adversarial model Multiple Cycle Consistency
15. Proposed method: CollaGAN • Imputation using Multiple inputs Multiple Inputs Single Generator Single Discriminator Adversarial model Multiple Cycle Consistency Input images Input images Target domain Input images Fake image G ! "# = %( "' , ") , "* ; ,)
16. Proposed method: CollaGAN • Mask vector for Single Generator Multiple Inputs Single Generator Single Discriminator Adversarial model Multiple Cycle Consistency Input images Input images Target domain Input images Fake image G Input : images + mask vector + Mask Vector = 2D one-hot vector ! "# = %( "' , ") , "* ; ,)
17. Proposed method: CollaGAN • Adversarial model using Dgan Multiple Inputs Single Generator Single Discriminator Adversarial model Multiple Cycle Consistency Input images Input images Target domain Input images Fake image G Fake image Real image Real / Fake (1) (2) D Dgan (1),(2) ℒ"#$%& $ = () * [(-./0 1 2 − 1 )6] ℒ"#$%& 8 = (* [(-./0 2 − 1)6] + () * [(-./0 1 2 )6] [LSGAN] X. Mao et al. 2017, ICCV
18. Proposed method: CollaGAN • Dclsf for Single Discriminator Multiple Inputs Single Generator Single Discriminator Adversarial model Multiple Cycle Consistency Input images Input images Target domain Input images Fake image G Fake image Real image Real / Fake Domain classification (1) (2) D Dgan Dclsf (1),(2) (2) ℒ"#$% & = ()* [−-./(1"#$% 2; 45 )] ℒ"#$% 8 = (9 )*|* [−-./(1"#$% 2; 9 45 )]
19. Proposed method: CollaGAN • Multiple Cycle Consistency Loss Multiple Inputs Single Generator Single Discriminator Adversarial model Multiple Cycle Consistency Input images Input images Target domain Input images New Input images Original domain New Input images Cyclic input images New Input images Original domain New Input images Cyclic input images Fake image G Reconstru cted image Reconstru cted image Reconstruct ed image G New Input images Original domain New Input images Cyclic input images Fake image Real image Real / Fake Domain classification (1) (2) D Dgan Dclsf (1),(2) (2) ! "#|% = '( ") , + "% , ", ; .) ! ",|% = '( ") , + "% , ", ; 0) ℒ2##,% = ") − ! ")|% 4 + "# −! "#|% 4 + ", −! ",|% 4 ! ")|% = '( + "% , "# , ", ; 6)
20. Multi domain image imputation Tasks • Task#1 : Synthetic MR contrast image imputation [1] R Gross et al. 2010, Image and Vision Computing [2] O. Langner, 2010, Cognition and emotion Poster #135 - CollaGAN: Collaborative GAN for Missing Image Data Imputation (Tues, June 18, 15:20-18:00, Poster1.2, Synthesis) • Task #2 : Illumination image imputation (CMU Multi-PIE[1]) • Task #3 : Facial expression image imputation (Radboud Face Data[2])
21. Results #1: MR contrast imputation T1 FLAIR T2 weighted T2 FLAIR T2 FLAIR* Proposed CycleGAN[1] StarGAN[2] Target [1] Zhu, JY et al. 2017, ICCV [3] Choi, Y et al. 2017, CVPR Poster #135 - CollaGAN: Collaborative GAN for Missing Image Data Imputation (Tues, June 18, 15:20-18:00, Poster1.2, Synthesis)
22. Results #2: illumination imputation -90 º -45 º 45 º 90 º Target Proposed StarGAN[2] CycleGAN[1] 0 º CollaGAN reconstructs not only the overall illumination but also the details of illumination from the nose, chin and cheeks. [1] Zhu, JY et al. 2017, ICCV [3] Choi, Y et al. 2017, CVPR Poster #135 - CollaGAN: Collaborative GAN for Missing Image Data Imputation (Tues, June 18, 15:20-18:00, Poster1.2, Synthesis)
23. Results #3: Facial expression imputation CollaGAN reconstructs more natural and expressive facial expressions compared to CycleGAN and StarGAN Neutral Angry Contemptuous Disgusted Fearful Happy Sad Surprised Target Proposed StarGAN[2] CycleGAN[1] [1] Zhu, JY et al. 2017, ICCV [3] Choi, Y et al. 2017, CVPR Poster #135 - CollaGAN: Collaborative GAN for Missing Image Data Imputation (Tues, June 18, 15:20-18:00, Poster1.2, Synthesis)
24. Results #3: Facial expression imputation • Redundancy of multiple inputs • Some of inputs may not contribute Poster #135 - CollaGAN: Collaborative GAN for Missing Image Data Imputation (Tues, June 18, 15:20-18:00, Poster1.2, Synthesis)
25. Results #3: Facial expression imputation • Redundancy of multiple inputs • Some of inputs may not contribute Poster #135 - CollaGAN: Collaborative GAN for Missing Image Data Imputation (Tues, June 18, 15:20-18:00, Poster1.2, Synthesis) • To achieve the “collaborative” learning • Apply random nulling on N inputs (Missing N) : input dropout
26. Results #3: Facial expression imputation Poster #135 - CollaGAN: Collaborative GAN for Missing Image Data Imputation (Tues, June 18, 15:20-18:00, Poster1.2, Synthesis) Missing 5 Missing 1 Missing 2 Missing 3 Missing 4 Missing 6 Target Missing 7 w/o dropout dropout Sad Amount of input information • Robust performance on various missing # • More input information à Better reconstruction quality
27. Summary • CollaGAN for Multi-domain missing image imputation • Uses multiple input images • Multiple cycle consistency • Multi-domain : Single G & Single D • Synergistically combining the information from the available data with a help of input dropout. • Produces images with higher visual quality compared to the existing methods. Poster #135 - CollaGAN: Collaborative GAN for Missing Image Data Imputation (Tues, June 18, 15:20-18:00, Poster1.2, Synthesis)
28. CollaGAN: Collaborative GAN for Missing Image Data Imputation Dongwook Lee, Junyoung Kim, Won-Jin Moon, Jong Chul Ye KONKUK UNIVERSITY MEDICAL CENTER Thank you for your attention. Poster #135 (Tues, June 18, 15:20-18:00, Poster1.2) Come over to my poster for more discussions!