Upgrade to Pro — share decks privately, control downloads, hide ads and more …

CollaGAN: Collaborative GAN for Missing Image Data Imputation

CollaGAN: Collaborative GAN for Missing Image Data Imputation

Oral Presentation at CVPR 2019, Long Beach, CA, June 11th, 2019.

A3d61bc22cd700a92e7d4136a4d29e8f?s=128

Jong Chul Ye

June 11, 2019
Tweet

Transcript

  1. CollaGAN: Collaborative GAN for Missing Image Data Imputation Dongwook Lee

    and Junyoung Kim, Won-Jin Moon, Jong Chul Ye Bio Imaging Signal Processing and Learning lab. KAIST KONKUK UNIVERSITY MEDICAL CENTER
  2. Motivation : Missing data problem • Widespread problem in research

    fields (dealing with big data) Poster #135 - CollaGAN: Collaborative GAN for Missing Image Data Imputation (Tues, June 18, 15:20-18:00, Poster1.2, Synthesis) [1] C. Chen et al. 2014 CVPR [2] C. Shang et al. 2017, IEEE Int Conf Big Data. Pose View Example of Missing data problems in Computer vision[1] and Medical data[2] Age Gender Heart rate Subjects Obesity …
  3. Motivation : Missing data problem Poster #135 - CollaGAN: Collaborative

    GAN for Missing Image Data Imputation (Tues, June 18, 15:20-18:00, Poster1.2, Synthesis) • Consequences of Missing data • Introduce bias/error because of incomplete data • What is ‘imputation’? • Replace missing data with substituted values è Algorithms for the numerical data Extend the problem to Missing ‘Image’ Imputation on multi-domain. Matrix Completion (Mazumder et al., 2010) Denoising Auto-Encoder (Vincent et al., 2008, Gondora et al. 2017) GAIN, Generative Adversarial Imputation Nets (Yoon et al., 2018)
  4. Motivation : Missing ‘image’ imputation • Needs #1 – medical

    imaging • Multiple contrast images for cancer diagnosis[1] • Missing Contrast problem • Needs #2 – illumination images • Facial images from various angles of illumination • Finding the specific illumination (standard) Figure. Example for missing images in medical imaging Poster #135 - CollaGAN: Collaborative GAN for Missing Image Data Imputation (Tues, June 18, 15:20-18:00, Poster1.2, Synthesis) [1] A. Drevelegas et al. 2011, Imaging modalities in brain tumors Figure. Example for missing images in illumination database T1w T2w T1-FLAIR T2-FLAIR -90º -45º 0º 45º 90º
  5. Motivation : Missing ‘image’ imputation • Proposed Missing image imputation

    technique called Collaborative Generative Adversarial Network (CollaGAN) Poster #135 - CollaGAN: Collaborative GAN for Missing Image Data Imputation (Tues, June 18, 15:20-18:00, Poster1.2, Synthesis) Figure. Example for missing illumination imputation Figure. Example for missing MR contrast imputation
  6. Related work • Background knowledge for CollaGAN • Generative Adversarial

    Networks (GAN) • CycleGAN : two-domain image-to-image translation • StarGAN : multi-domain image-to-image translation Poster #135 - CollaGAN: Collaborative GAN for Missing Image Data Imputation (Tues, June 18, 15:20-18:00, Poster1.2, Synthesis)
  7. Related work • Generative Adversarial Networks[1] Database Generator, G Discriminator,

    D Loss Real Sampler, z Fake min $ max ' ( ), + = -.~01232(.) log ) 9 + -;~0<(;) [log(1 − ) + @ )] [1] I. Goodfellow et al. 2014, NIPS Poster #135 - CollaGAN: Collaborative GAN for Missing Image Data Imputation (Tues, June 18, 15:20-18:00, Poster1.2, Synthesis)
  8. Related work • CycleGAN[1] , DiscoGAN[2] [1] Zhu, JY et

    al. 2017, ICCV [2] Kim, T et al. 2017, ICML G_cat G_dog Real D_cat Loss D_dog Loss Real Real Real Fake Fake min $ max ' ( )* , ), , -* , -, = /0~23454(0) log )* ;* + /=~23454(=) [log(1 − )* -* A )] +/=~23454(=) log ), A, + /0~23454(0) [log(1 − ), -, ; )] Poster #135 - CollaGAN: Collaborative GAN for Missing Image Data Imputation (Tues, June 18, 15:20-18:00, Poster1.2, Synthesis)
  9. Related work • CycleGAN[1] , DiscoGAN[2] à Cycle Consistency [1]

    Zhu, JY et al. 2017, ICCV [2] Kim, T et al. 2017, ICML G_cat Fake G_dog Rec. Fake Real Real G_dog G_cat Rec. !"#"$ !"#"% !"#" &' , &) = +,~./010(,) &) &' 4 − 4 6 ++#~./010(#) &' &) 7 − 7 6 Poster #135 - CollaGAN: Collaborative GAN for Missing Image Data Imputation (Tues, June 18, 15:20-18:00, Poster1.2, Synthesis)
  10. Related work: Expansion on Multi-domain • CycleGAN[1], DiscoGAN[2] • Problem

    on scalability • Not shared feature learning • StarGAN[3] & RadialGAN[4] • Deals with multiple domain • Shared learning on latent space • Single input to single output • Cannot handle the multiple inputs [1] Zhu, JY et al. 2017, ICCV [2] Kim, T et al. 2017, ICML, [3] Choi, Y et al. 2017, CVPR [4] Yoon, J et al. 2018, ICML #Domain #G #D 2 2 2 3 6 3 ! "# 4 ⋮ ⋮ ⋮ N N(N-1) N Figure. Example of 4-distinct Cross-domain model[3] and numbers of G and D for N cross-domain models. Figure. Concepts of StarGAN[3] and RadialGAN[4] Poster #135 - CollaGAN: Collaborative GAN for Missing Image Data Imputation (Tues, June 18, 15:20-18:00, Poster1.2, Synthesis)
  11. Related work: Expansion on Multi-domain Poster #135 - CollaGAN: Collaborative

    GAN for Missing Image Data Imputation (Tues, June 18, 15:20-18:00, Poster1.2, Synthesis) • CycleGAN & StarGAN • Try to approach the problem by image-to-image translation way. • Limitations: Scalability, Single input to single output CycleGAN[1] StarGAN[2] [1] Zhu, JY et al. 2017, ICCV [2] Choi, Y et al. 2017, CVPR
  12. Related work: Expansion on Multi-domain Poster #135 - CollaGAN: Collaborative

    GAN for Missing Image Data Imputation (Tues, June 18, 15:20-18:00, Poster1.2, Synthesis) • CycleGAN & StarGAN • Try to approach the problem by image-to-image translation way. • Limitations: Scalability, Single input to single output [1] Zhu, JY et al. 2017, ICCV [2] Choi, Y et al. 2017, CVPR CollaGAN
  13. Proposed method Poster #135 - CollaGAN: Collaborative GAN for Missing

    Image Data Imputation (Tues, June 18, 15:20-18:00, Poster1.2, Synthesis) works on Various number of inputs (with a help of input dropout) # Inputs = 2 # Inputs = 1 CollaGAN
  14. Proposed method •CollaGAN • Multi domain Image imputation method •

    Using Multiple inputs • Using Single Generator • Using Adversarial Model • Using Single Discriminator • Using Multiple Cycle Consistency Multiple Inputs Single Generator Single Discriminator Adversarial model Multiple Cycle Consistency
  15. Proposed method: CollaGAN • Imputation using Multiple inputs Multiple Inputs

    Single Generator Single Discriminator Adversarial model Multiple Cycle Consistency Input images Input images Target domain Input images Fake image G ! "# = %( "' , ") , "* ; ,)
  16. Proposed method: CollaGAN • Mask vector for Single Generator Multiple

    Inputs Single Generator Single Discriminator Adversarial model Multiple Cycle Consistency Input images Input images Target domain Input images Fake image G Input : images + mask vector + Mask Vector = 2D one-hot vector ! "# = %( "' , ") , "* ; ,)
  17. Proposed method: CollaGAN • Adversarial model using Dgan Multiple Inputs

    Single Generator Single Discriminator Adversarial model Multiple Cycle Consistency Input images Input images Target domain Input images Fake image G Fake image Real image Real / Fake (1) (2) D Dgan (1),(2) ℒ"#$%& $ = () * [(-./0 1 2 − 1 )6] ℒ"#$%& 8 = (* [(-./0 2 − 1)6] + () * [(-./0 1 2 )6] [LSGAN] X. Mao et al. 2017, ICCV
  18. Proposed method: CollaGAN • Dclsf for Single Discriminator Multiple Inputs

    Single Generator Single Discriminator Adversarial model Multiple Cycle Consistency Input images Input images Target domain Input images Fake image G Fake image Real image Real / Fake Domain classification (1) (2) D Dgan Dclsf (1),(2) (2) ℒ"#$% & = ()* [−-./(1"#$% 2; 45 )] ℒ"#$% 8 = (9 )*|* [−-./(1"#$% 2; 9 45 )]
  19. Proposed method: CollaGAN • Multiple Cycle Consistency Loss Multiple Inputs

    Single Generator Single Discriminator Adversarial model Multiple Cycle Consistency Input images Input images Target domain Input images New Input images Original domain New Input images Cyclic input images New Input images Original domain New Input images Cyclic input images Fake image G Reconstru cted image Reconstru cted image Reconstruct ed image G New Input images Original domain New Input images Cyclic input images Fake image Real image Real / Fake Domain classification (1) (2) D Dgan Dclsf (1),(2) (2) ! "#|% = '( ") , + "% , ", ; .) ! ",|% = '( ") , + "% , ", ; 0) ℒ2##,% = ") − ! ")|% 4 + "# −! "#|% 4 + ", −! ",|% 4 ! ")|% = '( + "% , "# , ", ; 6)
  20. Multi domain image imputation Tasks • Task#1 : Synthetic MR

    contrast image imputation [1] R Gross et al. 2010, Image and Vision Computing [2] O. Langner, 2010, Cognition and emotion Poster #135 - CollaGAN: Collaborative GAN for Missing Image Data Imputation (Tues, June 18, 15:20-18:00, Poster1.2, Synthesis) • Task #2 : Illumination image imputation (CMU Multi-PIE[1]) • Task #3 : Facial expression image imputation (Radboud Face Data[2])
  21. Results #1: MR contrast imputation T1 FLAIR T2 weighted T2

    FLAIR T2 FLAIR* Proposed CycleGAN[1] StarGAN[2] Target [1] Zhu, JY et al. 2017, ICCV [3] Choi, Y et al. 2017, CVPR Poster #135 - CollaGAN: Collaborative GAN for Missing Image Data Imputation (Tues, June 18, 15:20-18:00, Poster1.2, Synthesis)
  22. Results #2: illumination imputation -90 º -45 º 45 º

    90 º Target Proposed StarGAN[2] CycleGAN[1] 0 º CollaGAN reconstructs not only the overall illumination but also the details of illumination from the nose, chin and cheeks. [1] Zhu, JY et al. 2017, ICCV [3] Choi, Y et al. 2017, CVPR Poster #135 - CollaGAN: Collaborative GAN for Missing Image Data Imputation (Tues, June 18, 15:20-18:00, Poster1.2, Synthesis)
  23. Results #3: Facial expression imputation CollaGAN reconstructs more natural and

    expressive facial expressions compared to CycleGAN and StarGAN Neutral Angry Contemptuous Disgusted Fearful Happy Sad Surprised Target Proposed StarGAN[2] CycleGAN[1] [1] Zhu, JY et al. 2017, ICCV [3] Choi, Y et al. 2017, CVPR Poster #135 - CollaGAN: Collaborative GAN for Missing Image Data Imputation (Tues, June 18, 15:20-18:00, Poster1.2, Synthesis)
  24. Results #3: Facial expression imputation • Redundancy of multiple inputs

    • Some of inputs may not contribute Poster #135 - CollaGAN: Collaborative GAN for Missing Image Data Imputation (Tues, June 18, 15:20-18:00, Poster1.2, Synthesis)
  25. Results #3: Facial expression imputation • Redundancy of multiple inputs

    • Some of inputs may not contribute Poster #135 - CollaGAN: Collaborative GAN for Missing Image Data Imputation (Tues, June 18, 15:20-18:00, Poster1.2, Synthesis) • To achieve the “collaborative” learning • Apply random nulling on N inputs (Missing N) : input dropout
  26. Results #3: Facial expression imputation Poster #135 - CollaGAN: Collaborative

    GAN for Missing Image Data Imputation (Tues, June 18, 15:20-18:00, Poster1.2, Synthesis) Missing 5 Missing 1 Missing 2 Missing 3 Missing 4 Missing 6 Target Missing 7 w/o dropout dropout Sad Amount of input information • Robust performance on various missing # • More input information à Better reconstruction quality
  27. Summary • CollaGAN for Multi-domain missing image imputation • Uses

    multiple input images • Multiple cycle consistency • Multi-domain : Single G & Single D • Synergistically combining the information from the available data with a help of input dropout. • Produces images with higher visual quality compared to the existing methods. Poster #135 - CollaGAN: Collaborative GAN for Missing Image Data Imputation (Tues, June 18, 15:20-18:00, Poster1.2, Synthesis)
  28. CollaGAN: Collaborative GAN for Missing Image Data Imputation Dongwook Lee,

    Junyoung Kim, Won-Jin Moon, Jong Chul Ye KONKUK UNIVERSITY MEDICAL CENTER Thank you for your attention. Poster #135 (Tues, June 18, 15:20-18:00, Poster1.2) Come over to my poster for more discussions!