Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Optimal Transport-driven CycleGAN for Unsupervised Learning in Inverse Problems

Optimal Transport-driven CycleGAN for Unsupervised Learning in Inverse Problems

IEEE SPS Computational Imaging Webinar Series: Signal Processing And Computational imagE formation (SPACE)

Homepage: https://sites.google.com/view/sps-space
Youtube: https://www.youtube.com/watch?v=DZG2ObAPS94

Jong Chul Ye

July 14, 2020
Tweet

More Decks by Jong Chul Ye

Other Decks in Research

Transcript

  1. Optimal Transport-driven CycleGAN for Unsupervised Learning in Inverse Problems Jong

    Chul Ye, Ph.D. Professor BISPL - BioImaging, Signal Processing, and Learning lab. Dept. of Bio/Brain Engineering Dept. of Mathematical Sciences KAIST, Korea
  2. Classical Learning vs Deep Learning Diagnosis Classical machine learning Deep

    learning (no feature engineering) Feature Engineering Esteva et al, Nature Medicine, (2019)
  3. Penalized LS for Inverse Problems Likelihood term Data fidelity term

    Prior term Regularization term • Classical approaches for inverse problems • Tikhonov, TV, Compressed sensing • Top-down model • Transductive à non-inductive • Computational expensive
  4. Feed-Forward Neural Network Approaches • CNN as a direct inverse

    operation • Most simplest and fastest method • Supervised learning with lots of data
  5. Model-based, PnP using CNN Prior Likelihood term Data fidelity term

    CNN based regularization • CNN is used as a denoiser • Can use relative small CNNs à fewer training data • Supervised learning • Still iterative Aggarwal et al, IEEE TMI, 2018; Liu et al, IEEE JSTSP, 2020; Wu et al, IEEE JSTSP, 2020
  6. Deep Image Prior (DIP) • CNN architecture as a regularization

    • Unsupervised learning • Extensive computation >> PLS Ulyanov et al, CVPR, 2018
  7. Sim et al, arXiv:1909.12116, 2019, Lee et al, arXiv:2007.03480, 2020

    Forward physics (unknown, partially known, known) inverse solution Our Geometric View of Unsupervised Learning
  8. Optimal Transport: Kantorovich Kantorovich’s OT • Allows mass splitting •

    Probabilistic OT • Linear programming à Nobel prize in economy Transportation cost Joint distribution
  9. Geometry of CycleGAN Sim et al, arXiv:1909.12116, 2019 Khan et

    al, arXiv:2006.14773, 2020 Lee et al, arXiv:2007.03480, 2020
  10. Geometry of CycleGAN Sim et al, arXiv:1909.12116, 2019, Lee et

    al, arXiv:2007.03480, 2020 Forward physics (unknown, partially known, known) inverse solution
  11. CycleGAN vs Penalized LS Data fidelity term Regularization term Data

    fidelity term Regularization term CycleGAN PLS CycleGAN can be considered as stochastic generalization of PLS
  12. 30 • Multiphase Cardiac CT denoising – Phase 1, 2:

    low-dose, Phase 3 ~ 10: normal dose – Goal: dynamic changes of heart structure – No reference available Kang et al, Medical Physics, 2018 Case 1: Unsupervised Denoising for Low-Dose CT
  13. Lose dose (5%) à high dose Kang et al, unpublished

    data Input: phase 1 Proposed Target: phase 8 Input- output
  14. (a) (b) (c) (d) (e) (f) (g) (h) Input: phase

    1 Proposed Without identity loss GAN Ablation Study Kang et al, Medical Physics, 2018
  15. Input: phase 1 Proposed Without identity loss GAN Ablation Study

    (a) (b) (c) (d) (e) (f) (g) (h) Kang et al, Medical Physics, 2018
  16. Results on Real Microscopy Data ü Qualitative results • Runtime

    for 512 x 512 x 50 volume inference: 15 s Transverse view Sagittal view Lim et al, IEEE TCI, 2020
  17. Loss function (ELBO: Evidence Lower Bound) Variational Auto-Encoder (VAE) Likelihood

    Latent space distance Kingma, et al arXiv:1312.6114 (2013)
  18. -CycleGAN for Unsupervised Metal Artifact Reduction Lee et al, arXiv:2007.03480,

    2020 Metal artifact images Artifact-free images Metal artifact generation physics (beam-hardening, photon starvation, etc.) à Highly complicated to learn Motivation • Less focus on the artifact generation • More emphasis on MAR
  19. Domain 3 Domain 1 Domain 2 G G Multiple-Domain CycleGAN

    :asymmetric transfer Domain 1 Domain 2 Domain 3 G CycleGAN Multi-Domain CycleGAN Huh, et al, arXiv:2007.05205 (2020)
  20. Multiple-Domain CycleGAN Domain 1 Domain 2 Domain 3 G StarGAN

    (symmetric) Proposed (asymmetric) Huh, et al, arXiv:2007.05205 (2020) Domain 1 Domain 2 Domain 3 G Choi et al, CVPR, 2019
  21. Summary • Unsupervised learning becomes very important topics in deep

    image reconstruction • Our theoretical findings • Optimal transport is an important mathematical tool for designing unsupervised networks • CycleGAN can be derived by minimizing two Wasserstein-1 distances in input and target spaces • Variation extensions of CycleGAN • Geometric view can be generalized for other problems