Optimal Transport-driven CycleGAN for Unsupervised Learning in Inverse Problems

Optimal Transport-driven CycleGAN for Unsupervised Learning in Inverse Problems

IEEE SPS Computational Imaging Webinar Series: Signal Processing And Computational imagE formation (SPACE)

Homepage: https://sites.google.com/view/sps-space
Youtube: https://www.youtube.com/watch?v=DZG2ObAPS94

A3d61bc22cd700a92e7d4136a4d29e8f?s=128

Jong Chul Ye

July 14, 2020
Tweet

Transcript

  1. Optimal Transport-driven CycleGAN for Unsupervised Learning in Inverse Problems Jong

    Chul Ye, Ph.D. Professor BISPL - BioImaging, Signal Processing, and Learning lab. Dept. of Bio/Brain Engineering Dept. of Mathematical Sciences KAIST, Korea
  2. Classical Learning vs Deep Learning Diagnosis Classical machine learning Deep

    learning (no feature engineering) Feature Engineering Esteva et al, Nature Medicine, (2019)
  3. Deep Learning for Scientific Discovery Diagnosis Diagnosis & analysis New

    frontiers of deep learning: inverse problems
  4. Penalized LS for Inverse Problems Likelihood term Data fidelity term

    Prior term Regularization term • Classical approaches for inverse problems • Tikhonov, TV, Compressed sensing • Top-down model • Transductive à non-inductive • Computational expensive
  5. Feed-Forward Neural Network Approaches • CNN as a direct inverse

    operation • Most simplest and fastest method • Supervised learning with lots of data
  6. Model-based, PnP using CNN Prior Likelihood term Data fidelity term

    CNN based regularization • CNN is used as a denoiser • Can use relative small CNNs à fewer training data • Supervised learning • Still iterative Aggarwal et al, IEEE TMI, 2018; Liu et al, IEEE JSTSP, 2020; Wu et al, IEEE JSTSP, 2020
  7. Deep Image Prior (DIP) • CNN architecture as a regularization

    • Unsupervised learning • Extensive computation >> PLS Ulyanov et al, CVPR, 2018
  8. Unsupervised Feed-forward CNN?

  9. Yann LeCun’s Cake Analogy Slide courtesy of Yann LeCun’s ICIP

    2019 talk
  10. Why Unsupervised Learning in Inverse Problems? Low-dose CT Remote sensing

    Metal artifact removal Blind deconvolution
  11. Sim et al, arXiv:1909.12116, 2019, Lee et al, arXiv:2007.03480, 2020

    Forward physics (unknown, partially known, known) inverse solution Our Geometric View of Unsupervised Learning
  12. Geometry of GAN subject to Lei, Na, arXiv:1710.05488 (2017)

  13. Statistical Distances f-Divergence Wasserstein-1 metric GAN, f-GAN W-GAN divergence metric

  14. Absolute Continuity

  15. Optimal Transport : A Gentle Review

  16. Optimal Transport Transportation map

  17. Push-Forward of a Measure

  18. Optimal Transport: Monge Monge’s Original OT • Difficulty from the

    push-forward constraint Transportation cost
  19. Optimal Transport: Kantorovich Kantorovich’s OT • Allows mass splitting •

    Probabilistic OT • Linear programming à Nobel prize in economy Transportation cost Joint distribution
  20. Kantorovich Dual Formulation c-transform marginal distribution Kantorovich potential

  21. Wasserstein-1 Metric and Its Dual Kantorovich Dual 1-Lipschitz. (Lip1)

  22. Wasserstein GAN with Gradient Penalty 1-Lipschitz penalty generator discriminator

  23. Generative Adversarial Nets (GANs) Generator (transport map) Discriminator (Kantorovich Potential)

    https://www.cfml.se/blog/generative_adversarial_networks/
  24. Geometry of CycleGAN Sim et al, arXiv:1909.12116, 2019 Khan et

    al, arXiv:2006.14773, 2020 Lee et al, arXiv:2007.03480, 2020
  25. Geometry of CycleGAN Sim et al, arXiv:1909.12116, 2019, Lee et

    al, arXiv:2007.03480, 2020 Forward physics (unknown, partially known, known) inverse solution
  26. Two Wasserstein Metrics in Unsupervised Learning

  27. Joint Minimizationà CycleGAN Dual formulation Sim et al, arXiv:1909.12116, 2019

  28. CycleGAN: Loss Function 1-Lipschitz Cycle-consistency Discriminators Forward operator • Unknown

    • Partial known • known
  29. CycleGAN vs Penalized LS Data fidelity term Regularization term Data

    fidelity term Regularization term CycleGAN PLS CycleGAN can be considered as stochastic generalization of PLS
  30. 30 • Multiphase Cardiac CT denoising – Phase 1, 2:

    low-dose, Phase 3 ~ 10: normal dose – Goal: dynamic changes of heart structure – No reference available Kang et al, Medical Physics, 2018 Case 1: Unsupervised Denoising for Low-Dose CT
  31. Unsupervised Denoising by CycleGAN Kang et al, Medical Physics, 2019

  32. Lose dose (5%) à high dose Kang et al, unpublished

    data Input: phase 1 Proposed Target: phase 8 Input- output
  33. (a) (b) (c) (d) (e) (f) (g) (h) Input: phase

    1 Proposed Without identity loss GAN Ablation Study Kang et al, Medical Physics, 2018
  34. Input: phase 1 Proposed Without identity loss GAN Ablation Study

    (a) (b) (c) (d) (e) (f) (g) (h) Kang et al, Medical Physics, 2018
  35. Case 2: Unsupervised Deconvolution Microscopy Lim et al, IEEE TCI,

    2020
  36. Results on Real Microscopy Data ü Qualitative results • Runtime

    for 512 x 512 x 50 volume inference: 15 s Transverse view Sagittal view Lim et al, IEEE TCI, 2020
  37. Case 3: Unsupervised Learning for Accelerated MRI Sim et al,

    arXiv:1909.12116, 2019
  38. Results on Fast MR Data Set Sim et al, arXiv:1909.12116,

    2019
  39. Extensions • -CycleGAN • Multi-domain CycleGAN • Wavelet directional CycleGAN

  40. Loss function (ELBO: Evidence Lower Bound) Variational Auto-Encoder (VAE) Likelihood

    Latent space distance Kingma, et al arXiv:1312.6114 (2013)
  41. Latent Space Geometry of VAEs KL-divergence l2-distance Sample Space

  42. -Variational Auto Encoder (VAE) Feature Space Disentanglement Higgins et al,

    ICLR, 2017
  43. -CycleGAN for Unsupervised Metal Artifact Reduction Lee et al, arXiv:2007.03480,

    2020 Metal artifact images Artifact-free images Metal artifact generation physics (beam-hardening, photon starvation, etc.) à Highly complicated to learn Motivation • Less focus on the artifact generation • More emphasis on MAR
  44. -CycleGAN Disentangled Representation Dual formulation Lee et al, arXiv:2007.03480, 2020

  45. Discriminator: 1/ - Lipschitz -CycleGAN for Unsupervised Metal Artifact Removal

  46. Attention-guided -CycleGAN

  47. LI NMAR Input LI NMAR Input

  48. Preservation of Original Details Non-metallic cases

  49. Multiple-Domain CycleGAN :asymmetric transfer Motivation DAS image Despeckle Deconvolution US

    artifact removal Huh, et al, arXiv:2007.05205 (2020)
  50. Domain 3 Domain 1 Domain 2 G G Multiple-Domain CycleGAN

    :asymmetric transfer Domain 1 Domain 2 Domain 3 G CycleGAN Multi-Domain CycleGAN Huh, et al, arXiv:2007.05205 (2020)
  51. Multiple-Domain CycleGAN Domain 1 Domain 2 Domain 3 G StarGAN

    (symmetric) Proposed (asymmetric) Huh, et al, arXiv:2007.05205 (2020) Domain 1 Domain 2 Domain 3 G Choi et al, CVPR, 2019
  52. Geometry of Multi-Domain CycleGAN Domain 1 Domain 2 Huh, et

    al, arXiv:2007.05205 (2020)
  53. Multi-Domain CycleGAN: loss function KL divergence Wasserstein distance Huh, et

    al, arXiv:2007.05205 (2020)
  54. Multi-Domain CycleGAN: Dual Formulation 1-Lipschitz penalty Huh, et al, arXiv:2007.05205

    (2020)
  55. Multi-Domain CycleGAN for US Artifact Removal Huh, et al, arXiv:2007.05205

    (2020)
  56. None
  57. Wavelet Directional CycleGAN Song, et al, arXiv:2002.09847 (2020)

  58. Unsupervised Learning with CycleGAN Song, et al, arXiv:2002.09847 (2020)

  59. Agricultural area

  60. Cloud

  61. Ocean

  62. Summary • Unsupervised learning becomes very important topics in deep

    image reconstruction • Our theoretical findings • Optimal transport is an important mathematical tool for designing unsupervised networks • CycleGAN can be derived by minimizing two Wasserstein-1 distances in input and target spaces • Variation extensions of CycleGAN • Geometric view can be generalized for other problems
  63. Questions?