Optimal Transport-driven CycleGAN for Unsupervised Learning in Inverse Problems

IEEE SPS Computational Imaging Webinar Series: Signal Processing And Computational imagE formation (SPACE)

Homepage: https://sites.google.com/view/sps-space
Youtube: https://www.youtube.com/watch?v=DZG2ObAPS94

Transcript

1. Optimal Transport-driven CycleGAN for Unsupervised Learning in Inverse Problems Jong Chul Ye, Ph.D. Professor BISPL - BioImaging, Signal Processing, and Learning lab. Dept. of Bio/Brain Engineering Dept. of Mathematical Sciences KAIST, Korea
2. Classical Learning vs Deep Learning Diagnosis Classical machine learning Deep learning (no feature engineering) Feature Engineering Esteva et al, Nature Medicine, (2019)
3. Deep Learning for Scientific Discovery Diagnosis Diagnosis & analysis New frontiers of deep learning: inverse problems
4. Penalized LS for Inverse Problems Likelihood term Data fidelity term Prior term Regularization term • Classical approaches for inverse problems • Tikhonov, TV, Compressed sensing • Top-down model • Transductive à non-inductive • Computational expensive
5. Feed-Forward Neural Network Approaches • CNN as a direct inverse operation • Most simplest and fastest method • Supervised learning with lots of data
6. Model-based, PnP using CNN Prior Likelihood term Data fidelity term CNN based regularization • CNN is used as a denoiser • Can use relative small CNNs à fewer training data • Supervised learning • Still iterative Aggarwal et al, IEEE TMI, 2018; Liu et al, IEEE JSTSP, 2020; Wu et al, IEEE JSTSP, 2020
7. Deep Image Prior (DIP) • CNN architecture as a regularization • Unsupervised learning • Extensive computation >> PLS Ulyanov et al, CVPR, 2018
8. Unsupervised Feed-forward CNN?
9. Yann LeCun’s Cake Analogy Slide courtesy of Yann LeCun’s ICIP 2019 talk
10. Why Unsupervised Learning in Inverse Problems? Low-dose CT Remote sensing Metal artifact removal Blind deconvolution
11. Sim et al, arXiv:1909.12116, 2019, Lee et al, arXiv:2007.03480, 2020 Forward physics (unknown, partially known, known) inverse solution Our Geometric View of Unsupervised Learning
12. Geometry of GAN subject to Lei, Na, arXiv:1710.05488 (2017)
13. Statistical Distances f-Divergence Wasserstein-1 metric GAN, f-GAN W-GAN divergence metric
14. Absolute Continuity
15. Optimal Transport : A Gentle Review
16. Optimal Transport Transportation map
17. Push-Forward of a Measure
18. Optimal Transport: Monge Monge’s Original OT • Difficulty from the push-forward constraint Transportation cost
19. Optimal Transport: Kantorovich Kantorovich’s OT • Allows mass splitting • Probabilistic OT • Linear programming à Nobel prize in economy Transportation cost Joint distribution
20. Kantorovich Dual Formulation c-transform marginal distribution Kantorovich potential
21. Wasserstein-1 Metric and Its Dual Kantorovich Dual 1-Lipschitz. (Lip1)
22. Wasserstein GAN with Gradient Penalty 1-Lipschitz penalty generator discriminator
23. Generative Adversarial Nets (GANs) Generator (transport map) Discriminator (Kantorovich Potential) https://www.cfml.se/blog/generative_adversarial_networks/
24. Geometry of CycleGAN Sim et al, arXiv:1909.12116, 2019 Khan et al, arXiv:2006.14773, 2020 Lee et al, arXiv:2007.03480, 2020
25. Geometry of CycleGAN Sim et al, arXiv:1909.12116, 2019, Lee et al, arXiv:2007.03480, 2020 Forward physics (unknown, partially known, known) inverse solution
26. Two Wasserstein Metrics in Unsupervised Learning
27. Joint Minimizationà CycleGAN Dual formulation Sim et al, arXiv:1909.12116, 2019
28. CycleGAN: Loss Function 1-Lipschitz Cycle-consistency Discriminators Forward operator • Unknown • Partial known • known
29. CycleGAN vs Penalized LS Data fidelity term Regularization term Data fidelity term Regularization term CycleGAN PLS CycleGAN can be considered as stochastic generalization of PLS
30. 30 • Multiphase Cardiac CT denoising – Phase 1, 2: low-dose, Phase 3 ~ 10: normal dose – Goal: dynamic changes of heart structure – No reference available Kang et al, Medical Physics, 2018 Case 1: Unsupervised Denoising for Low-Dose CT
31. Unsupervised Denoising by CycleGAN Kang et al, Medical Physics, 2019
32. Lose dose (5%) à high dose Kang et al, unpublished data Input: phase 1 Proposed Target: phase 8 Input- output
33. (a) (b) (c) (d) (e) (f) (g) (h) Input: phase 1 Proposed Without identity loss GAN Ablation Study Kang et al, Medical Physics, 2018
34. Input: phase 1 Proposed Without identity loss GAN Ablation Study (a) (b) (c) (d) (e) (f) (g) (h) Kang et al, Medical Physics, 2018
35. Case 2: Unsupervised Deconvolution Microscopy Lim et al, IEEE TCI, 2020
36. Results on Real Microscopy Data ü Qualitative results • Runtime for 512 x 512 x 50 volume inference: 15 s Transverse view Sagittal view Lim et al, IEEE TCI, 2020
37. Case 3: Unsupervised Learning for Accelerated MRI Sim et al, arXiv:1909.12116, 2019
38. Results on Fast MR Data Set Sim et al, arXiv:1909.12116, 2019
39. Extensions • -CycleGAN • Multi-domain CycleGAN • Wavelet directional CycleGAN
40. Loss function (ELBO: Evidence Lower Bound) Variational Auto-Encoder (VAE) Likelihood Latent space distance Kingma, et al arXiv:1312.6114 (2013)
41. Latent Space Geometry of VAEs KL-divergence l2-distance Sample Space
42. -Variational Auto Encoder (VAE) Feature Space Disentanglement Higgins et al, ICLR, 2017
43. -CycleGAN for Unsupervised Metal Artifact Reduction Lee et al, arXiv:2007.03480, 2020 Metal artifact images Artifact-free images Metal artifact generation physics (beam-hardening, photon starvation, etc.) à Highly complicated to learn Motivation • Less focus on the artifact generation • More emphasis on MAR
44. -CycleGAN Disentangled Representation Dual formulation Lee et al, arXiv:2007.03480, 2020
45. Discriminator: 1/ - Lipschitz -CycleGAN for Unsupervised Metal Artifact Removal
46. Attention-guided -CycleGAN
47. LI NMAR Input LI NMAR Input
48. Preservation of Original Details Non-metallic cases
49. Multiple-Domain CycleGAN :asymmetric transfer Motivation DAS image Despeckle Deconvolution US artifact removal Huh, et al, arXiv:2007.05205 (2020)
50. Domain 3 Domain 1 Domain 2 G G Multiple-Domain CycleGAN :asymmetric transfer Domain 1 Domain 2 Domain 3 G CycleGAN Multi-Domain CycleGAN Huh, et al, arXiv:2007.05205 (2020)
51. Multiple-Domain CycleGAN Domain 1 Domain 2 Domain 3 G StarGAN (symmetric) Proposed (asymmetric) Huh, et al, arXiv:2007.05205 (2020) Domain 1 Domain 2 Domain 3 G Choi et al, CVPR, 2019
52. Geometry of Multi-Domain CycleGAN Domain 1 Domain 2 Huh, et al, arXiv:2007.05205 (2020)
53. Multi-Domain CycleGAN: loss function KL divergence Wasserstein distance Huh, et al, arXiv:2007.05205 (2020)
54. Multi-Domain CycleGAN: Dual Formulation 1-Lipschitz penalty Huh, et al, arXiv:2007.05205 (2020)
55. Multi-Domain CycleGAN for US Artifact Removal Huh, et al, arXiv:2007.05205 (2020)
57. Wavelet Directional CycleGAN Song, et al, arXiv:2002.09847 (2020)
58. Unsupervised Learning with CycleGAN Song, et al, arXiv:2002.09847 (2020)
59. Agricultural area
60. Cloud
61. Ocean
62. Summary • Unsupervised learning becomes very important topics in deep image reconstruction • Our theoretical findings • Optimal transport is an important mathematical tool for designing unsupervised networks • CycleGAN can be derived by minimizing two Wasserstein-1 distances in input and target spaces • Variation extensions of CycleGAN • Geometric view can be generalized for other problems
63. Questions?