Deep Learning for Biomedical Image Reconstruction

Tutorial Talk, IEEE Symp. on Biomedical Imaging (ISBI), April 11th, 2019, Venice, Italy

Transcript

1. Deep Learning for Biomedical Image Reconstruction Jong Chul Ye, Ph.D Endowed Chair Professor BISPL - BioImaging, Signal Processing and Learning Lab. Dept. Bio & Brain Engineering KAIST, Korea Time: 14:45-18:00 Date: Thursday April 11th Place: Venetian Ballroom C Tutorial ThPM1T2 Latest material can be downloaded from https://bispl.weebly.com
2. Outline q Introduction to biomedical image reconstruction q Deep Learning: a brief review q Examples of deep learning for biomedical image reconstruction ü MRI ü Low-dose CT ü Optics ü Ultrasound q Interpretation of deep image recon ü Unrolled sparse recovery, FBPConvNet ü Variational neural network ü ADMM-Net, Learned primal dual ü Learned projected gradient method ü Deep convolutional framelets ü Representation learning q Advanced topics of deep image recon ü Unsupervised learning ü Contrast/image imputation
3. Analytic recon (FBP, Fourier) MBIR (MRF, Bayesian) Compressed sensing Deep Learning Four Waves of Image Reconstruction
4. Analytic Reconstruction (b) Delay and Sum (DAS) Time-reversal of a scattered wave (a) MR Imaging Beautiful analytic reconstruction results from fully sampled data
5. Differentiated Back-projection (DBP) 1. Differentiation 2. Backprojection 3. Filtration Analytic Reconstruction Zou, Y et al, PMB (2004).
6. Analytic Recon- Unmet Needs in Medical Imaging q High radiation dose CT for high quality imaging increases the risk of cancer for patients. q Long scan time of MR significantly reduces the scanner usage and reduce hospital revenue. q Low image equality of US is a technical huddle for portable ultrasound imaging system.
7. 7 Forward mapping By physics Measurement data Prior Knowledge (smoothness, sparsity,etc) Reconstructed image ˆ x = arg min x ky Axk2 2 + kDxk Model-based Iterative Recon (MBIR)
8. MBIR on the Market
9. 9 measurements vector # non-zeros • Incoherent projection • Underdetermined system • Sparse unknown vector Courtesy of Dr. Dror Baron b A x n ⇥ 1 k m ⇥ 1 m ' k log(n) ⌧ n Compressed Sensing (CS)
10. Compressed Sensing (CS) k-space
11. Clue: Redundancy = Sparsity
12. CS on the Market
13. Low-Rank Structured Matrix Approaches
14. TV-domain sparse signal cases Lee D. et al, MRM, 2016
15. Structured Low-Rank Approaches Jin et al, TIP, 2015; Jin et al, TIP, 2018 Ye et al, TIT, 2017; Ongie et al, SIIMS,2017 Ongie et al, TSP, 2018 MR Applications Image processing/computer vision Super-resolution Microscopy Theoretical guarantee Shin et al, MRM, 2014; Haldar et al, TMI, 2014 Lee et al , MRM, 2016; Ongie et al, SIIMS, 2017 Min et al, TIP, 2018
16. Year 2016: Deep Learning Breakthrough Kwon et al, Medical Physics, 2017 Hammernik et al, MRM, 2018 Wang et al, ISBI, 2016 Yang et al, NIPS, 2016 Multilayer perceptron Variational network Deep learning prior ADMM-Net
17. Year 2016: Deep Learning Breakthrough in MR Kwon et al, Medical Physics, 2017 Hammernik et al, MRM, 2018 Wang et al, ISBI, 2016 Yang et al, NIPS, 2016 Multilayer perceptron Variational network Deep learning prior ADMM-Net
18. 18 Year 2016: Deep Learning Breakthrough in CT
19. Jin et al. TIP 2017 Year 2016: Deep Learning Breakthrough in CT FBPConvNet
20. Challenges to Imaging Community q Real or cosmetic changes ? q Why it works ? q What is the optimal architecture ? q What is the link to the signal processing approaches ? q No use of domain expertise ?
21. DEEP LEARNING: A BRIEF REVIEW
22. ImageNet Challenge (Fei-Fei, 2009) • 1,000 object categories • Images: 1.2M (training), 100k (test) • ImageNet Large Scale Visual Recognition Challenge (LSVRC) Deng et al, CVPR, 2009
23. Deep Learning Age • Deep learning has been successfully used for classification, low-level computer vision, etc • Even outperforms human observers Figure modified from Kaiming He’s presentation
24. 24 Reinforcement Learning (AlphaGo)
25. Generative Adversarial Networks (GoodFellow, NIPS, 2016) Figure adopted from • BEGAN: Boundary Equilibrium Generative Adversarial Networks, 17’03 • StackGAN : Text to Photo-realistic Image Synthesis with Stacked Generative Adversarial Networks, ‘16.12
27. First CNN: LeNet (LeCun, 1998) LeCun, Yann, et al. "Gradient-based learning applied to document recognition." Proceedings of the IEEE 86.11 (1998): 2278-2324.
28. Convolutional Layer
29. Pooling & Unpooling 12 20 30 0 8 12 2 0 34 70 37 7 112 100 22 12 20 30 112 37 13 8 79 18 20 0 30 0 0 0 0 0 112 0 37 0 0 0 0 0 max pooling average pooling unpooling 13 0 8 0 0 0 0 0 79 0 18 0 0 0 0 0
30. AlexNet (Krizhevsky, NIPS, 2012) 30
31. VGGNet (Simonyan et al, 2014) 31
32. GoogleLeNet (Szegedy et al, CVPR, 2015) 32
33. ResNet (He et al, 2015)
34. U-Net (Ronneberger et al, 2015)
35. 35 Lee et al, ICML, 2009 Emergence of Hierarchical Features
36. 36 Hierarchical Representation LeCun et al, Nature, 2015
37. Visual Information Processing in Brain 37 Kravitz et al, Trends in Cognitive Sciences January 2013, Vol. 17, No. 1
38. Retina, V1 Layer Receptive fields of two ganglion cells in retina à convolution Orientation column in V1 http://darioprandi.com/docs/talks/image-reconstruction-recognition/graphics/pinwheels.jpg Figure courtesy by distillery.com
39. Visual Pathway Riesenhuber, Nature Neuroscience, 1999
40. “The Jennifer Anniston Cell” 40 Quiroga et al, Nature, Vol 435, 24, June 2005
42. EXAMPLE OF DEEP LEARNING FOR BIOMEDICAL IMAGE RECONSTRUCTION
43. DEEP NETWORKS FOR ACCELERATED MRI
44. Unmet Needs in MRI q MR is an essential tool for diagnosis q MR exam protocol : 30~60 min/patient ü should increase the throughput of MR scanning q Cardiac imaging, fMRI ü Should improve temporal resolution q Multiple contrast acquisition
45. MR Acceleration Lustig et al, MRM, 2007 Jung et al, PMB, 2007; MRM, 2009 Ma et al, Nature, 2013 fast pulse sequence parallel/multiband imaging Compressed sensing, MRF structured low-rank method Shin et al, MRM, 2014; Haldar et al, TMI, 2014 Lee et al, MRM, 2016; Ongie et al, 2017 Sodickson et al, MRM, 1997; Pruessmann et al, MRM 1999; Griswold et al, MRM, 2002 Mansfield, JPC 1977; Ahn et al, TMI, 1986
47. Image Domain Learning In vivo golden angle radial acquisition results (collaboration with KH Sung at UCLA) Object # of views Ground truth 302 views X : Input 75 views Target : abdomen Acceleration factor : x4 Training dataset : 15 slices 13.118e-2 (a) Ground truth (2nd slice) (b) X : Input (75) Accelerated Projection Reconstruction MR imaging using Deep Residual Learning MRM Highlight September 2018 2.3705e-2 (a) Ground truth (2nd slice) (c) Proposed (15) Object # of views Ground truth 302 views X : Input 75 views Target : abdomen Acceleration factor : x4 Training dataset : 15 slices In vivo golden angle radial acquisition results (collaboration with KH Sung at UCLA) Han et al. MRM, 2018; Lee et al, TBME, 2018 Domain adaptation network
48. Image Domain Learning QSMNet Yoon et al, NeuroImage, 2018 Courtesy of Jongho Lee
49. Image Domain Learning Mardani et al, IEEE TMI 2019 GANCS
50. Image Domain Learning Zero-Filling R=5 R=5 MANTIS R=5 Global Low Rank Local Low Rank Joint X-P Recon R=5 R=5 Direct parameter mapping (MANTIS) Liu et al. MRM, 2019 Courtesy of Fang Liu
51. Hybrid Domain Learning Deep Cascade of CNNs for MRI Reconstruction Schlemper et al. IEEE TMI 2017 Courtesy of D. Rueckert
52. Hybrid Domain Learning Eo et al , MRM, 2018 KIKI-net: cross-domain CNN Courtesy of Doshik Hwang
53. Hybrid Domain Learning Aggarwal et al, TMI, 2019 MoDL: model-based DL
54. Domain-transform Learning AUTOMAP Zhu et al, Nature, 2018
55. Sensor-domain Learning RAKI: Robust ANN for k-space Interpolation 1Akçakaya et al, MRM, 2019 Courtesy of Mehmet Akcakaya
56. Sensor-domain Learning CNN k-space deep learning Han et al, in revision, 2019
57. Hmm, ML was already used for MR recon…
58. 58 What’s so special this time ? q High quality recon: better than CS q Fast reconstruction time q Business model: vendor-driven training q Interpretable models Imaging time Reconstruction time Conventional Compressed Sensing Machine Learning
59. Variational Network (R=4) CG SENSE PI-CS: TGV Learning: VN PI PI-CS Learning Hammernik MRM 2018 Courtesy of Florian Knoll
60. K-space Deep Learning (Radial) Han et al, in revision, 2018
61. K-space Deep Learning (Radial) Han et al, in revision, 2018
62. K-space Deep Learning (Radial R=6) Han et al, in revision, 2018 Ground-truth Acceleration Image learning CS K-space learning
63. K-space Deep Learning (Radial R=6) Han et al, in revision, 2018 Ground-truth Acceleration Image learning CS K-space learning
64. Research Goal Ø To improve temporal resolution of TWIST imaging using deep k-space learning Ø To generate multiple reconstruction results with various spatial and temporal resolution using one network VS = 5 VS = 2 CNN K-space Deep Learning for Time-resolved MRI Cha et al, in revision, 2018
65. K-space Learning TWIST Cha et al, in revision, 2018 TWIST True Dynamics VS = 5
66. Ours with VS=2 True Dynamics VS = 2 K-space Learning TWIST Cha et al, in revision, 2018
67. Ours with VS=5 True Dynamics VS = 5 K-space Learning TWIST Cha et al, in revision, 2018
68. TWIST True Dynamics VS = 5 K-space Learning TWIST Cha et al, in revision, 2018
69. pCASL Denoising by CNN Avg=6 Avg=3 (Avg=3) + CNN Kim et al, Radiology 2018
70. DEEP NETWORKS FOR CT RECONSTRUCTION
71. Unmet Needs for X-ray CT
72. Dose Reduction Techniques • To reduce the radiation exposure, sparse view CT, low-dose CT and interior tomography. Sparse-view CT (Down-sampled View) Low-dose CT (Reduced X-ray dose) Interior Tomography (Truncated FOV)
73. Low-dose CT • To reduce the radiation exposure, sparse-view CT, low-dose CT and interior tomography. Sparse-view CT (Down-sampled View) Low-dose CT (Reduced X-ray dose) Interior Tomography (Truncated FOV)
74. Energy dependent attenuation coefficient metal bone Water & soft tissue X-ray source Detector Streaking artifacts+ random noises Difficult to design shrinkage using statistical approaches
75. 75 Wavelet transform level 2 level 1 level 3 level 4 Wavelet recomposition + Residual learning : Low-resolution image bypass High SNR band CNN (Kang, et al, Medical Physics 44(10))
76. 76 Da Cunha, Arthur L., Jianping Zhou, and Minh N. Do. "The nonsubsampled contourlet transform: theory, design, and applications." Image Processing, IEEE Transactions on 15.10 (2006): 3089-3101.
77. 77 Quarter dose Low freq. 1st level 2nd level 3nd level 4rd level
78. 78 Low freq. 1st level 2nd level 3nd level 4rd level Full dose
79. Routine dose Quarter dose (Kang, et al, Medical Physics 44(10) 2017)
80. Routine dose AAPM-Net results (Kang, et al, Medical Physics 44(10) 2017)
81. WavResNet results (Kang et al, TMI, 2018)
82. WavResNet results (Kang et al, TMI, 2018)
83. MBIR C D WavResNet results MBIR (Kang et al, TMI, 2018)
84. Full dose Quarter dose
85. Full dose Quarter dose
86. RED-CNN Chen et al. TMI 2017 Image Domain Learning
87. Wasserstein GAN and Perceptual Loss Yang et al, TMI 2018 Image Domain Learning
88. GAN Loss for unpaired training Wolterink et al, TMI 2017 Image Domain Learning
89. Sinogram Domain Learning Ramp Filter Deep Learning Wurfl et al. TMI 2018
90. Sparse-View CT • To reduce the radiation exposure, sparse-view CT, low-dose CT and interior tomography. Sparse-view CT (Down-sampled View) Low-dose CT (Reduced X-ray dose) Interior Tomography (Truncated FOV)
91. Streaking Artifacts Characteristics Required large receptive field Han et al. TMI 2018
92. Image Domain Learning FBPConvNet Jin et al. TIP 2017
93. Image Domain Learning Tight Frame U-Net JC Ye et al, SIAM Journal Imaging Sciences, 2018 Han et al, TMI, 2018
94. 90 view recon U-Net vs. Tight-Frame U-Net • JC Ye et al, SIAM Journal Imaging Sciences, 2018 • Y. Han and J. C. Ye, TMI, 2018
97. Hybrid Domain Learning Learned Primal and Dual Adler et al, TMI 2018
98. Hybrid Domain Learning Learned Projected Gradient Descent Gupta et al, TMI 2018
99. • Figures from internet Hybrid Domain Learning Extreme Sparse View CT
100. 9 View Dual Energy CT for Baggage Screening
101. 9 View Dual Energy CT for Baggage Screening Han et al, arXiv preprint arXiv:1712.10248, (2017); CT Meeting (2017)
102. 1st view 2nd view 3rd view 4th view 5th view 6th view 7th view 8th view 9th view
103. FBP
104. TV
106. ROI CT (Interior CT) • To reduce the radiation exposure, sparse-view CT, low-dose CT and interior tomography. Sparse-view CT (Down-sampled View) Low-dose CT (Reduced X-ray dose) Interior Tomography (Truncated FOV) Ward et al, SIIMS, 2015; Lee et al, SIIMS, 2015
107. Differentiated Back-projection (DBP) 1. Differentiation 2. Backprojection 3. Filtration Backprojection Filtration (BPF)
108. Truncation Artifacts Han et al, arXiv:1810.00500. 2018
109. Two Types of Architectures Image domain Learning DBP domain Learning Han et al, arXiv:1810.00500. 2018
110. Ground Truth FBP Truncation Artifacts Han et al, CT meeting, 2018
111. TV Chord Line Ours 8~10 dB gain Image Domain Learning Han et al, CT meeting, 2018
112. Ground Truth Chord Line TV Ours Image Domain Learning
113. DBP Domain Learning Generalizes better
114. DEEP NETWORKS FOR OPTICAL IMAGING
115. Microscopy Image Enhancement Rivenson et al, Optica, 2017
116. Holographic Microscopy Rivenson et al, Light, 2018
117. Imaging through a diffuser Lee et al, Optica, 2018
118. Super-resolution Microscopy Nehme et al, Optica, 2018
119. Super-resolution Microscopy Kim et al, ISBI 2019
120. Laser L1 L2 P BS1 GM L3 CL OL L4 L5 M1 L6 M2 BS2 Camera M3 Experimental Set-up [1] P : pinhole, L : lens, CL, condenser lens, OL : objective lens, BS : beam splitter, M : mirror, GM : galvano mirror, Measurement Total E-field Incident E-field Scattered E-field K-space-Trajectory [1] Yoon, Jonghee, et al. "Label-free characterization of white blood cells by measuring 3D refractive index maps." arXiv preprint arXiv:1505.02609 (2015). Optical Diffraction Tomography
121. Coherent Artifact Removal in ODT Choi et al, Optics Express, 2019
122. DEEP NETWORKS FOR ULTRASOUND IMAGING
123. B-mode / Plane Wave Ultrasound Imaging B-mode Imaging Plane Wave Imaging Couade M, JVDI, 2015 Yoon et al, TMI, 2018
124. Unmet Needs in US Imaging • Power consumption ü Many Rx should be used ü All ADC at RX needs power • Temporal resolution ü #scan line * echo sound speed ü Limiting factor for 3D or fast acquisition • Inaccuracy of time-reversal model ü DAS is based on time-rersal ü Time reversal is based on continuous model ü Needs adaptive beamformer • Needs for new US products ü Portable US ü 3-D US ü Ultrafast US
125. Deep Learning for US Reconstruction Luchies AC, et al. TMI, 2018 Vedula, et al, MICCAI, 2018. Zhou et al. TUFFC , 2018 Gasse et al. TUFFC , 2017 Deep Fourier Beamformer Deep Coherent Compounding ML to SL conversion Super-resolution Plane Wave
126. Low-power Fast US using RF Subsampling Rx subsampled SC subsampled - Use the portion of receivers - Reduce power consumption of ADC - Portable ultrasound systems - Use the portion of scanlines - Reduce RF data acquisition time - Ultra-fast ultrasound systems Yoon et al, TMI, 2018
127. Redundancy in Raw Data 127 Low-rank Hankel matrix à Deep learning
128. RF Interpolation via Deep Neural Networks Alphinion EC12R • Linear (L3-12H) : 8.48Mhz • Convex (SC1-4H): 3.2Mhz • Rx-SC (64x384) • 15,000 Depth Yoon et al, TMI, 2018
129. Yoon et al, TMI, 2018 RF Interpolation via Deep Neural Networks
130. Adaptive Beamforming Holm et al, 2009
131. Universal Deep Beamformer Conventional Beamforming Pipeline Universal Deep Beamformer Khan et al, arXiv:1901.01706 (2019)
132. Universal Deep Beamformer B-mode Imaging Plane Wave Imaging Khan et al, arXiv:1901.01706 (2019)
133. B-mode Universal Deep Beamformer Khan et al, arXiv:1901.01706 (2019)
134. Plane Wave Universal Deep Beamformer Khan et al, arXiv:1901.01706 (2019)
135. INTERPRETATION OF DEEP IMAGE RECON
136. WHY DEEP LEARNING WORKS FOR RECON ? DOES IT CREATE ANY ARTIFICIAL FEATURES ?
137. INTERPRETATION OF DEEP NETWORK
138. CNN: Too Simple to Analyze..? Convolution & pooling à stone age tools of signal processing What do they do ?
139. Skipped or Not ? Residual Network Clean image Standard Network Zhang, K., et al, IEEE TIP, 2017.
140. Visualization of conv1 filters from AlexNet Symmetric Filters ? Shang, et al, ICMR, 2016.
141. Overparameterization ? Han et al, CVPR, 2017
142. • What is the role of the nonlinearity such as rectified linear unit (ReLU) ? • Why do we need a pooling and unpooling in some architectures ? • Why do some networks need fully connected layers whereas the others do not ? • What is the role of by-pass connection or residual network ? • What is the role of the filter channels in convolutional layer ? Many Mysteries…
143. Dark Age of Signal Processing ?
144. • Direct connection to sparse recovery • Cannot explain the role of channel Learned ISTA (LISTA) Gregor et al,ICML, 2010
145. FBPConvNet Jin et al. TIP 2017 • Extension of LISTA when the normal operator is shift-invariant
146. • Multichannel filters from the decomposition of regularization term • Different from standard CNN Variational Neural Networks Courtesy of Florian Knoll
147. Learned Primal Dual Adler et al, TMI, 2018 Replacing the primal & dual proximal operators with CNN
148. Learned Projected Gradient Gupta et al, TMI, 2018 Replacing the projection operator with CNN
149. Generative Model • Image reconstruction as a distribution matching – However, difficult to explain the role of black-box network Bora et al, Compressed Sensing using Generative Models, arXiv:1703.03208
150. INTERPRETATION OF DEEP NETWORK DEEP CONVOLUTIONAL FRAMELETS
151. Matrix Representation of CNN Figure courtesy of Shoieb et al, 2016
152. Hankel Matrix: Lifting to Higher Dimensional Space
153. Hd(f) = U⌃V T : Non-local basis : Local basis Convolution Framelets (Yin et al; 2017) > = I > = I Hd(f)
154. Hd(f) Hd(f) = ˜ T ˜ T C C = T Hd(f) C = T (f ~ ) Encoder: ˜ T = I ˜ = PR(V ) Hd(f) = U⌃V T Unlifting: f = (˜C) ~ ⌧(˜ ) : Non-local basis : Local basis : Frame condition : rank condition convolution pooling un-pooling convolution : User-defined pooling : Learnable filters Hpi (gi) = X k,l [Ci]kl e Bkl i Decoder: Deep Convolutional Framelets (Y, Han, Cha; 2018)
155. Single Resolution Network Architecture
156. Missing elements can be found by low rank Hankel structured matrix completion Nuclear norm Projection on sampling positions min m kH(m)k⇤ subject to P⌦(b) = P⌦(f) RankH(f) = k * Jin KH et al IEEE TCI, 2016 * Jin KH et al.,IEEE TIP, 2015 * Ye JC et al., IEEE TIT, 2016 m Why Hankel Matrix?
157. 18. APR. 2015. 157 * Image Inpainting Results Jin et al, IEEE TIP, 2015
158. Multi-Resolution Network Architecture
159. Problem of U-net Pooling does NOT satisfy the frame condition JC Ye et al, SIAM Journal Imaging Sciences, 2018 Y. Han et al, TMI, 2018. ext > ext = I + > 6= I
160. Improving U-net using Deep Conv Framelets • Dual Frame U-net • Tight Frame U-net JC Ye et al, SIAM Journal Imaging Sciences, 2018 Y. Han and J. C. Ye, TMI, 2018
161. U-Net versus Dual Frame U-Net Y. Han and J. C. Ye, TMI, 2018; Yoo et al, SIJAM, 2018
162. Tight-Frame U-Net JC Ye et al, SIAM Journal Imaging Sciences, 2018
163. Denoising: U-Net vs. Tight-Frame U-Net
164. Inpainting: U-Net vs. Tight-Frame U-Net
165. Isola, et al. CVPR. 2017. Zhu, et al. CVPR, 2017. Style Transfer : power of wavelet pooling Pix2pix CycleGAN
166. Style Transfer : power of wavelet pooling
167. Single-level wavelets Multi-level wavelets
169. Limitations of the Interpretation qRole of ReLU ? qGeneralizability ? qExpressivity ? qOptimization landscape ?
170. INTERPRETATION OF DEEP NETWORK REPRESENTATION LEARNING
171. Representation Learning Representation space Cat axis D og axis Bull-Dog = 0.9x [Dog] + 0.01x[Cat]
172. = X i=1 nbn <latexit sha1_base64="hF8YSuka+Y36npfrys6bhoSxFsk=">AAACCHicbVDLSgMxFM34rPVVdenCYBFclRkVdFMounFZwT6gU4YkzbShSWZIMkIZZunGX3HjQhG3foI7/8ZMOwttPRA4nHMuuffgmDNtXPfbWVpeWV1bL22UN7e2d3Yre/ttHSWK0BaJeKS6GGnKmaQtwwyn3VhRJDCnHTy+yf3OA1WaRfLeTGLaF2goWcgIMlYKKkd1XyciSFndy3xMDQok9AUyIxymOAtsourW3CngIvEKUgUFmkHlyx9EJBFUGsKR1j3PjU0/RcowwmlW9hNNY0TGaEh7lkokqO6n00MyeGKVAQwjZZ80cKr+nkiR0HoisE3mO+p5Lxf/83qJCa/6KZNxYqgks4/ChEMTwbwVOGCKEsMnliCimN0VkhFSiBjbXdmW4M2fvEjaZzXvvObeXVQb10UdJXAIjsEp8MAlaIBb0AQtQMAjeAav4M15cl6cd+djFl1yipkD8AfO5w/SO5nZ</latexit> Eigenface Representation of a Face x <latexit sha1_base64="xEIMch3yuo7JxT4Wy1udoMzDhIk=">AAAB8XicbVDLSgMxFL1TX7W+qi7dBIvgqsyooMuiG5cV7APbUjLpnTY0kxmSjFiG/oUbF4q49W/c+Tdm2llo64HA4Zx7ybnHjwXXxnW/ncLK6tr6RnGztLW9s7tX3j9o6ihRDBssEpFq+1Sj4BIbhhuB7VghDX2BLX98k/mtR1SaR/LeTGLshXQoecAZNVZ66IbUjPwgfZr2yxW36s5AlomXkwrkqPfLX91BxJIQpWGCat3x3Nj0UqoMZwKnpW6iMaZsTIfYsVTSEHUvnSWekhOrDEgQKfukITP190ZKQ60noW8ns4R60cvE/7xOYoKrXsplnBiUbP5RkAhiIpKdTwZcITNiYgllitushI2ooszYkkq2BG/x5GXSPKt651X37qJSu87rKMIRHMMpeHAJNbiFOjSAgYRneIU3RzsvzrvzMR8tOPnOIfyB8/kD/eORHg==</latexit> PCA basis basis X <latexit sha1_base64="mRxQ3nFrUAg1mos1/MJ2As7dETY=">AAAB63icbVDLSgNBEOyNrxhfUY9eBoPgKeyqoMegF48RzAOSJcxOZpMhM7PLPISw5Be8eFDEqz/kzb9xNtmDJhY0FFXddHdFKWfa+P63V1pb39jcKm9Xdnb39g+qh0dtnVhFaIskPFHdCGvKmaQtwwyn3VRRLCJOO9HkLvc7T1RplshHM01pKPBIspgRbHKpr60YVGt+3Z8DrZKgIDUo0BxUv/rDhFhBpSEca90L/NSEGVaGEU5nlb7VNMVkgke056jEguowm986Q2dOGaI4Ua6kQXP190SGhdZTEblOgc1YL3u5+J/Xsya+CTMmU2uoJItFseXIJCh/HA2ZosTwqSOYKOZuRWSMFSbGxVNxIQTLL6+S9kU9uKz7D1e1xm0RRxlO4BTOIYBraMA9NKEFBMbwDK/w5gnvxXv3PhatJa+YOYY/8D5/ADKVjlU=</latexit>      
              
       coefficient
173. GLM Basis Representation of fMRI = X i=1 nbn <latexit sha1_base64="hF8YSuka+Y36npfrys6bhoSxFsk=">AAACCHicbVDLSgMxFM34rPVVdenCYBFclRkVdFMounFZwT6gU4YkzbShSWZIMkIZZunGX3HjQhG3foI7/8ZMOwttPRA4nHMuuffgmDNtXPfbWVpeWV1bL22UN7e2d3Yre/ttHSWK0BaJeKS6GGnKmaQtwwyn3VhRJDCnHTy+yf3OA1WaRfLeTGLaF2goWcgIMlYKKkd1XyciSFndy3xMDQok9AUyIxymOAtsourW3CngIvEKUgUFmkHlyx9EJBFUGsKR1j3PjU0/RcowwmlW9hNNY0TGaEh7lkokqO6n00MyeGKVAQwjZZ80cKr+nkiR0HoisE3mO+p5Lxf/83qJCa/6KZNxYqgks4/ChEMTwbwVOGCKEsMnliCimN0VkhFSiBjbXdmW4M2fvEjaZzXvvObeXVQb10UdJXAIjsEp8MAlaIBb0AQtQMAjeAav4M15cl6cd+djFl1yipkD8AfO5w/SO5nZ</latexit> x <latexit sha1_base64="xEIMch3yuo7JxT4Wy1udoMzDhIk=">AAAB8XicbVDLSgMxFL1TX7W+qi7dBIvgqsyooMuiG5cV7APbUjLpnTY0kxmSjFiG/oUbF4q49W/c+Tdm2llo64HA4Zx7ybnHjwXXxnW/ncLK6tr6RnGztLW9s7tX3j9o6ihRDBssEpFq+1Sj4BIbhhuB7VghDX2BLX98k/mtR1SaR/LeTGLshXQoecAZNVZ66IbUjPwgfZr2yxW36s5AlomXkwrkqPfLX91BxJIQpWGCat3x3Nj0UqoMZwKnpW6iMaZsTIfYsVTSEHUvnSWekhOrDEgQKfukITP190ZKQ60noW8ns4R60cvE/7xOYoKrXsplnBiUbP5RkAhiIpKdTwZcITNiYgllitushI2ooszYkkq2BG/x5GXSPKt651X37qJSu87rKMIRHMMpeHAJNbiFOjSAgYRneIU3RzsvzrvzMR8tOPnOIfyB8/kD/eORHg==</latexit> GLM basis Regression coefficient
174. Sparse Representation in CS bn <latexit sha1_base64="+PJYnVb53ACFuJdwoQMCxK7vOoI=">AAAB83icbVDLSsNAFL2pr1pfVZduBovgqiQq6LLoxmUF+4AmlMn0ph06mYSZiVBCf8ONC0Xc+jPu/BunbRbaemDgcM693DMnTAXXxnW/ndLa+sbmVnm7srO7t39QPTxq6yRTDFssEYnqhlSj4BJbhhuB3VQhjUOBnXB8N/M7T6g0T+SjmaQYxHQoecQZNVby/ZiaURjl4bQv+9WaW3fnIKvEK0gNCjT71S9/kLAsRmmYoFr3PDc1QU6V4UzgtOJnGlPKxnSIPUsljVEH+TzzlJxZZUCiRNknDZmrvzdyGms9iUM7Ocuol72Z+J/Xy0x0E+RcpplByRaHokwQk5BZAWTAFTIjJpZQprjNStiIKsqMraliS/CWv7xK2hd177LuPlzVGrdFHWU4gVM4Bw+uoQH30IQWMEjhGV7hzcmcF+fd+ViMlpxi5xj+wPn8AWXkkek=</latexit> hx, e bn i <latexit sha1_base64="U7NhedCxI11UMRr+85PC6B6cwUI=">AAACGXicbVDLSsNAFJ34rPVVdelmsAgupCQq6LLoxmUF+4AmhMnkph06mYSZiVpCf8ONv+LGhSIudeXfOG2z0NYDA4dzzmXuPUHKmdK2/W0tLC4tr6yW1srrG5tb25Wd3ZZKMkmhSROeyE5AFHAmoKmZ5tBJJZA44NAOBldjv30HUrFE3OphCl5MeoJFjBJtJL9iu5yIHgc3JrofRPnD6Ni9ZyFoxkPICxUHI1+4chL0K1W7Zk+A54lTkCoq0PArn26Y0CwGoSknSnUdO9VeTqRmlMOo7GYKUkIHpAddQwWJQXn55LIRPjRKiKNEmic0nqi/J3ISKzWMA5Mcr6pmvbH4n9fNdHTh5UykmQZBpx9FGcc6weOacMgkUM2HhhAqmdkV0z6RhGpTZtmU4MyePE9aJzXntGbfnFXrl0UdJbSPDtARctA5qqNr1EBNRNEjekav6M16sl6sd+tjGl2wipk99AfW1w9mz6HH</latexit> x <latexit sha1_base64="774qhuNAFXKctSHUINibxc5Dim4=">AAAB8nicbVBNS8NAFHypX7V+VT16WSyCp5KooMeiF48VbC20oWy2m3bpZhN2X8QS+jO8eFDEq7/Gm//GTZuDtg4sDDPvsfMmSKQw6LrfTmlldW19o7xZ2dre2d2r7h+0TZxqxlsslrHuBNRwKRRvoUDJO4nmNAokfwjGN7n/8Mi1EbG6x0nC/YgOlQgFo2ilbi+iOArC7GlK+tWaW3dnIMvEK0gNCjT71a/eIGZpxBUySY3pem6CfkY1Cib5tNJLDU8oG9Mh71qqaMSNn80iT8mJVQYkjLV9CslM/b2R0ciYSRTYyTyiWfRy8T+vm2J45WdCJSlyxeYfhakkGJP8fjIQmjOUE0so08JmJWxENWVoW6rYErzFk5dJ+6zundfdu4ta47qoowxHcAyn4MElNOAWmtACBjE8wyu8Oei8OO/Ox3y05BQ7h/AHzucPV6aRSA==</latexit> X n <latexit sha1_base64="eQZvkOUKW8DFp/whBQaQiuX1XSc=">AAAB7XicbVDLSgNBEOyNrxhfUY9eBoPgKexqQI9BLx4jmAckS5idzCZj5rHMzAphyT948aCIV//Hm3/jJNmDJhY0FFXddHdFCWfG+v63V1hb39jcKm6Xdnb39g/Kh0cto1JNaJMornQnwoZyJmnTMstpJ9EUi4jTdjS+nfntJ6oNU/LBThIaCjyULGYEWye1eiYVfdkvV/yqPwdaJUFOKpCj0S9/9QaKpIJKSzg2phv4iQ0zrC0jnE5LvdTQBJMxHtKuoxILasJsfu0UnTllgGKlXUmL5urviQwLYyYicp0C25FZ9mbif143tfF1mDGZpJZKslgUpxxZhWavowHTlFg+cQQTzdytiIywxsS6gEouhGD55VXSuqgGl1X/vlap3+RxFOEETuEcAriCOtxBA5pA4BGe4RXePOW9eO/ex6K14OUzx/AH3ucPtu+PNg==</latexit> basis Wavelet basis Learned Dictionary Sparse coefficient
175. Sparse Representation in CS k-space
176. Ultimate Signal Representation ? = <latexit sha1_base64="2wsinhV7OEj9020G2B+xBypL2+k=">AAAB6HicbVBNS8NAEJ34WetX1aOXxSJ4KokKehGKXjy2YD+gDWWznbRrN5uwuxFK6C/w4kERr/4kb/4bt20O2vpg4PHeDDPzgkRwbVz321lZXVvf2CxsFbd3dvf2SweHTR2nimGDxSJW7YBqFFxiw3AjsJ0opFEgsBWM7qZ+6wmV5rF8MOME/YgOJA85o8ZK9ZteqexW3BnIMvFyUoYctV7pq9uPWRqhNExQrTuemxg/o8pwJnBS7KYaE8pGdIAdSyWNUPvZ7NAJObVKn4SxsiUNmam/JzIaaT2OAtsZUTPUi95U/M/rpCa89jMuk9SgZPNFYSqIicn0a9LnCpkRY0soU9zeStiQKsqMzaZoQ/AWX14mzfOKd1Fx65fl6m0eRwGO4QTOwIMrqMI91KABDBCe4RXenEfnxXl3PuatK04+cwR/4Hz+AI13jMM=</latexit> x = X hx, e bn ibn <latexit sha1_base64="tCLG2nbXwywzwFFUoqMNNZJmas8=">AAACOnicbVDLSsNAFJ34rPVVdelmsAgupCQq6EYounHZgn1AU8pkctMOnUzCzEQtod/lxq9w58KNC0Xc+gFO2oBaPTBwOOdc5t7jxZwpbdtP1tz8wuLScmGluLq2vrFZ2tpuqiiRFBo04pFse0QBZwIammkO7VgCCT0OLW94mfmtG5CKReJaj2LohqQvWMAo0UbqlepuSPTAC9K7MT53VRJilxPR54C/jUPs3jIfNOM+pLmMvXFPuPJ3NNNwr1S2K/YE+C9xclJGOWq90qPrRzQJQWjKiVIdx451NyVSM8phXHQTBTGhQ9KHjqGChKC66eT0Md43io+DSJonNJ6oPydSEio1Cj2TzJZUs14m/ud1Eh2cdVMm4kSDoNOPgoRjHeGsR+wzCVTzkSGESmZ2xXRAJKHatF00JTizJ/8lzaOKc1yx6yfl6kVeRwHtoj10gBx0iqroCtVQA1F0j57RK3qzHqwX6936mEbnrHxmB/2C9fkFVQWu8g==</latexit> basis 1 1 x coefficient
177. = <latexit sha1_base64="2wsinhV7OEj9020G2B+xBypL2+k=">AAAB6HicbVBNS8NAEJ34WetX1aOXxSJ4KokKehGKXjy2YD+gDWWznbRrN5uwuxFK6C/w4kERr/4kb/4bt20O2vpg4PHeDDPzgkRwbVz321lZXVvf2CxsFbd3dvf2SweHTR2nimGDxSJW7YBqFFxiw3AjsJ0opFEgsBWM7qZ+6wmV5rF8MOME/YgOJA85o8ZK9ZteqexW3BnIMvFyUoYctV7pq9uPWRqhNExQrTuemxg/o8pwJnBS7KYaE8pGdIAdSyWNUPvZ7NAJObVKn4SxsiUNmam/JzIaaT2OAtsZUTPUi95U/M/rpCa89jMuk9SgZPNFYSqIicn0a9LnCpkRY0soU9zeStiQKsqMzaZoQ/AWX14mzfOKd1Fx65fl6m0eRwGO4QTOwIMrqMI91KABDBCe4RXenEfnxXl3PuatK04+cwR/4Hz+AI13jMM=</latexit> x = X hx, e bn ibn <latexit sha1_base64="tCLG2nbXwywzwFFUoqMNNZJmas8=">AAACOnicbVDLSsNAFJ34rPVVdelmsAgupCQq6EYounHZgn1AU8pkctMOnUzCzEQtod/lxq9w58KNC0Xc+gFO2oBaPTBwOOdc5t7jxZwpbdtP1tz8wuLScmGluLq2vrFZ2tpuqiiRFBo04pFse0QBZwIammkO7VgCCT0OLW94mfmtG5CKReJaj2LohqQvWMAo0UbqlepuSPTAC9K7MT53VRJilxPR54C/jUPs3jIfNOM+pLmMvXFPuPJ3NNNwr1S2K/YE+C9xclJGOWq90qPrRzQJQWjKiVIdx451NyVSM8phXHQTBTGhQ9KHjqGChKC66eT0Md43io+DSJonNJ6oPydSEio1Cj2TzJZUs14m/ud1Eh2cdVMm4kSDoNOPgoRjHeGsR+wzCVTzkSGESmZ2xXRAJKHatF00JTizJ/8lzaOKc1yx6yfl6kVeRwHtoj10gBx0iqroCtVQA1F0j57RK3qzHqwX6936mEbnrHxmB/2C9fkFVQWu8g==</latexit> basis 1 1 x coefficient x = X hx, e bn(x)ibn(x) <latexit sha1_base64="erKoPFMSsbyGoza+nkJyT8tbnTk=">AAACdHicfVFNa9tAEF0pTeu4SeM2hx7awzbG4EIxUltoL4GQXnJ0of4Ay5jVauQsWa3E7qi1EfoF+Xe55WfkknNWjsC1XTqw8Hhv3szsTJhJYdDz7hx379n+8xeNg+bLw6NXx63Xb4YmzTWHAU9lqschMyCFggEKlDDONLAklDAKr39U+ug3aCNS9QuXGUwTNlciFpyhpWatmyBheBXGxaKkZ4HJExpIpuYS6Fr4RIM/IgIUMoKipmlYzlSAsMDVDIWGqCy6a8/HMtCbdSoD/a9j1mp7PW8VdBf4NWiTOvqz1m0QpTxPQCGXzJiJ72U4LZhGwSWUzSA3kDF+zeYwsVCxBMy0WLUvaccyEY1TbZ9CumL/dhQsMWaZhDazmtFsaxX5L22SY/x9WgiV5QiKPzWKc0kxpdUFaCQ0cJRLCxjXws5K+RXTjKO9U9Muwd/+8i4Yfu75X3rez6/t84t6HQ3yjpySLvHJN3JOLkmfDAgn985bhzofnAf3vdt2O0+prlN7TshGuL1HHSHCMQ==</latexit> Ideal basis should be adaptive to the input Ultimate Signal Representation ?
178. Nonlinear Convolutional Framelet Representation • Ye et al, SIAM Journal Imaging Sciences, 2018. • Ye et al, arXiv:1901.07647, 2019 encoder basis decoder basis
179. Nonlinear Convolutional Framelet Representation ReLU (input adaptive) pooling un-pooling Learned filters Can be obtained from vectorized version of deep convolutional framelets
180. Input Space Partitioning in ReLU Networks Heinecke et al, Arxiv 1903.12384
181. Expressivity of CNN # of representation # of network elements # of channel Network depth Skipped connection Ye et al, arXiv:1901.07647, 2019
182. Take-away message y = X i h ,bi(x)ie bi(x) <latexit sha1_base64="wvdFNgdWBgyp03OsXJvyc2GFH4c=">AAACVnicbVFdS+QwFE2r7ujsh9V99OXiICgsQ6uCvgiiL/vowo4K06Gk6e0YTNKapOpQ+ifdl/Wn7MuymVrBVS8knJxzLrk5SUvBjQ3DR89fWFz60Fte6X/89PnLarC2fm6KSjMcsUIU+jKlBgVXOLLcCrwsNVKZCrxIr0/n+sUtasML9dPOSpxIOlU854xaRyWBjCW1V2lezxo4gthUMuEQC6qmAuObimbQ7t/g2Zc2Cd/uDnC/A7FuvRDf8QwtFxlC/Sy/9Nb3zQ4kwSAchm3BWxB1YEC6OkuChzgrWCVRWSaoMeMoLO2kptpyJrDpx5XBkrJrOsWxg4pKNJO6jaWBLcdkkBfaLWWhZV921FQaM5Opc85nNK+1OfmeNq5sfjipuSori4o9XZRXAmwB84wh4xqZFTMHKNPczQrsimrKrPuJvgshev3kt+B8dxjtDcMf+4Pjky6OZbJBNsk2icgBOSbfyRkZEUZ+kT+e7y14v72//pLfe7L6XtfzlfxXfvAPgDC0RA==</latexit> y = X i h ,bi(x)ie bi(x) <latexit sha1_base64="wvdFNgdWBgyp03OsXJvyc2GFH4c=">AAACVnicbVFdS+QwFE2r7ujsh9V99OXiICgsQ6uCvgiiL/vowo4K06Gk6e0YTNKapOpQ+ifdl/Wn7MuymVrBVS8knJxzLrk5SUvBjQ3DR89fWFz60Fte6X/89PnLarC2fm6KSjMcsUIU+jKlBgVXOLLcCrwsNVKZCrxIr0/n+sUtasML9dPOSpxIOlU854xaRyWBjCW1V2lezxo4gthUMuEQC6qmAuObimbQ7t/g2Zc2Cd/uDnC/A7FuvRDf8QwtFxlC/Sy/9Nb3zQ4kwSAchm3BWxB1YEC6OkuChzgrWCVRWSaoMeMoLO2kptpyJrDpx5XBkrJrOsWxg4pKNJO6jaWBLcdkkBfaLWWhZV921FQaM5Opc85nNK+1OfmeNq5sfjipuSori4o9XZRXAmwB84wh4xqZFTMHKNPczQrsimrKrPuJvgshev3kt+B8dxjtDcMf+4Pjky6OZbJBNsk2icgBOSbfyRkZEUZ+kT+e7y14v72//pLfe7L6XtfzlfxXfvAPgDC0RA==</latexit> • Deep learning is a novel image representation with automatic input adaptivity. • Extension of classical regression, CS, PCA, etc • More training data gives better representatio à Don’t be afraid of using it !
183. ADVANCED TOPICS: SEMI- & UN- SUPERVISED LEARNING
184. Unsupervised Learning for low-dose CT 184 • Multiphase Cardiac CT denoising – Phase 1, 2: low-dose, Phase 3 ~ 10: normal dose – Goal: dynamic changes of heart structure – No reference available Kang et al, Medical Physics, 2018
185. 185 Cycle Consistent Adversarial Denoising Network for Multiphase Coronary CT Angiography Kang et al, Medical Physics, 2018 Unsupervised Learning for low-dose CT
186. Lose dose (20%) Kang et al, Medical Physics, 2018 Input: phase 1 Proposed Target: phase 8 Input- output
187. Lose dose (5%) Kang et al, unpublished data Input: phase 1 Proposed Target: phase 8 Input- output
188. (a) (b) (c) (d) (e) (f) (g) (h) Input: phase 1 Proposed Without identity loss GAN Ablation Study Kang et al, Medical Physics, 2018
189. Input: phase 1 Proposed Without identity loss GAN Ablation Study (a) (b) (c) (d) (e) (f) (g) (h) Kang et al, Medical Physics, 2018
190. CycleGAN with Explicit PSF layer for Blind Deconv Lim et al, ISBI, 2019; arXiv:1904.02910 (2019)
191. CycleGAN with Explicit PSF layer for Blind Deconv Lim et al, ISBI, 2019; arXiv:1904.02910 (2019)
192. ADVANCED TOPICS: MR CONTRAST IMPUTATION ADVANCED TOPICS: IMAGE SYNTHESIS & IMPUTATION
193. Unpaired MR to CT Synthesis Wolterink et al. ISSM, 2017
194. CycleGAN for MR Contrast Synthesis Dar, et al. TMI, 2019
195. missing contrast for radiomics study ID T1w T2w T1-FLAIR T2-FLAIR #1 #2 ⋮ ⋮ ⋮ ⋮ ⋮ ID T1w T2w T1-FLAIR T2-FLAIR #1 #2 ⋮ ⋮ ⋮ ⋮ ⋮ incorrect contrast from synthetic MRI T2FLAIR (left) MAGIC T2FLAIR (right) • Partial volume effects • Motion artifact • Basilar artery and CSF pulsation artifact Synergistic Imputation from Multiple MR Contrast
196. Proposed method : CollaGAN • Imputation using Multiple inputs Input images Input images Target domain Input images Fake image G Multiple Inputs Single Generator Single Discriminator Adversarial model Multiple Cycle Consistency Lee et al, CVPR, 2019
197. • Mask vector for Single Generator Input images Input images Target domain Input images Fake image G Input : images + mask vector + Mask Vector = 2D one-hot vector Multiple Inputs Single Generator Single Discriminator Adversarial model Multiple Cycle Consistency Proposed method : CollaGAN Lee et al, CVPR, 2019
198. • Adversarial model using Dgan Input images Input images Target domain Input images Fake image G Fake image Real image Real / Fake (1) (2) D Dgan (1),(2) Multiple Inputs Single Generator Single Discriminator Adversarial model Multiple Cycle Consistency Proposed method : CollaGAN Lee et al, CVPR, 2019
199. • Dclsf for Single Discriminator Input images Input images Target domain Input images Fake image G Fake image Real image Real / Fake Domain classification (1) (2) D Dgan Dclsf (1),(2) (2) Multiple Inputs Single Generator Single Discriminator Adversarial model Multiple Cycle Consistency Proposed method : CollaGAN Lee et al, CVPR, 2019
200. • Multiple Cycle Consistency Loss Input images Input images Target domain Input images New Input images Original domain New Input images Cyclic input images New Input images Original domain New Input images Cyclic input images Fake image G Reconstructe d image Reconstructe d image Reconstructed image G New Input images Original domain New Input images Cyclic input images Fake image Real image Real / Fake Domain classification (1) (2) D Dgan Dclsf (1),(2) (2) Multiple Inputs Single Generator Single Discriminator Adversarial model Multiple Cycle Consistency Proposed method : CollaGAN Lee et al, CVPR, 2019
201. MR Contrast Imputation Lee et al, CVPR, 2019
202. CollaGAN for Synthetic MRI Lee et al, CVPR, 2019
203. Lee et al, unpublished data MAGiC T2-FLAIR T2-FLAIR MAGiC T2-FLAIR T2-FLAIR MAGiC T2-FLAIR T2-FLAIR MAGiC T2-FLAIR T2-FLAIR
204. Quantitative evaluation Segmentation performance on BRATS[1-2] Segmentation network Original T1 T2 T2F T1c Labels T1Colla T2Colla T2FColla T1GdColla • T1 / T1Gd / T2w / T2-FLAIR
205. Lee et al, unpublished data
206. Quantitative evaluation • Except the T1Gd, CollaGAN can accurately impute the missing contrast.
207. Outlooks q End-to-End AI for radiological imaging Ø From AI-powered image acquisition to diagnosis for clear and rapid radiological imaging Existing AI Solutions: Diagnosis Our future: from acquisition to diagnosis
208. Acknowledgement • Daniel Rueckert (Imperial College) • Florian Knoll (NYU) • Fang Liu (Univ. of Wisconsin) • Mehmet Akcakaya (Univ. of Minnesota) • Dong Liang (SIAT, China) • Grant – NRF of Korea – Ministry of Trade Industry and Energy • Hyunwook Park (KAIST) • Sung-hong Park (KAIST) • Jongho Lee (SNU) • Doshik Hwang (Yonsei Univ) • Won-Jin Moon (KonkukUniv Medical Center) • Eungyeop Kim (Gachon Univ. Medical Center) • Leonard Sunwoo (SNUBH) • Won Chang (SNUBH) • Chang-Min Park (SNUH) • Joon-Beom Seo (AMC) • Donghyun Yang (AMC) • Hakhee Kim (AMC) • Jungu Ri (AMC)
209. math Thank You
210. References (in the order they appeared in the presentation) 1. Zou, Y., & Pan, X. (2004). Exact image reconstruction on PI-lines from minimum data in helical cone-beam CT. Physics in Medicine & Biology, 49(6), 941. 2. Lee, D., Jin, K. H., Kim, E. Y., Park, S. H., & Ye, J. C. (2016). Acceleration of MR parameter mapping using annihilating filter-based low rank Hankel matrix (ALOHA). Magnetic resonance in medicine, 76(6), 1848-1864. 3. Lee, J., Jin, K. H., & Ye, J. C. (2016). Reference-free single-pass EPI N yquist ghost correction using annihilating filter-based low rank H ankel matrix (ALOHA). Magnetic resonance in medicine, 76(6), 1775-1789. 4. Jin, K. H., Lee, D., & Ye, J. C. (2016). A general framework for compressed sensing and parallel MRI using annihilating filter based low-rank Hankel matrix. IEEE Transactions on Computational Imaging, 2(4), 480-495. 5. Jin, K. H., Um, J. Y., Lee, D., Lee, J., Park, S. H., & Ye, J. C. (2017). MRI artifact correction using sparse+ low-rank decomposition of annihilating filter-based hankel matrix. Magnetic resonance in medicine, 78(1), 327-340. 6. Jin, K. H., & Ye, J. C. (2018). Sparse and low-rank decomposition of a Hankel structured matrix for impulse noise removal. IEEE Transactions on Image Processing, 27(3), 1448-1461. 7. Ye, J. C., Kim, J. M., Jin, K. H., & Lee, K. (2017). Compressive sampling using annihilating filter-based low-rank interpolation. IEEE Transactions on Information Theory, 63(2), 777-801.
211. 8. Shin, P. J., Larson, P. E., Ohliger, M. A., Elad, M., Pauly, J. M., Vigneron, D. B., & Lustig, M. (2014). Calibrationless parallel imaging reconstruction based on structured low-rank matrix completion. Magnetic resonance in medicine, 72(4), 959-970. 9. Haldar, J. P. (2014). Low-rank modeling of local $ k $-space neighborhoods (LORAKS) for constrained MRI. IEEE transactions on medical imaging, 33(3), 668-681. 10.Haldar, J. P., & Zhuo, J. (2016). P-LORAKS: Low-rank modeling of local k-space neighborhoods with parallel imaging data. Magnetic resonance in medicine, 75(4), 1499-1514. 11.Ongie, G., & Jacob, M. (2016). Off-the-grid recovery of piecewise constant images from few Fourier samples. SIAM journal on imaging sciences, 9(3), 1004-1041. 12.Ongie, G., & Jacob, M. (2017). A fast algorithm for convolutional structured low-rank matrix recovery. IEEE transactions on computational imaging, 3(4), 535-550. 13.Ongie, G., Biswas, S., & Jacob, M. (2018). Convex recovery of continuous domain piecewise constant images from nonuniform fourier samples. IEEE Transactions on Signal Processing, 66(1), 236-250. 14.Jin, K. H., & Ye, J. C. (2015). Annihilating filter-based low-rank Hankel matrix approach for image inpainting. IEEE Transactions on Image Processing, 24(11), 3498-3511.
212. 15.Min, J., Jin, K. H., Unser, M., & Ye, J. C. (2018). Grid-Free Localization Algorithm Using Low-Rank Hankel Matrix for Super-Resolution Microscopy. IEEE Transactions on Image Processing, 27(10), 4771-4786. 16.Kwon, K., Kim, D., & Park, H. (2017). A parallel MR imaging method using multilayer perceptron. Medical physics, 44(12), 6209-6224. 17.Hammernik, K., Klatzer, T., Kobler, E., Recht, M. P., Sodickson, D. K., Pock, T., & Knoll, F. (2018). Learning a variational network for reconstruction of accelerated MRI data. Magnetic resonance in medicine, 79(6), 3055-3071. 18.Wang, S., Su, Z., Ying, L., Peng, X., Zhu, S., Liang, F., ... & Liang, D. (2016, April). Accelerating magnetic resonance imaging via deep learning. In 2016 IEEE 13th International Symposium on Biomedical Imaging (ISBI) (pp. 514- 517). IEEE. 19.Sun, J., Li, H., & Xu, Z. (2016). Deep ADMM-Net for compressive sensing MRI. In Advances in neural information processing systems (pp. 10-18) 20.Kang, E., Min, J., & Ye, J. C. (2017). A deep convolutional neural network using directional wavelets for low-dose X-ray CT reconstruction. Medical physics, 44(10), e360-e375. 21.Deng, J., Dong, W., Socher, R., Li, L. J., Li, K., & Fei-Fei, L. (2009, June). Imagenet: A large-scale hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition (pp. 248-255). Ieee.
213. 22.Goodfellow, I., Pouget-Abadie, J., Mirza, M., Xu, B., Warde-Farley, D., Ozair, S., ... & Bengio, Y. (2014). Generative adversarial nets. In Advances in neural information processing systems (pp. 2672-2680). 23.Arjovsky, M., Chintala, S., & Bottou, L. (2017, July). Wasserstein generative adversarial networks. In International Conference on Machine Learning (pp. 214-223). 24.Berthelot, D., Schumm, T., & Metz, L. (2017). Began: Boundary equilibrium generative adversarial networks. arXiv preprint arXiv:1703.10717. 25.Zhang, H., Xu, T., Li, H., Zhang, S., Wang, X., Huang, X., & Metaxas, D. N. (2017). Stackgan: Text to photo-realistic image synthesis with stacked generative adversarial networks. In Proceedings of the IEEE International Conference on Computer Vision (pp. 5907-5915). 26.LeCun, Y., Bottou, L., Bengio, Y., & Haffner, P. (1998). Gradient-based learning applied to document recognition. Proceedings of the IEEE, 86(11), 2278-2324. 27.Krizhevsky, A., Sutskever, I., & Hinton, G. E. (2012). Imagenet classification with deep convolutional neural networks. In Advances in neural information processing systems (pp. 1097-1105). 28.Simonyan, K., & Zisserman, A. (2014). Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1409.1556.
214. 29.Szegedy, Christian, et al. "Going deeper with convolutions." Proceedings of the IEEE conference on computer vision and pattern recognition. 2015. 30.He, Kaiming, et al. "Deep residual learning for image recognition." Proceedings of the IEEE conference on computer vision and pattern recognition. 2016. 31.Ronneberger, Olaf, Philipp Fischer, and Thomas Brox. "U-net: Convolutional networks for biomedical image segmentation." International Conference on Medical image computing and computer-assisted intervention. Springer, Cham, 2015. 32.LeCun, Yann, Yoshua Bengio, and Geoffrey Hinton. "Deep learning." nature 521.7553 (2015): 436. 33.Lee, Honglak, et al. "Convolutional deep belief networks for scalable unsupervised learning of hierarchical representations." Proceedings of the 26th annual international conference on machine learning. ACM, 2009. 34.Kravitz, Dwight J., et al. "The ventral visual pathway: an expanded neural framework for the processing of object quality." Trends in cognitive sciences 17.1 (2013): 26-49. 35.Riesenhuber, M., & Poggio, T. (1999). Hierarchical models of object recognition in cortex. Nature neuroscience, 2(11), 1019. 36.Quiroga, R. Quian, et al. "Invariant visual representation by single neurons in the human brain." Nature 435.7045 (2005): 1102. 37.Mansfield, Peter. "Multi-planar image formation using NMR spin echoes." Journal of Physics C: Solid State Physics 10.3 (1977): L55.
215. 38.Ahn, C. B., J. H. Kim, and Z. H. Cho. "High-speed spiral-scan echo planar NMR imaging-I." IEEE transactions on medical imaging 5.1 (1986): 2-7. 39.Sodickson, Daniel K., and Warren J. Manning. "Simultaneous acquisition of spatial harmonics (SMASH): fast imaging with radiofrequency coil arrays." Magnetic resonance in medicine38.4 (1997): 591-603. 40.Pruessmann, Klaas P., et al. "SENSE: sensitivity encoding for fast MRI." Magnetic resonance in medicine 42.5 (1999): 952-962. 41.Griswold, Mark A., et al. "Generalized autocalibrating partially parallel acquisitions (GRAPPA)." Magnetic Resonance in Medicine: An Official Journal of the International Society for Magnetic Resonance in Medicine 47.6 (2002): 1202- 1210. 42.Ma, Dan, et al. "Magnetic resonance fingerprinting." Nature495.7440 (2013): 187. 43.Lustig, M., Donoho, D., & Pauly, J. M. (2007). Sparse MRI: The application of compressed sensing for rapid MR imaging. Magnetic Resonance in Medicine: An Official Journal of the International Society for Magnetic Resonance in Medicine, 58(6), 1182-1195. 44.Jung, Hong, et al. "k-t FOCUSS: a general compressed sensing framework for high resolution dynamic MRI." Magnetic Resonance in Medicine: An Official Journal of the International Society for Magnetic Resonance in Medicine61.1 (2009): 103-116. 45.Jung, H., Ye, J. C., & Kim, E. Y. (2007). Improved k–t BLAST and k–t SENSE using FOCUSS. Physics in Medicine & Biology, 52(11), 3201.
216. 46.Han, Y., Yoo, J., Kim, H. H., Shin, H. J., Sung, K., & Ye, J. C. (2018). Deep learning with domain adaptation for accelerated projection-reconstruction MR. Magnetic resonance in medicine, 80(3), 1189-1205. 47.Lee, D., Yoo, J., Tak, S., & Ye, J. C. (2018). Deep residual learning for accelerated MRI using magnitude and phase networks. IEEE Transactions on Biomedical Engineering, 65(9), 1985-1995. 48.Yoon, Jaeyeon, et al. "Quantitative susceptibility mapping using deep neural network: QSMnet." Neuroimage 179 (2018): 199-206. 49.Mardani, Morteza, et al. "Deep Generative Adversarial Neural Networks for Compressive Sensing MRI." IEEE transactions on medical imaging 38.1 (2019): 167-179. 50.Liu, F., Feng, L., & Kijowski, R. (2019). MANTIS: Model-Augmented Neural neTwork with Incoherent k-space Sampling for efficient MR parameter mapping. Magnetic resonance in medicine. 51.Schlemper, J., Caballero, J., Hajnal, J. V., Price, A. N., & Rueckert, D. (2018). A deep cascade of convolutional neural networks for dynamic MR image reconstruction. IEEE transactions on Medical Imaging, 37(2), 491-503. 52.Eo, T., Jun, Y., Kim, T., Jang, J., Lee, H. J., & Hwang, D. (2018). KIKI-net: cross-domain convolutional neural networks for reconstructing undersampled magnetic resonance images. Magnetic resonance in medicine, 80(5), 2188- 2201.
217. 53.Aggarwal, H. K., Mani, M. P., & Jacob, M. (2019). MoDL: Model-Based Deep Learning Architecture for Inverse Problems. IEEE transactions on medical imaging, 38(2), 394-405. 54.Zhu, B., Liu, J. Z., Cauley, S. F., Rosen, B. R., & Rosen, M. S. (2018). Image reconstruction by domain-transform manifold learning. Nature, 555(7697), 487. 55.Akçakaya, Mehmet, et al. "Scan-specific robust artificial-neural-networks for k-space interpolation (RAKI) reconstruction: Database-free deep learning for fast imaging." Magnetic resonance in medicine 81.1 (2019): 439-453. 56.Han, Y., & Ye, J. C. (2018). k-space deep learning for accelerated MRI. arXiv preprint arXiv:1805.03779. 57.Cha, E., Kim, E. Y., & Ye, J. C. (2018). k-Space Deep Learning for Parallel MRI: Application to Time-Resolved MR Angiography. arXiv preprint arXiv:1806.00806. 58.Lee, J., Han, Y., & Ye, J. C. (2018). k-Space Deep Learning for Reference-free EPI Ghost Correction. arXiv preprint arXiv:1806.00153. 59.Yan, H., & Mao, J. (1993). Data truncation artifact reduction in MR imaging using a multilayer neural network. IEEE transactions on medical imaging, 12(1), 73- 77. 60.Kim, K. H., Choi, S. H., & Park, S. H. (2017). Improving arterial spin labeling by using deep learning. Radiology, 287(2), 658-666. 61.Brenner, D. J., & Hall, E. J. (2007). Computed tomography—an increasing source of radiation exposure. New England Journal of Medicine, 357(22), 2277- 2284.
218. 62.Pearce, Mark S., et al. "Radiation exposure from CT scans in childhood and subsequent risk of leukaemia and brain tumours: a retrospective cohort study." The Lancet 380.9840 (2012): 499-505. 63.Da Cunha, A. L., Zhou, J., & Do, M. N. (2006). The nonsubsampled contourlet transform: theory, design, and applications. IEEE transactions on image processing, 15(10), 3089-3101. 64.Kang, E., Chang, W., Yoo, J., & Ye, J. C. (2018). Deep convolutional framelet denosing for low-dose CT via wavelet residual network. IEEE transactions on medical imaging, 37(6), 1358-1369. 65.Kang, E., & Ye, J. C. (2017). Wavelet domain residual network (WavResNet) for low-dose X-ray CT reconstruction. arXiv preprint arXiv:1703.01383. 66.Yang, Qingsong, et al. "Low-dose CT image denoising using a generative adversarial network with Wasserstein distance and perceptual loss." IEEE transactions on medical imaging 37.6 (2018): 1348-1357. 67.Chen, Hu, et al. "Low-dose CT with a residual encoder-decoder convolutional neural network." IEEE transactions on medical imaging 36.12 (2017): 2524- 2535. 68.Wolterink, J. M., Leiner, T., Viergever, M. A., & Išgum, I. (2017). Generative adversarial networks for noise reduction in low-dose CT. IEEE transactions on medical imaging, 36(12), 2536-2545. 69.Würfl, T., Hoffmann, M., Christlein, V., Breininger, K., Huang, Y., Unberath, M., & Maier, A. K. (2018). Deep learning computed tomography: learning projection-domain weights from image domain in limited angle problems. IEEE transactions on medical imaging, 37(6), 1454-1463.
219. 70.Han, Y., & Ye, J. C. (2018). Framing U-Net via deep convolutional framelets: Application to sparse-view CT. IEEE transactions on medical imaging, 37(6), 1418-1429. 71.Jin, K. H., McCann, M. T., Froustey, E., & Unser, M. (2017). Deep convolutional neural network for inverse problems in imaging. IEEE Transactions on Image Processing, 26(9), 4509-4522. 72.Ye, J. C., Han, Y., & Cha, E. (2018). Deep convolutional framelets: A general deep learning framework for inverse problems. SIAM Journal on Imaging Sciences, 11(2), 991-1048. 73.Adler, Jonas, and Ozan Öktem. "Learned primal-dual reconstruction." IEEE transactions on medical imaging 37.6 (2018): 1322-1332. 74.Gupta, H., Jin, K. H., Nguyen, H. Q., McCann, M. T., & Unser, M. (2018). CNN- based projected gradient descent for consistent CT image reconstruction. IEEE transactions on medical imaging, 37(6), 1440-1453. 75.Han, Y., Kang, J., & Ye, J. C. (2018). Deep learning reconstruction for 9-view dual energy CT baggage scanner. arXiv preprint arXiv:1801.01258. 76.Ward, J. P., Lee, M., Ye, J. C., & Unser, M. (2015). Interior tomography using 1D generalized total variation. Part I: Mathematical foundation. SIAM Journal on Imaging Sciences, 8(1), 226-247. 77.Lee, M., Han, Y., Ward, J. P., Unser, M., & Ye, J. C. (2015). Interior tomography using 1D generalized total variation. Part II: Multiscale implementation. SIAM Journal on Imaging Sciences, 8(4), 2452-2486.
220. 78.Han, Y., & Ye, J. C. (2018). One Network to Solve All ROIs: Deep Learning CT for Any ROI using Differentiated Backprojection. arXiv preprint arXiv:1810.00500. 79.Rivenson, Y., Göröcs, Z., Günaydin, H., Zhang, Y., Wang, H., & Ozcan, A. (2017). Deep learning microscopy. Optica, 4(11), 1437-1443. 80.Rivenson, Y., Zhang, Y., Günaydın, H., Teng, D., & Ozcan, A. (2018). Phase recovery and holographic image reconstruction using deep learning in neural networks. Light: Science & Applications, 7(2), 17141. 81.Li, S., Deng, M., Lee, J., Sinha, A., & Barbastathis, G. (2018). Imaging through glass diffusers using densely connected convolutional networks. Optica, 5(7), 803-813. 82.Nehme, E., Weiss, L. E., Michaeli, T., & Shechtman, Y. (2018). Deep-STORM: super-resolution single-molecule microscopy by deep learning. Optica, 5(4), 458- 464. 83.Lim, J., Lee, K., Jin, K. H., Shin, S., Lee, S., Park, Y., & Ye, J. C. (2015). Comparative study of iterative reconstruction algorithms for missing cone problems in optical diffraction tomography. Optics express, 23(13), 16933-16948. 84.Choi, G., Ryu, D., Jo, Y., Kim, Y. S., Park, W., Min, H. S., & Park, Y. (2019). Cycle-consistent deep learning approach to coherent noise reduction in optical diffraction tomography. Optics express, 27(4), 4927-4943. 85.Yoon, Y. H., & Ye, J. C. (2018, April). Deep learning for accelerated ultrasound imaging. In 2018 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) (pp. 6673-6676). IEEE.
221. 86.Yoon, Y. H., Khan, S., Huh, J., & Ye, J. C. (2019). Efficient B-mode ultrasound image reconstruction from sub-sampled rf data using deep learning. IEEE transactions on medical imaging, 38(2), 325-336. 87.Khan, S., Huh, J., & Ye, J. C. (2019). Universal Deep Beamformer for Variable Rate Ultrasound Imaging. arXiv preprint arXiv:1901.01706. 88.Luchies, A. C., & Byram, B. C. (2018). Deep neural networks for ultrasound beamforming. IEEE transactions on medical imaging, 37(9), 2010-2021. 89.Zhou, Z., Wang, Y., Yu, J., Guo, Y., Guo, W., & Qi, Y. (2018). High Spatial– Temporal Resolution Reconstruction of Plane-Wave Ultrasound Images With a Multichannel Multiscale Convolutional Neural Network. IEEE transactions on ultrasonics, ferroelectrics, and frequency control, 65(11), 1983-1996. 90.Gasse, M., Millioz, F., Roux, E., Garcia, D., Liebgott, H., & Friboulet, D. (2017). High-quality plane wave compounding using convolutional neural networks. IEEE transactions on ultrasonics, ferroelectrics, and frequency control, 64(10), 1637-1639. 91.Vedula, S., Senouf, O., Bronstein, A. M., Michailovich, O. V., & Zibulevsky, M. (2017). Towards CT-quality ultrasound imaging using deep learning. arXiv preprint arXiv:1710.06304. 92.Senouf, Ortal, et al. "High frame-rate cardiac ultrasound imaging with deep learning." International Conference on Medical Image Computing and Computer-Assisted Intervention. Springer, Cham, 2018. 93.Vedula, Sanketh, et al. "High quality ultrasonic multi-line transmission through deep learning." International Workshop on Machine Learning for Medical Image Reconstruction. Springer, Cham, 2018.
222. 94.Bora, A., Jalal, A., Price, E., & Dimakis, A. G. (2017, August). Compressed sensing using generative models. In Proceedings of the 34th International Conference on Machine Learning-Volume 70 (pp. 537-546). 95.Yoo, J., Wahab, A., & Ye, J. C. (2018). A mathematical framework for deep learning in elastic source imaging. SIAM Journal on Applied Mathematics, 78(5), 2791-2818. 96.Zhu, J. Y., Park, T., Isola, P., & Efros, A. A. (2017). Unpaired image-to-image translation using cycle-consistent adversarial networks. In Proceedings of the IEEE international conference on computer vision (pp. 2223-2232). 97.Zhu, J. Y., Park, T., Isola, P., & Efros, A. A. (2017). Unpaired image-to-image translation using cycle-consistent adversarial networks. In Proceedings of the IEEE international conference on computer vision (pp. 2223-2232). 98.Yoo, J., Uh, Y., Chun, S., Kang, B., & Ha, J. W. (2019). Photorealistic Style Transfer via Wavelet Transforms. arXiv preprint arXiv:1903.09760. 99.Ye, J. C., & Sung, W. K. (2019). Understanding Geometry of Encoder- Decoder CNNs. arXiv preprint arXiv:1901.07647. 100.Kang, E., Koo, H. J., Yang, D. H., Seo, J. B., & Ye, J. C. (2019). Cycle-consistent adversarial denoising network for multiphase coronary CT angiography. Medical physics, 46(2), 550-562. 101.Wolterink, Jelmer M., et al. "Deep MR to CT synthesis using unpaired data." International Workshop on Simulation and Synthesis in Medical Imaging. Springer, Cham, 2017.
223. 102.Dar, S. U., Yurt, M., Karacan, L., Erdem, A., Erdem, E., & Çukur, T. (2019). Image synthesis in multi-contrast MRI with conditional generative adversarial networks. IEEE transactions on medical imaging. 103.Lee, D., Kim, J., Moon, W. J., & Ye, J. C. (2019). CollaGAN: Collaborative GAN for Missing Image Data Imputation. CVPR, ArXiv preprint arXiv:1901.09764. 104.Choi, Y., Choi, M., Kim, M., Ha, J. W., Kim, S., & Choo, J. (2018). StarGAN: Unified generative adversarial networks for multi-domain image-to-image translation. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (pp. 8789-8797). 105.Zhang, K., Zuo, W., Chen, Y., Meng, D., & Zhang, L. (2017). Beyond a gaussian denoiser: Residual learning of deep cnn for image denoising. IEEE Transactions on Image Processing, 26(7), 3142-3155. 106.Shang, W., Sohn, K., Almeida, D., & Lee, H. (2016, June). Understanding and improving convolutional neural networks via concatenated rectified linear units. In international conference on machine learning (pp. 2217-2225). 107.Gregor, K., & LeCun, Y. (2010, June). Learning fast approximations of sparse coding. In Proceedings of the 27th International Conference on International Conference on Machine Learning (pp. 399-406). Omnipress. 108.Lim et al, Blind Deconvolution Microscopy Using Cycle Consistent CNN with Explicit PSF Layer, arXiv:1904.02910 (2019) 109.Khan et, Deep Learning-based Universal Beamformer for Ultrasound Imaging, arXiv:1904.02843 (2019)