Share
Embed
Machine Learning in Medical Imaging
Plenary Talk at ISMRM Annual Meeting, Montreal, Canada, May 14th, 2019
Jong Chul Ye
May 14, 2019
Transcript
45 slides
-
Machine Learning in Medical Imaging Jong Chul Ye, Ph.D Endowed
Chair Professor BISPL - BioImaging, Signal Processing, and Learning lab. Dept. Bio & Brain Engineering Dept. Mathematical Sciences KAIST, Korea -
Speaker Name: Jong Chul Ye I have no financial interests
or relationships to disclose with regard to the subject matter of this presentation. Declaration of Financial Interests or Relationships -
Deep Learning Era in Medical Imaging Diabetic eye diagnosis Gulshan,
V. et al. JAMA (2016) Skin Cancer diagnosis Esteva et al, Nature (2017) OCT diagnosis De Fauw et al, Nature Medicine (2018) Figure courtesy of X. Cao & D. Shen Image registration Image segmentation Ronneberger et al, MICCAI, 2015 -
Classical Learning vs Deep Learning Diagnosis Classical machine learning Deep
learning (no feature engineering) Feature Engineering Esteva et al, Nature Medicine, (2019) -
Deep Learning for MRI Diagnosis Diagnosis & analysis Focus of
this talk: Reconstruction -
Unmet Needs in MRI q MR is an essential tool
for diagnosis q MR exam protocol : 30~60 min/patient ü should increase the throughput of MR scanning q Cardiac imaging, fMRI ü Should improve temporal resolution q Multiple contrast acquisition in a short time -
MR Acceleration Lustig et al, MRM, 2007; Jung et al,
PMB, 2007; MRM, 2009 fast pulse sequence parallel/multiband imaging Compressed sensing Shin et al, MRM, 2014; Haldar et al, TMI, 2014 Lee et al, MRM, 2016; Ongie et al, 2017 Sodickson et al, MRM, 1997; Pruessmann et al, MRM 1999; Griswold et al, MRM, 2002 Mansfield, JPC 1977; Ahn et al, TMI, 1986 MRF Ma et al, Nature, 2013; Jiang et al, MRM 2015 -
Year 2016: Deep Learning Revolution in MR Kwon et al,
Medical Physics, 2017 Hammernik et al, MRM, 2018 Wang et al, ISBI, 2016 Yang et al, NIPS, 2016 Multilayer perceptron Variational network Deep learning prior ADMM-Net -
Image Domain Learning In vivo golden angle radial acquisition results
(collaboration with KH Sung at UCLA) Object # of views Ground truth 302 views X : Input 75 views Target : abdomen Acceleration factor : x4 Training dataset : 15 slices 13.118e-2 (a) Ground truth (2nd slice) (b) X : Input (75) Accelerated Projection Reconstruction MR imaging using Deep Residual Learning MRM Highlight September 2018 2.3705e-2 (a) Ground truth (2nd slice) (c) Proposed (15) Object # of views Ground truth 302 views X : Input 75 views Target : abdomen Acceleration factor : x4 Training dataset : 15 slices In vivo golden angle radial acquisition results (collaboration with KH Sung at UCLA) Han et al. MRM, 2018 Domain adaptation network -
Image Domain Learning Zero-Filling R=5 R=5 MANTIS R=5 Global Low
Rank Local Low Rank Joint X-P Recon R=5 R=5 Direct parameter mapping (MANTIS) Liu et al. MRM, 2019 Courtesy of Fang Liu -
Image Domain Learning Accelerated MR fingerprinting Extracted Features FNN FNN
Signal Evolution Feature Extraction (FE) Module Low- Dimensional Feature Maps U-Net U-Net Spatially-Constrained Quantification (SQ) Module Output Tissue Property Maps T2 Input High- Dimension al MRF Signal T1 FNN: fully- connected neural network 200ms 100ms 0 R = 8 0 500 1000 1500 2000 2500 [X,Y]: [1 256] Index: 0 [R,G,B]: [0.03922 0 0] Template Matching Deep Learning 0 500 1000 1500 2000 2500 [X,Y]: [1 256] Index: 0 [R,G,B]: [0.03922 0 0] Courtesy of Dinggang Shen Feng Z et al, IEEE TMI, 2019 -
Hybrid Domain Learning Deep Cascade of CNNs for MRI Reconstruction
Schlemper et al. IEEE TMI 2017 Courtesy of D. Rueckert (a) 11x Undersampled (b) CNN reconstruction (c) Ground Truth -
Hybrid Domain Learning Eo et al , MRM, 2018 KIKI-net:
cross-domain CNN Courtesy of Doshik Hwang -
Sensor-domain Learning RAKI: Robust ANN for k-space Interpolation 1Akçakaya et
al, MRM, 2019 Courtesy of Mehmet Akcakaya -
What’s so special this time ? q Accuracy: high quality
recon > CS q Fast reconstruction time q Business model: vendor-driven training q Interpretable models q Flexibility: more than recon Imaging time Reconstruction time Conventional Compressed Sensing Machine Learning -
Variational Network (R=4) CG SENSE PI-CS: TGV Learning: VN PI
PI-CS Learning Hammernik MRM 2018 Courtesy of Florian Knoll -
K-space Deep Learning (Radial R=6) Ground-truth Acceleration Image learning CS
K-space learning Han et al, arXiv:1805.03779 (2018) -
K-space Deep Learning (Radial R=6) Ground-truth Acceleration Image learning CS
K-space learning Han et al, arXiv:1805.03779 (2018) -
Research Goal Ø To improve temporal resolution of TWIST imaging
using deep k-space learning Ø To generate multiple reconstruction results with various spatial and temporal resolution using one network VS = 5 VS = 2 CNN K-space Deep Learning for Time-resolved MRI Cha et al, arXiv:1806.00806 (2018). -
Rec_PF Zero-filling PD-net ADMM-net r e f PSNR: 41.8442 SSIM:
0.9920 PSNR: 38.5132 SSIM: 0.9651 PSNR: 32.5067 SSIM: 0.9127 PSNR: 23.3076 SSIM: 0.5980 Kim et al, Radiology 2018 pCASL denoising Primal Dual Net DPI-net Cheng et al, ISMRM 2019, p3983 Jun et al, MRM, 2019 Metal Artifact Removal Kwon et al, MICCAI 2018 T2 reconstruction from T1 and partial T2 Courtesy of Dinggang Shen Conventional minimum phase RF Reinforced learning-designed RF Peak RF reduced by x2.1 RF design D Shin, ISMRM 2019 #757 -
Deep Learning for Synthetic/Pseudo CT Leynes, Hope, Larson, et al.
J Nuc Med 2017. Input: Multi-contrast MRI Output: Synthetic CT Architecture: 3D Convolutional Neural Network Slide Courtesy of Peder Larson -
GAN for Image Translation Kang et al, Medical Physics, 2018
Input: phase 1 Proposed Target: phase 8 Input- output Unsupervised Denoising for Multiphase Coronary CT Angiography -
CollaGAN for MAGiC T2FLAIR Correction MAGiC T2-FLAIR T2-FLAIR MAGiC T2-FLAIR
T2-FLAIR MAGiC T2-FLAIR T2-FLAIR MAGiC T2-FLAIR T2-FLAIR -
WHY DEEP LEARNING WORKS FOR RECON ? DOES IT CREATE
ANY ARTIFICIAL FEATURES ? -
Signal Representation: Key to Classical Recon x = X hx,
e bn ibn <latexit sha1_base64="tCLG2nbXwywzwFFUoqMNNZJmas8=">AAACOnicbVDLSsNAFJ34rPVVdelmsAgupCQq6EYounHZgn1AU8pkctMOnUzCzEQtod/lxq9w58KNC0Xc+gFO2oBaPTBwOOdc5t7jxZwpbdtP1tz8wuLScmGluLq2vrFZ2tpuqiiRFBo04pFse0QBZwIammkO7VgCCT0OLW94mfmtG5CKReJaj2LohqQvWMAo0UbqlepuSPTAC9K7MT53VRJilxPR54C/jUPs3jIfNOM+pLmMvXFPuPJ3NNNwr1S2K/YE+C9xclJGOWq90qPrRzQJQWjKiVIdx451NyVSM8phXHQTBTGhQ9KHjqGChKC66eT0Md43io+DSJonNJ6oPydSEio1Cj2TzJZUs14m/ud1Eh2cdVMm4kSDoNOPgoRjHeGsR+wzCVTzkSGESmZ2xXRAJKHatF00JTizJ/8lzaOKc1yx6yfl6kVeRwHtoj10gBx0iqroCtVQA1F0j57RK3qzHqwX6936mEbnrHxmB/2C9fkFVQWu8g==</latexit> Synthesis basis Analysis basis coefficients Basis engineering: Wavelets, sparse basis(CS), dictionary learning, low-rank basis -
Ultimate Signal Representation for MR ? = <latexit sha1_base64="2wsinhV7OEj9020G2B+xBypL2+k=">AAAB6HicbVBNS8NAEJ34WetX1aOXxSJ4KokKehGKXjy2YD+gDWWznbRrN5uwuxFK6C/w4kERr/4kb/4bt20O2vpg4PHeDDPzgkRwbVz321lZXVvf2CxsFbd3dvf2SweHTR2nimGDxSJW7YBqFFxiw3AjsJ0opFEgsBWM7qZ+6wmV5rF8MOME/YgOJA85o8ZK9ZteqexW3BnIMvFyUoYctV7pq9uPWRqhNExQrTuemxg/o8pwJnBS7KYaE8pGdIAdSyWNUPvZ7NAJObVKn4SxsiUNmam/JzIaaT2OAtsZUTPUi95U/M/rpCa89jMuk9SgZPNFYSqIicn0a9LnCpkRY0soU9zeStiQKsqMzaZoQ/AWX14mzfOKd1Fx65fl6m0eRwGO4QTOwIMrqMI91KABDBCe4RXenEfnxXl3PuatK04+cwR/4Hz+AI13jMM=</latexit> x
= X hx, e bn ibn <latexit sha1_base64="tCLG2nbXwywzwFFUoqMNNZJmas8=">AAACOnicbVDLSsNAFJ34rPVVdelmsAgupCQq6EYounHZgn1AU8pkctMOnUzCzEQtod/lxq9w58KNC0Xc+gFO2oBaPTBwOOdc5t7jxZwpbdtP1tz8wuLScmGluLq2vrFZ2tpuqiiRFBo04pFse0QBZwIammkO7VgCCT0OLW94mfmtG5CKReJaj2LohqQvWMAo0UbqlepuSPTAC9K7MT53VRJilxPR54C/jUPs3jIfNOM+pLmMvXFPuPJ3NNNwr1S2K/YE+C9xclJGOWq90qPrRzQJQWjKiVIdx451NyVSM8phXHQTBTGhQ9KHjqGChKC66eT0Md43io+DSJonNJ6oPydSEio1Cj2TzJZUs14m/ud1Eh2cdVMm4kSDoNOPgoRjHeGsR+wzCVTzkSGESmZ2xXRAJKHatF00JTizJ/8lzaOKc1yx6yfl6kVeRwHtoj10gBx0iqroCtVQA1F0j57RK3qzHqwX6936mEbnrHxmB/2C9fkFVQWu8g==</latexit> basis 1 1 x coefficient -
= <latexit sha1_base64="2wsinhV7OEj9020G2B+xBypL2+k=">AAAB6HicbVBNS8NAEJ34WetX1aOXxSJ4KokKehGKXjy2YD+gDWWznbRrN5uwuxFK6C/w4kERr/4kb/4bt20O2vpg4PHeDDPzgkRwbVz321lZXVvf2CxsFbd3dvf2SweHTR2nimGDxSJW7YBqFFxiw3AjsJ0opFEgsBWM7qZ+6wmV5rF8MOME/YgOJA85o8ZK9ZteqexW3BnIMvFyUoYctV7pq9uPWRqhNExQrTuemxg/o8pwJnBS7KYaE8pGdIAdSyWNUPvZ7NAJObVKn4SxsiUNmam/JzIaaT2OAtsZUTPUi95U/M/rpCa89jMuk9SgZPNFYSqIicn0a9LnCpkRY0soU9zeStiQKsqMzaZoQ/AWX14mzfOKd1Fx65fl6m0eRwGO4QTOwIMrqMI91KABDBCe4RXenEfnxXl3PuatK04+cwR/4Hz+AI13jMM=</latexit> x = X hx, e bn ibn
<latexit sha1_base64="tCLG2nbXwywzwFFUoqMNNZJmas8=">AAACOnicbVDLSsNAFJ34rPVVdelmsAgupCQq6EYounHZgn1AU8pkctMOnUzCzEQtod/lxq9w58KNC0Xc+gFO2oBaPTBwOOdc5t7jxZwpbdtP1tz8wuLScmGluLq2vrFZ2tpuqiiRFBo04pFse0QBZwIammkO7VgCCT0OLW94mfmtG5CKReJaj2LohqQvWMAo0UbqlepuSPTAC9K7MT53VRJilxPR54C/jUPs3jIfNOM+pLmMvXFPuPJ3NNNwr1S2K/YE+C9xclJGOWq90qPrRzQJQWjKiVIdx451NyVSM8phXHQTBTGhQ9KHjqGChKC66eT0Md43io+DSJonNJ6oPydSEio1Cj2TzJZUs14m/ud1Eh2cdVMm4kSDoNOPgoRjHeGsR+wzCVTzkSGESmZ2xXRAJKHatF00JTizJ/8lzaOKc1yx6yfl6kVeRwHtoj10gBx0iqroCtVQA1F0j57RK3qzHqwX6936mEbnrHxmB/2C9fkFVQWu8g==</latexit> basis 1 1 x coefficient x = X hx, e bn(x)ibn(x) <latexit sha1_base64="erKoPFMSsbyGoza+nkJyT8tbnTk=">AAACdHicfVFNa9tAEF0pTeu4SeM2hx7awzbG4EIxUltoL4GQXnJ0of4Ay5jVauQsWa3E7qi1EfoF+Xe55WfkknNWjsC1XTqw8Hhv3szsTJhJYdDz7hx379n+8xeNg+bLw6NXx63Xb4YmzTWHAU9lqschMyCFggEKlDDONLAklDAKr39U+ug3aCNS9QuXGUwTNlciFpyhpWatmyBheBXGxaKkZ4HJExpIpuYS6Fr4RIM/IgIUMoKipmlYzlSAsMDVDIWGqCy6a8/HMtCbdSoD/a9j1mp7PW8VdBf4NWiTOvqz1m0QpTxPQCGXzJiJ72U4LZhGwSWUzSA3kDF+zeYwsVCxBMy0WLUvaccyEY1TbZ9CumL/dhQsMWaZhDazmtFsaxX5L22SY/x9WgiV5QiKPzWKc0kxpdUFaCQ0cJRLCxjXws5K+RXTjKO9U9Muwd/+8i4Yfu75X3rez6/t84t6HQ3yjpySLvHJN3JOLkmfDAgn985bhzofnAf3vdt2O0+prlN7TshGuL1HHSHCMQ==</latexit> Ideal basis should be adaptive to the input in real-time Ultimate Signal Representation for MR ? -
Encoder-Decoder CNN as Signal Representation Ye et al, ICML, 2019
Ye et al, SIAM J. Imaging Sciences, 2018 encoder decoder Synthesis basis Analysis basis -
Nonlinear Convolutional Framelet Representation ReLU (input adaptive) pooling un-pooling Learned
filters Ye et al, ICML, 2019 -
Channel, Depth, and Skipped Connection # of representation # of
network elements # of channel Network depth Skipped connection Ye et al, ICML, 2019 -
Take-away message • Deep learning is very powerful for recon
• Deep learning is a novel image representation with automatic input adaptivity. • Extension of classical regression, CS, PCA, etc • More training data gives better representation Don’t be afraid of using it ! -
Acknowledgement • Daniel Rueckert (Imperial College) • Florian Knoll (NYU)
• Fang Liu (Univ. of Wisconsin) • Mehmet Akcakaya (Univ. of Minnesota) • Dong Liang (SIAT, China) • Dinggang Shen (UNC) • Peder Larson (UCSF) • Grant – NRF of Korea – Ministry of Trade Industry and Energy • Hyunwook Park (KAIST) • Sung-hong Park (KAIST) • Jongho Lee (SNU) • Doshik Hwang (Yonsei Univ) • Won-Jin Moon (KonkukUniv Medical Center) • Eungyeop Kim (Gachon Univ. Medical Center) • Leonard Sunwoo (SNUBH) • Kyuhwan Jung (Vuno)