Understanding Geometry of Encoder-Decoder CNNs for Inverse Problems

Plenary Talk, Applied Inverse Problems (AIP) conference, Grenoble, July 11th, 2019

Transcript

1. Understanding Geometry of Encoder Decoder CNN for Inverse Problems Jong Chul Ye, Ph.D Endowed Chair Professor BISPL - BioImaging, Signal Processing, and Learning lab. Dept. Bio & Brain Engineering Dept. Mathematical Sciences KAIST, Korea
2. Deep Learning Revolution • Deep learning has been successfully used for classification, low- level computer vision, etc • Even outperforms human observers Figure modified from Kaiming He’s presentation
3. Classical Learning vs Deep Learning Diagnosis Classical machine learning Deep learning (no feature engineering) Feature Engineering Esteva et al, Nature Medicine, (2019)
4. Deep Learning Era in Medical Imaging Diabetic eye diagnosis Gulshan, V. et al. JAMA (2016) Skin Cancer diagnosis Esteva et al, Nature (2017) OCT diagnosis De Fauw et al, Nature Medicine (2018) Figure courtesy of X. Cao & D. Shen Image registration Image segmentation Ronneberger et al, MICCAI, 2015
5. Deep Learning for Inverse Problems Diagnosis Diagnosis & analysis Focus of this talk: Reconstruction
7. 7 Wavelet transform level 2 level 1 level 3 level 4 Wavelet recomposition + Residual learning : Low-resolution image bypass High SNR band CNN (Kang, et al, Medical Physics 44(10))
8. Full dose Quarter dose
9. Full dose Ours from Quarter dose
10. Ours (Kang et al, TMI, 2018)
11. Ours (Kang et al, TMI, 2018)
12. MBIR C D Ours MBIR (Kang et al, TMI, 2018)
13. Figures from internet Extreme Sparse View CT Stationary CT Carry on baggage scanner Han et al, arXiv preprint arXiv:1712.10248, (2017); CT Meeting (2017)
14. Source/detector configuration Han et al, arXiv preprint arXiv:1712.10248, (2017); CT Meeting (2017)
15. 1st view 2nd view 3rd view 4th view 5th view 6th view 7th view 8th view 9th view
16. FBP
17. TV
18. Deep Learning Han et al, CT meetings, 2018
19. Deep Learning Pioneers in MR Kwon et al, Medical Physics, 2017 Hammernik et al, MRM, 2018 Wang et al, ISBI, 2016 Yang et al, NIPS, 2016 Multilayer perceptron Variational network Deep learning prior ADMM-Net
20. Variational Network (R=4) CG SENSE PI-CS: TGV Learning: VN PI PI-CS Learning Hammernik MRM 2018 Courtesy of Florian Knoll
22. Image Domain Learning FBPConvNet Jin et al. TIP 2017
23. Hybrid Domain Learning Deep Cascade of CNNs for MRI Reconstruction Schlemper et al. IEEE TMI 2017 Courtesy of D. Rueckert (a) 11x Undersampled (b) CNN reconstruction (c) Ground Truth
24. Hybrid Domain Learning Learned Primal and Dual Adler et al, IEEE TMI 2018
25. Domain-transform Learning AUTOMAP Zhu et al, Nature, 2018
26. Sensor-domain Learning CNN k-space deep learning Han et al, IEEE TMI; Lee et al, MRM (in press, 2019)
27. Why so popular this time ? q Accuracy: high quality recon > CS q Fast reconstruction time q Business model: vendor-driven training q Interpretable models q Flexibility: more than recon Imaging time Reconstruction time Conventional Compressed Sensing Machine Learning
28. INTERPRETATION OF DEEP LEARNING FOR RECON
29. Too Simple to Analyze..? Convolution & pooling à stone age tools of signal processing What do they do ?
30. Dark Age of Applied Mathematics ?
31. WHY DEEP LEARNING WORKS FOR RECON ? DOES IT CREATE ANY ARTIFICIAL FEATURES ?
32. Ye et al, SIAM J. Imaging Sciences, 2018; Ye et al, ICML, 2019 Understanding Geometry of CNN
33. CNN Encoder-Decoder CNN for Inverse Problems
34. CNN Encoder-Decoder CNN for Inverse Problems
35. CNN Successful applications to various inverse problems Encoder-Decoder CNN for Inverse Problems
36. Why Same Architecture Works for Different Inverse Problems ?
37. Classical Methods for Inverse Problems Synthesis frame Analysis frame coefficients Step 1: Signal Representation x = X i hbi, xi˜ bi <latexit sha1_base64="1xOFabKTay95z7c6ZUXCSJWXZAE=">AAACE3icbVDLSsNAFJ3UV62vqEs3g0UQkZKooBuh4MZlC/YBTQiTyaQdOpmEmYm0hIKf4MZfceNCEbdu3Pk3TpMutPXAhTPn3Mvce/yEUaks69soLS2vrK6V1ysbm1vbO+buXlvGqcCkhWMWi66PJGGUk5aiipFuIgiKfEY6/vBm6nfuiZA05ndqnBA3Qn1OQ4qR0pJnnoyuHZlGHoXQYYj3GYG+R0/hyBHFy1GUBbnomVWrZuWAi8SekSqYoeGZX04Q4zQiXGGGpOzZVqLcDAlFMSOTipNKkiA8RH3S05SjiEg3y2+awCOtBDCMhS6uYK7+nshQJOU48nVnhNRAzntT8T+vl6rwys0oT1JFOC4+ClMGVQynAcGACoIVG2uCsKB6V4gHSCCsdIwVHYI9f/IiaZ/V7POa1byo1psPRRxlcAAOwTGwwSWog1vQAC2AwSN4Bq/gzXgyXox346NoLRmzCPfBHxifP6mnndg=</latexit>
38. Eg. Compressed Sensing Classical Methods for Inverse Problems Step 2: Basis Search by Optimization x = X i ˜ bi hbi, xi <latexit sha1_base64="dRyoeK2luEuIb3X8ywuvlvflFPU=">AAACEnicbZBNS8MwGMdTX+d8q3r0EhyCgoxWBb0IQy8eJ7gXWEtJ03QLS9OSpLJR9hm8+FW8eFDEqydvfhvTrgfd/EPgl//zPCTP308Ylcqyvo2FxaXlldXKWnV9Y3Nr29zZbcs4FZi0cMxi0fWRJIxy0lJUMdJNBEGRz0jHH97k9c4DEZLG/F6NE+JGqM9pSDFS2vLM49GVI9PIo9BRlAUE+jkyxPus4BM4ckRx88yaVbcKwXmwS6iBUk3P/HKCGKcR4QozJGXPthLlZkgoihmZVJ1UkgThIeqTnkaOIiLdrFhpAg+1E8AwFvpwBQv390SGIinHka87I6QGcraWm//VeqkKL92M8iRVhOPpQ2HKoIphng8MqCBYsbEGhAXVf4V4gATCSqdY1SHYsyvPQ/u0bp/VrbvzWuO6jKMC9sEBOAI2uAANcAuaoAUweATP4BW8GU/Gi/FufExbF4xyZg/8kfH5AyCNnR8=</latexit>
39. Why do They Look so Different ? Any Link between Them ?
40. Our Theoretical Findings y = X i hbi(x), xi˜ bi(x) <latexit sha1_base64="DaaFmbtzayW3V2tBvW3rbADydJY=">AAACGXicbZDLSsNAFIYnXmu9RV26GSxCBSmJCroRim5cVrAXaEKYTCbt0MkkzEykIfQ13Pgqblwo4lJXvo3TNoK2/jDw851zOHN+P2FUKsv6MhYWl5ZXVktr5fWNza1tc2e3JeNUYNLEMYtFx0eSMMpJU1HFSCcRBEU+I21/cD2ut++JkDTmdypLiBuhHqchxUhp5JlWdunINPIohA5DvMcI9D1aHR4dw6EjpsBRlAU/3DMrVs2aCM4buzAVUKjhmR9OEOM0IlxhhqTs2lai3BwJRTEjo7KTSpIgPEA90tWWo4hIN59cNoKHmgQwjIV+XMEJ/T2Ro0jKLPJ1Z4RUX87WxvC/WjdV4YWbU56kinA8XRSmDKoYjmOCARUEK5Zpg7Cg+q8Q95FAWOkwyzoEe/bkedM6qdmnNev2rFK/KuIogX1wAKrABuegDm5AAzQBBg/gCbyAV+PReDbejPdp64JRzOyBPzI+vwEaXJ8Y</latexit> Ye et al, SIIMS, 2018; Ye et al, ICML, 2019
41. y = X i hbi(x), xi˜ bi(x) <latexit sha1_base64="DaaFmbtzayW3V2tBvW3rbADydJY=">AAACGXicbZDLSsNAFIYnXmu9RV26GSxCBSmJCroRim5cVrAXaEKYTCbt0MkkzEykIfQ13Pgqblwo4lJXvo3TNoK2/jDw851zOHN+P2FUKsv6MhYWl5ZXVktr5fWNza1tc2e3JeNUYNLEMYtFx0eSMMpJU1HFSCcRBEU+I21/cD2ut++JkDTmdypLiBuhHqchxUhp5JlWdunINPIohA5DvMcI9D1aHR4dw6EjpsBRlAU/3DMrVs2aCM4buzAVUKjhmR9OEOM0IlxhhqTs2lai3BwJRTEjo7KTSpIgPEA90tWWo4hIN59cNoKHmgQwjIV+XMEJ/T2Ro0jKLPJ1Z4RUX87WxvC/WjdV4YWbU56kinA8XRSmDKoYjmOCARUEK5Zpg7Cg+q8Q95FAWOkwyzoEe/bkedM6qdmnNev2rFK/KuIogX1wAKrABuegDm5AAzQBBg/gCbyAV+PReDbejPdp64JRzOyBPzI+vwEaXJ8Y</latexit> Our Theoretical Findings Ye et al, SIIMS, 2018; Ye et al, ICML, 2019
42. y = X i hbi(x), xi˜ bi(x) <latexit sha1_base64="DaaFmbtzayW3V2tBvW3rbADydJY=">AAACGXicbZDLSsNAFIYnXmu9RV26GSxCBSmJCroRim5cVrAXaEKYTCbt0MkkzEykIfQ13Pgqblwo4lJXvo3TNoK2/jDw851zOHN+P2FUKsv6MhYWl5ZXVktr5fWNza1tc2e3JeNUYNLEMYtFx0eSMMpJU1HFSCcRBEU+I21/cD2ut++JkDTmdypLiBuhHqchxUhp5JlWdunINPIohA5DvMcI9D1aHR4dw6EjpsBRlAU/3DMrVs2aCM4buzAVUKjhmR9OEOM0IlxhhqTs2lai3BwJRTEjo7KTSpIgPEA90tWWo4hIN59cNoKHmgQwjIV+XMEJ/T2Ro0jKLPJ1Z4RUX87WxvC/WjdV4YWbU56kinA8XRSmDKoYjmOCARUEK5Zpg7Cg+q8Q95FAWOkwyzoEe/bkedM6qdmnNev2rFK/KuIogX1wAKrABuegDm5AAzQBBg/gCbyAV+PReDbejPdp64JRzOyBPzI+vwEaXJ8Y</latexit> Our Theoretical Findings Ye et al, SIIMS, 2018; Ye et al, ICML, 2019
43. analysis basis y = X i hbi(x), xi˜ bi(x) <latexit sha1_base64="DaaFmbtzayW3V2tBvW3rbADydJY=">AAACGXicbZDLSsNAFIYnXmu9RV26GSxCBSmJCroRim5cVrAXaEKYTCbt0MkkzEykIfQ13Pgqblwo4lJXvo3TNoK2/jDw851zOHN+P2FUKsv6MhYWl5ZXVktr5fWNza1tc2e3JeNUYNLEMYtFx0eSMMpJU1HFSCcRBEU+I21/cD2ut++JkDTmdypLiBuhHqchxUhp5JlWdunINPIohA5DvMcI9D1aHR4dw6EjpsBRlAU/3DMrVs2aCM4buzAVUKjhmR9OEOM0IlxhhqTs2lai3BwJRTEjo7KTSpIgPEA90tWWo4hIN59cNoKHmgQwjIV+XMEJ/T2Ro0jKLPJ1Z4RUX87WxvC/WjdV4YWbU56kinA8XRSmDKoYjmOCARUEK5Zpg7Cg+q8Q95FAWOkwyzoEe/bkedM6qdmnNev2rFK/KuIogX1wAKrABuegDm5AAzQBBg/gCbyAV+PReDbejPdp64JRzOyBPzI+vwEaXJ8Y</latexit> Encoder Our Theoretical Findings Ye et al, SIIMS, 2018; Ye et al, ICML, 2019
44. analysis basis synthesis basis y = X i hbi(x), xi˜ bi(x) <latexit sha1_base64="DaaFmbtzayW3V2tBvW3rbADydJY=">AAACGXicbZDLSsNAFIYnXmu9RV26GSxCBSmJCroRim5cVrAXaEKYTCbt0MkkzEykIfQ13Pgqblwo4lJXvo3TNoK2/jDw851zOHN+P2FUKsv6MhYWl5ZXVktr5fWNza1tc2e3JeNUYNLEMYtFx0eSMMpJU1HFSCcRBEU+I21/cD2ut++JkDTmdypLiBuhHqchxUhp5JlWdunINPIohA5DvMcI9D1aHR4dw6EjpsBRlAU/3DMrVs2aCM4buzAVUKjhmR9OEOM0IlxhhqTs2lai3BwJRTEjo7KTSpIgPEA90tWWo4hIN59cNoKHmgQwjIV+XMEJ/T2Ro0jKLPJ1Z4RUX87WxvC/WjdV4YWbU56kinA8XRSmDKoYjmOCARUEK5Zpg7Cg+q8Q95FAWOkwyzoEe/bkedM6qdmnNev2rFK/KuIogX1wAKrABuegDm5AAzQBBg/gCbyAV+PReDbejPdp64JRzOyBPzI+vwEaXJ8Y</latexit> Encoder Decoder Our Theoretical Findings Ye et al, SIIMS, 2018; Ye et al, ICML, 2019
45. Linear Encoder-Decoder (ED) CNN Learned filters y = ˜ BB>x = X i hx, bi i˜ bi <latexit sha1_base64="bo3reUJLRRRgiLys4OrWvNpVArY=">AAACJ3icbVBNSwMxEM36bf2qevQSLIIHKbsq6KUievGoaK3QrSWbzrah2eySzIpl8d948a94EVREj/4T03YP2joQePPePCbzgkQKg6775UxMTk3PzM7NFxYWl5ZXiqtr1yZONYcqj2WsbwJmQAoFVRQo4SbRwKJAQi3onvb12h1oI2J1hb0EGhFrKxEKztBSzeJRr+KjkC2gJ/Tk1sc4ofe0Qn2TRk1BfclUWwK936FBv9XDNndYqlksuWV3UHQceDkokbzOm8VXvxXzNAKFXDJj6p6bYCNjGgWX8FDwUwMJ413WhrqFikVgGtngzge6ZZkWDWNtn0I6YH87MhYZ04sCOxkx7JhRrU/+p9VTDA8bmVBJiqD4cFGYSoox7YdGW0IDR9mzgHEt7F8p7zDNONpoCzYEb/TkcXC9W/b2yu7Ffun4Mo9jjmyQTbJNPHJAjskZOSdVwskjeSZv5N15cl6cD+dzODrh5J518qec7x9ZY6R3</latexit> pooling un-pooling
46. Linear E-D CNN w/ Skipped Connection more redundant expression Learned filters y = ˜ BB>x = X i hx, bi i˜ bi <latexit sha1_base64="bo3reUJLRRRgiLys4OrWvNpVArY=">AAACJ3icbVBNSwMxEM36bf2qevQSLIIHKbsq6KUievGoaK3QrSWbzrah2eySzIpl8d948a94EVREj/4T03YP2joQePPePCbzgkQKg6775UxMTk3PzM7NFxYWl5ZXiqtr1yZONYcqj2WsbwJmQAoFVRQo4SbRwKJAQi3onvb12h1oI2J1hb0EGhFrKxEKztBSzeJRr+KjkC2gJ/Tk1sc4ofe0Qn2TRk1BfclUWwK936FBv9XDNndYqlksuWV3UHQceDkokbzOm8VXvxXzNAKFXDJj6p6bYCNjGgWX8FDwUwMJ413WhrqFikVgGtngzge6ZZkWDWNtn0I6YH87MhYZ04sCOxkx7JhRrU/+p9VTDA8bmVBJiqD4cFGYSoox7YdGW0IDR9mzgHEt7F8p7zDNONpoCzYEb/TkcXC9W/b2yu7Ffun4Mo9jjmyQTbJNPHJAjskZOSdVwskjeSZv5N15cl6cD+dzODrh5J518qec7x9ZY6R3</latexit>
47. Deep Convolutional Framelets x = ˜ BB>x = X i hx, bi i˜ bi <latexit sha1_base64="9EuOyjKGC2x9hAgBpajvIdywLlA=">AAACJ3icbVBNSwMxEM36bf2qevQSLIIHKbsq6EWRevGoaK3QXZdsOq2h2eySzEpL6b/x4l/xIqiIHv0npu0etDoQePPePCbzolQKg6776UxMTk3PzM7NFxYWl5ZXiqtr1ybJNIcqT2SibyJmQAoFVRQo4SbVwOJIQi1qnw702j1oIxJ1hd0Ugpi1lGgKztBSYfG4c+SjkA2gFVq59TFJaYceUd9kcSioL5lqSaCdHRoNWj1qc4elwmLJLbvDon+Bl4MSyes8LL74jYRnMSjkkhlT99wUgx7TKLiEfsHPDKSMt1kL6hYqFoMJesM7+3TLMg3aTLR9CumQ/enosdiYbhzZyZjhnRnXBuR/Wj3D5mHQEyrNEBQfLWpmkmJCB6HRhtDAUXYtYFwL+1fK75hmHG20BRuCN37yX3C9W/b2yu7FfunkMo9jjmyQTbJNPHJATsgZOSdVwskDeSKv5M15dJ6dd+djNDrh5J518qucr29XoqR2</latexit> Perfect reconstruction Ye et al, SIIMS 2018; Ye et al, ICML 2019 Frame conditions w skipped connection w/o skipped connection
48. Deep Convolutional Framelets x = ˜ BB>x = X i hx, bi i˜ bi <latexit sha1_base64="9EuOyjKGC2x9hAgBpajvIdywLlA=">AAACJ3icbVBNSwMxEM36bf2qevQSLIIHKbsq6EWRevGoaK3QXZdsOq2h2eySzEpL6b/x4l/xIqiIHv0npu0etDoQePPePCbzolQKg6776UxMTk3PzM7NFxYWl5ZXiqtr1ybJNIcqT2SibyJmQAoFVRQo4SbVwOJIQi1qnw702j1oIxJ1hd0Ugpi1lGgKztBSYfG4c+SjkA2gFVq59TFJaYceUd9kcSioL5lqSaCdHRoNWj1qc4elwmLJLbvDon+Bl4MSyes8LL74jYRnMSjkkhlT99wUgx7TKLiEfsHPDKSMt1kL6hYqFoMJesM7+3TLMg3aTLR9CumQ/enosdiYbhzZyZjhnRnXBuR/Wj3D5mHQEyrNEBQfLWpmkmJCB6HRhtDAUXYtYFwL+1fK75hmHG20BRuCN37yX3C9W/b2yu7FfunkMo9jjmyQTbJNPHJATsgZOSdVwskDeSKv5M15dJ6dd+djNDrh5J518qucr29XoqR2</latexit> Perfect reconstruction Ye et al, SIAM J. Imaging Science, 2018 Frame conditions w skipped connection w/o skipped connection
49. Role of ReLUs? Generator for Multiple Expressions y = ˜ B(x)B(x)>x = X i hx, bi(x)i˜ bi(x) <latexit sha1_base64="T/1m1u26m8O8vLHErH3u6EKQhAM=">AAACM3icbVDLSgMxFM3Ud31VXboJFkFByowKuhGKbsSVotVCpw6Z9LYNZjJDckdaSv/JjT/iQhAXirj1H0zbWaj1QsLhPEjuCRMpDLrui5ObmJyanpmdy88vLC4tF1ZWr02cag4VHstYV0NmQAoFFRQooZpoYFEo4Sa8OxnoN/egjYjVFXYTqEespURTcIaWCgpn3SMfhWwAPd7qbA+vWx/jhHboEfVNGgWC+pKplgTa2aFhIAY2X4+YLDpig0LRLbnDoePAy0CRZHMeFJ78RszTCBRyyYypeW6C9R7TKLiEft5PDSSM37EW1CxULAJT7w137tNNyzRoM9b2KKRD9meixyJjulFonRHDtvmrDcj/tFqKzcN6T6gkRVB89FAzlRRjOiiQNoQGjrJrAeNa2L9S3maacbQ1520J3t+Vx8H1bsnbK7kX+8XyZVbHLFknG2SLeOSAlMkpOScVwskDeSZv5N15dF6dD+dzZM05WWaN/Brn6xvQsKgT</latexit> ⌃l(x) = 2 6 6 6 4 1 0 · · · 0 0 2 · · · 0 . . . . . . ... . . . 0 0 · · · ml 3 7 7 7 5 <latexit sha1_base64="1HHS4n8UkvGQcnzeL2YdPrnnXeg=">AAACkXicbVHLSgMxFM2M7/qqunQTLIpuyowK6kIpuhHcKFoVmjpkMrdtMMkMSUYsQ//H73Hn35i2g4/qhQvnnnNfuYkzwY0Ngg/Pn5qemZ2bX6gsLi2vrFbX1u9NmmsGTZaKVD/G1IDgCpqWWwGPmQYqYwEP8fPFUH94AW14qu5sP4O2pF3FO5xR66io+kZueVfSJ7H7uodPMYmhy1URS2o1fx1gYoZqFOIdHDgnLEmtGQWEjJlxwv6kWCEvZfQNkt+MSwom+5btChkJNxxU8rVKVK0F9WBk+C8IS1BDpV1H1XeSpCyXoCwT1JhWGGS2XVBtORMwqJDcQEbZM+1Cy0FFJZh2MbroAG87JsGdVDtXFo/YnxUFlcb0Zewy3X49M6kNyf+0Vm47x+2Cqyy3oNh4UCcX2KZ4+D044RqYFX0HKNPc7YpZj2rKrPvEijtCOPnkv+B+vx4e1IObw1rjvDzHPNpEW2gXhegINdAlukZNxLwV79A79c78Df/Eb/hlru+VNRvol/lXn1/Lv9k=</latexit> Input dependent {0,1} matrix --> Input adaptivity
50. Input Space Partitioning for Multiple Expressions
51. Input Space Partitioning for Multiple Expressions A CNN performs automatic assignment of distinct linear representation depending on input
52. Expressivity of E-D CNN # of representation # of network elements
53. Expressivity of E-D CNN # of representation # of network elements # of channel
54. Expressivity of E-D CNN # of representation # of network elements # of channel Network depth
55. Expressivity of E-D CNN # of representation # of network elements # of channel Network depth Skipped connection
56. Lipschitz Continuity K = max p Kp, Kp = k ˜ B(zp)B(zp)>k2 <latexit sha1_base64="zV0QFc8bcwR20HLOVcDQeQMOtmY=">AAACIHicbZDLSgMxFIYz9V5voy7dBItQQcpMFeqmUOpGcKNgbaFTh0wmtaGZmZicEWv1Udz4Km5cKKI7fRrTy0KtBxI+/v8ckvMHUnANjvNpZaamZ2bn5heyi0vLK6v22vq5TlJFWY0mIlGNgGgmeMxqwEGwhlSMRIFg9aB7OPDr10xpnsRn0JOsFZHLmLc5JWAk3y4dl72I3PgSH/ty17tKSTggXMbenQdchAxX87e+3BndFx4k0rvzi76dcwrOsPAkuGPIoXGd+PaHFyY0jVgMVBCtm64jodUnCjgV7D7rpZpJQrvkkjUNxiRiutUfLniPt40S4naizIkBD9WfE30Sad2LAtMZEejov95A/M9rptA+aPV5LFNgMR091E4FhgQP0sIhV4yC6BkgVHHzV0w7RBEKJtOsCcH9u/IknBcL7l7BOd3PVarjOObRJtpCeeSiEqqgI3SCaoiiB/SEXtCr9Wg9W2/W+6g1Y41nNtCvsr6+AcwXoYQ=</latexit> z1 <latexit sha1_base64="Ob3+IEXFhF5uWyRIGKNYQ89lNRY=">AAAB6nicbVBNS8NAEJ3Ur1q/qh69LBbBU0lU0GPRi8eK9gPaUDbbTbt0swm7E6GG/gQvHhTx6i/y5r9x2+agrQ8GHu/NMDMvSKQw6LrfTmFldW19o7hZ2tre2d0r7x80TZxqxhsslrFuB9RwKRRvoEDJ24nmNAokbwWjm6nfeuTaiFg94DjhfkQHSoSCUbTS/VPP65UrbtWdgSwTLycVyFHvlb+6/ZilEVfIJDWm47kJ+hnVKJjkk1I3NTyhbEQHvGOpohE3fjY7dUJOrNInYaxtKSQz9fdERiNjxlFgOyOKQ7PoTcX/vE6K4ZWfCZWkyBWbLwpTSTAm079JX2jOUI4toUwLeythQ6opQ5tOyYbgLb68TJpnVe+86t5dVGrXeRxFOIJjOAUPLqEGt1CHBjAYwDO8wpsjnRfn3fmYtxacfOYQ/sD5/AEPZo2k</latexit> zp <latexit sha1_base64="Q3WIlMLDjf+qfP58xUVUuKL5KD4=">AAAB6nicbVBNS8NAEJ3Ur1q/qh69LBbBU0lU0GPRi8eK9gPaUDbbSbt0swm7G6GG/gQvHhTx6i/y5r9x2+agrQ8GHu/NMDMvSATXxnW/ncLK6tr6RnGztLW9s7tX3j9o6jhVDBssFrFqB1Sj4BIbhhuB7UQhjQKBrWB0M/Vbj6g0j+WDGSfoR3QgecgZNVa6f+olvXLFrbozkGXi5aQCOeq98le3H7M0QmmYoFp3PDcxfkaV4UzgpNRNNSaUjegAO5ZKGqH2s9mpE3JilT4JY2VLGjJTf09kNNJ6HAW2M6JmqBe9qfif10lNeOVnXCapQcnmi8JUEBOT6d+kzxUyI8aWUKa4vZWwIVWUGZtOyYbgLb68TJpnVe+86t5dVGrXeRxFOIJjOAUPLqEGt1CHBjAYwDO8wpsjnBfn3fmYtxacfOYQ/sD5/AFu4o3j</latexit> Related to the generalizability Dependent on the Local Lipschitz
57. Optimization Issue • Optimization Landscape • All local minimizers are global minimizers • Ex) overparameterized network • Implicit bias • Specific global minimizers are determine d by the optimization algorithms Q) Nonconvex optimization problem à Can we prove convergence ? o <latexit sha1_base64="4DjPcWEGh5PQIKL0eIUnsOTwihE=">AAAB8XicbVDLSgMxFL1TX7W+qi7dBIvgqsyooCspuHFZwdZiO5RMmmlD8xiSjFCG/oUbF4q49W/c+Tdm2llo64HA4Zx7ybknSjgz1ve/vdLK6tr6RnmzsrW9s7tX3T9oG5VqQltEcaU7ETaUM0lblllOO4mmWEScPkTjm9x/eKLaMCXv7SShocBDyWJGsHXSY09gO4riTE371Zpf92dAyyQoSA0KNPvVr95AkVRQaQnHxnQDP7FhhrVlhNNppZcammAyxkPadVRiQU2YzRJP0YlTBihW2j1p0Uz9vZFhYcxERG4yT2gWvVz8z+umNr4KMyaT1FJJ5h/FKUdWofx8NGCaEssnjmCimcuKyAhrTKwrqeJKCBZPXibts3pwXvfvLmqN66KOMhzBMZxCAJfQgFtoQgsISHiGV3jzjPfivXsf89GSV+wcwh94nz/vApER</latexit> x <latexit sha1_base64="hUTvfx5dXOHFo5uS6vfKoPsIiQ8=">AAAB8XicbVDLSgMxFL1TX7W+qi7dBIvgqsyooCspuHFZwT6wLSWT3mlDM5khyYhl6F+4caGIW//GnX9jpp2Fth4IHM65l5x7/FhwbVz32ymsrK6tbxQ3S1vbO7t75f2Dpo4SxbDBIhGptk81Ci6xYbgR2I4V0tAX2PLHN5nfekSleSTvzSTGXkiHkgecUWOlh25IzcgP0qdpv1xxq+4MZJl4OalAjnq//NUdRCwJURomqNYdz41NL6XKcCZwWuomGmPKxnSIHUslDVH30lniKTmxyoAEkbJPGjJTf2+kNNR6Evp2MkuoF71M/M/rJCa46qVcxolByeYfBYkgJiLZ+WTAFTIjJpZQprjNStiIKsqMLalkS/AWT14mzbOqd1517y4qteu8jiIcwTGcggeXUINbqEMDGEh4hld4c7Tz4rw7H/PRgpPvHMIfOJ8//K+RGg==</latexit> o <latexit sha1_base64="4DjPcWEGh5PQIKL0eIUnsOTwihE=">AAAB8XicbVDLSgMxFL1TX7W+qi7dBIvgqsyooCspuHFZwdZiO5RMmmlD8xiSjFCG/oUbF4q49W/c+Tdm2llo64HA4Zx7ybknSjgz1ve/vdLK6tr6RnmzsrW9s7tX3T9oG5VqQltEcaU7ETaUM0lblllOO4mmWEScPkTjm9x/eKLaMCXv7SShocBDyWJGsHXSY09gO4riTE371Zpf92dAyyQoSA0KNPvVr95AkVRQaQnHxnQDP7FhhrVlhNNppZcammAyxkPadVRiQU2YzRJP0YlTBihW2j1p0Uz9vZFhYcxERG4yT2gWvVz8z+umNr4KMyaT1FJJ5h/FKUdWofx8NGCaEssnjmCimcuKyAhrTKwrqeJKCBZPXibts3pwXvfvLmqN66KOMhzBMZxCAJfQgFtoQgsISHiGV3jzjPfivXsf89GSV+wcwh94nz/vApER</latexit> o <latexit sha1_base64="4DjPcWEGh5PQIKL0eIUnsOTwihE=">AAAB8XicbVDLSgMxFL1TX7W+qi7dBIvgqsyooCspuHFZwdZiO5RMmmlD8xiSjFCG/oUbF4q49W/c+Tdm2llo64HA4Zx7ybknSjgz1ve/vdLK6tr6RnmzsrW9s7tX3T9oG5VqQltEcaU7ETaUM0lblllOO4mmWEScPkTjm9x/eKLaMCXv7SShocBDyWJGsHXSY09gO4riTE371Zpf92dAyyQoSA0KNPvVr95AkVRQaQnHxnQDP7FhhrVlhNNppZcammAyxkPadVRiQU2YzRJP0YlTBihW2j1p0Uz9vZFhYcxERG4yT2gWvVz8z+umNr4KMyaT1FJJ5h/FKUdWofx8NGCaEssnjmCimcuKyAhrTKwrqeJKCBZPXibts3pwXvfvLmqN66KOMhzBMZxCAJfQgFtoQgsISHiGV3jzjPfivXsf89GSV+wcwh94nz/vApER</latexit> x <latexit sha1_base64="hUTvfx5dXOHFo5uS6vfKoPsIiQ8=">AAAB8XicbVDLSgMxFL1TX7W+qi7dBIvgqsyooCspuHFZwT6wLSWT3mlDM5khyYhl6F+4caGIW//GnX9jpp2Fth4IHM65l5x7/FhwbVz32ymsrK6tbxQ3S1vbO7t75f2Dpo4SxbDBIhGptk81Ci6xYbgR2I4V0tAX2PLHN5nfekSleSTvzSTGXkiHkgecUWOlh25IzcgP0qdpv1xxq+4MZJl4OalAjnq//NUdRCwJURomqNYdz41NL6XKcCZwWuomGmPKxnSIHUslDVH30lniKTmxyoAEkbJPGjJTf2+kNNR6Evp2MkuoF71M/M/rJCa46qVcxolByeYfBYkgJiLZ+WTAFTIjJpZQprjNStiIKsqMLalkS/AWT14mzbOqd1517y4qteu8jiIcwTGcggeXUINbqEMDGEh4hld4c7Tz4rw7H/PRgpPvHMIfOJ8//K+RGg==</latexit> o <latexit sha1_base64="4DjPcWEGh5PQIKL0eIUnsOTwihE=">AAAB8XicbVDLSgMxFL1TX7W+qi7dBIvgqsyooCspuHFZwdZiO5RMmmlD8xiSjFCG/oUbF4q49W/c+Tdm2llo64HA4Zx7ybknSjgz1ve/vdLK6tr6RnmzsrW9s7tX3T9oG5VqQltEcaU7ETaUM0lblllOO4mmWEScPkTjm9x/eKLaMCXv7SShocBDyWJGsHXSY09gO4riTE371Zpf92dAyyQoSA0KNPvVr95AkVRQaQnHxnQDP7FhhrVlhNNppZcammAyxkPadVRiQU2YzRJP0YlTBihW2j1p0Uz9vZFhYcxERG4yT2gWvVz8z+umNr4KMyaT1FJJ5h/FKUdWofx8NGCaEssnjmCimcuKyAhrTKwrqeJKCBZPXibts3pwXvfvLmqN66KOMhzBMZxCAJfQgFtoQgsISHiGV3jzjPfivXsf89GSV+wcwh94nz/vApER</latexit> x <latexit sha1_base64="hUTvfx5dXOHFo5uS6vfKoPsIiQ8=">AAAB8XicbVDLSgMxFL1TX7W+qi7dBIvgqsyooCspuHFZwT6wLSWT3mlDM5khyYhl6F+4caGIW//GnX9jpp2Fth4IHM65l5x7/FhwbVz32ymsrK6tbxQ3S1vbO7t75f2Dpo4SxbDBIhGptk81Ci6xYbgR2I4V0tAX2PLHN5nfekSleSTvzSTGXkiHkgecUWOlh25IzcgP0qdpv1xxq+4MZJl4OalAjnq//NUdRCwJURomqNYdz41NL6XKcCZwWuomGmPKxnSIHUslDVH30lniKTmxyoAEkbJPGjJTf2+kNNR6Evp2MkuoF71M/M/rJCa46qVcxolByeYfBYkgJiLZ+WTAFTIjJpZQprjNStiIKsqMLalkS/AWT14mzbOqd1517y4qteu8jiIcwTGcggeXUINbqEMDGEh4hld4c7Tz4rw7H/PRgpPvHMIfOJ8//K+RGg==</latexit> o <latexit sha1_base64="4DjPcWEGh5PQIKL0eIUnsOTwihE=">AAAB8XicbVDLSgMxFL1TX7W+qi7dBIvgqsyooCspuHFZwdZiO5RMmmlD8xiSjFCG/oUbF4q49W/c+Tdm2llo64HA4Zx7ybknSjgz1ve/vdLK6tr6RnmzsrW9s7tX3T9oG5VqQltEcaU7ETaUM0lblllOO4mmWEScPkTjm9x/eKLaMCXv7SShocBDyWJGsHXSY09gO4riTE371Zpf92dAyyQoSA0KNPvVr95AkVRQaQnHxnQDP7FhhrVlhNNppZcammAyxkPadVRiQU2YzRJP0YlTBihW2j1p0Uz9vZFhYcxERG4yT2gWvVz8z+umNr4KMyaT1FJJ5h/FKUdWofx8NGCaEssnjmCimcuKyAhrTKwrqeJKCBZPXibts3pwXvfvLmqN66KOMhzBMZxCAJfQgFtoQgsISHiGV3jzjPfivXsf89GSV+wcwh94nz/vApER</latexit> x <latexit sha1_base64="hUTvfx5dXOHFo5uS6vfKoPsIiQ8=">AAAB8XicbVDLSgMxFL1TX7W+qi7dBIvgqsyooCspuHFZwT6wLSWT3mlDM5khyYhl6F+4caGIW//GnX9jpp2Fth4IHM65l5x7/FhwbVz32ymsrK6tbxQ3S1vbO7t75f2Dpo4SxbDBIhGptk81Ci6xYbgR2I4V0tAX2PLHN5nfekSleSTvzSTGXkiHkgecUWOlh25IzcgP0qdpv1xxq+4MZJl4OalAjnq//NUdRCwJURomqNYdz41NL6XKcCZwWuomGmPKxnSIHUslDVH30lniKTmxyoAEkbJPGjJTf2+kNNR6Evp2MkuoF71M/M/rJCa46qVcxolByeYfBYkgJiLZ+WTAFTIjJpZQprjNStiIKsqMLalkS/AWT14mzbOqd1517y4qteu8jiIcwTGcggeXUINbqEMDGEh4hld4c7Tz4rw7H/PRgpPvHMIfOJ8//K+RGg==</latexit> x <latexit sha1_base64="hUTvfx5dXOHFo5uS6vfKoPsIiQ8=">AAAB8XicbVDLSgMxFL1TX7W+qi7dBIvgqsyooCspuHFZwT6wLSWT3mlDM5khyYhl6F+4caGIW//GnX9jpp2Fth4IHM65l5x7/FhwbVz32ymsrK6tbxQ3S1vbO7t75f2Dpo4SxbDBIhGptk81Ci6xYbgR2I4V0tAX2PLHN5nfekSleSTvzSTGXkiHkgecUWOlh25IzcgP0qdpv1xxq+4MZJl4OalAjnq//NUdRCwJURomqNYdz41NL6XKcCZwWuomGmPKxnSIHUslDVH30lniKTmxyoAEkbJPGjJTf2+kNNR6Evp2MkuoF71M/M/rJCa46qVcxolByeYfBYkgJiLZ+WTAFTIjJpZQprjNStiIKsqMLalkS/AWT14mzbOqd1517y4qteu8jiIcwTGcggeXUINbqEMDGEh4hld4c7Tz4rw7H/PRgpPvHMIfOJ8//K+RGg==</latexit> x <latexit sha1_base64="hUTvfx5dXOHFo5uS6vfKoPsIiQ8=">AAAB8XicbVDLSgMxFL1TX7W+qi7dBIvgqsyooCspuHFZwT6wLSWT3mlDM5khyYhl6F+4caGIW//GnX9jpp2Fth4IHM65l5x7/FhwbVz32ymsrK6tbxQ3S1vbO7t75f2Dpo4SxbDBIhGptk81Ci6xYbgR2I4V0tAX2PLHN5nfekSleSTvzSTGXkiHkgecUWOlh25IzcgP0qdpv1xxq+4MZJl4OalAjnq//NUdRCwJURomqNYdz41NL6XKcCZwWuomGmPKxnSIHUslDVH30lniKTmxyoAEkbJPGjJTf2+kNNR6Evp2MkuoF71M/M/rJCa46qVcxolByeYfBYkgJiLZ+WTAFTIjJpZQprjNStiIKsqMLalkS/AWT14mzbOqd1517y4qteu8jiIcwTGcggeXUINbqEMDGEh4hld4c7Tz4rw7H/PRgpPvHMIfOJ8//K+RGg==</latexit>
58. Benign Optimization Landscape full-rank condition Independent features Nguyen, et al, ICML, 2018
59. Benign Optimization Landscape full-rank condition Independent features Independent features full-rank condition Nguyen, et al, ICML, 2018 Ye et al, ICML, 2019
60. Regularized Recon vs. Deep Recon Diagnosis Classical Regularized Recon (basis engineering) Deep Recon (no basis engineering) Basis Engineering
61. THEORY-DRIVEN CNN DESIGN :some snapshots
62. Deep Beamformer Khan et al, MICCAI, 2019
63. Focused / Plane Wave Ultrasound Imaging Focused Imaging Plane Wave Imaging Couade M, JVDI, 2015 Yoon et al, TMI, 2018
64. Adaptive Beamformer Conventional Beamforming Pipeline  I Q =  zl[n] H(zl)[n] <latexit sha1_base64="Mv7jnJQUjhxiLpE3LDXNSJ47/J8=">AAACSnicbZBPSwMxEMWztWpd/6169BIsQr2UXRX0IhS91JuCtYXuUrLptA3NZpckK9aln8+LJ29+CC8eFPFiWovY1oHA4zdvmMkLE86Udt0XK7eQX1xaLqzYq2vrG5vO1vatilNJoUZjHstGSBRwJqCmmebQSCSQKORQD/sXo379DqRisbjRgwSCiHQF6zBKtEEth/ghdJnIwohoye6Hl9j38TX2QbR/GT7DMy77ocWbIjBe2zeoRwnPqsOSoQcG21PTLafolt1x4XnhTUQRTeqq5Tz77ZimEQhNOVGq6bmJDjIiNaMchrafKkgI7ZMuNI0UJAIVZOMohnjfkDbuxNI8ofGY/p3ISKTUIAqNc3S4mu2N4H+9Zqo7p0HGRJJqEPRnUSflWMd4lCtuMwlU84ERhEpmbsW0RySh2qRvmxC82S/Pi9vDsndUdq+Pi5XzSRwFtIv2UAl56ARVUBVdoRqi6BG9onf0YT1Zb9an9fVjzVmTmR00Vbn8N6l4s94=</latexit> Data specific weights
65. Deep Beamformer Khan et al, MICCAI (accepted), 2019 Adaptive and Compressive Deep Beamformer Conventional Beamforming Pipeline Input partitioning à Data specific representation
66. B-mode Focused Mode Deep Beamformer Khan et al, MICCAI (accepted), 2019
67. Plane Wave Deep Beamformer Khan et al, MICCAI (accepted), 2019
68. k-Space Deep Learning Han et al, IEEE Trans. Medical Imaging (in press), 2019 Lee et al, MRM (in press), 2019
69. Duality between Sparsity and Low-Rankness Ye et al, IEEE Trans. Information Theory, 2017;Jin et al, IEEE TCI, 2017; Lee et al, MRM, 2017
70. Annihilating Filter-based Low-rank Hankel Matrix Approach (ALOHA) Jin KH et al IEEE TCI, 2016, Lee et al, MRM, 2015; Ye et al, IEEE TIT, 2016
71. Single coil MRI Jin KH et al IEEE TCI, 2016 Lee et al, MRM, 2015
72. Parallel MRI Jin KH et al IEEE TCI, 2016 Lee et al, MRM, 2015
73. Optimization Problem Equivalent to search for frame basis to make the nonzero-coefficents sparse
74. : Non-local basis : Local basis Convolution Framelets (Yin et al, SIIMS, 2017) > = I > = I
75. : Non-local basis : Local basis Convolution Framelets (Yin et al, SIIMS, 2017) > = I > = I
76. : Non-local basis : Local basis Convolution Framelets (Yin et al, SIIMS, 2017) > = I > = I
77. ˜ T = I : Non-local basis : Local basis :Nonlocal basis : Local basis Deep Convolutional Framelets (Y, Han, Cha, SIIMS, 2018) ˜ T = I <latexit sha1_base64="nsw6gd0e+4ZO1ci14Ht6KPpot+8=">AAAB/XicbZDLSsNAFIZPvNZ6i5edm8EiuCqJCroRim50V6E3aGKZTCbt0MmFmYlQQ/FV3LhQxK3v4c63cdJmoa0/DHz85xzOmd9LOJPKsr6NhcWl5ZXV0lp5fWNza9vc2W3JOBWENknMY9HxsKScRbSpmOK0kwiKQ4/Ttje8zuvtByoki6OGGiXUDXE/YgEjWGmrZ+47dckcxbhPc7pvoEt02zMrVtWaCM2DXUAFCtV75pfjxyQNaaQIx1J2bStRboaFYoTTcdlJJU0wGeI+7WqMcEilm02uH6Mj7fgoiIV+kUIT9/dEhkMpR6GnO0OsBnK2lpv/1bqpCi7cjEVJqmhEpouClCMVozwK5DNBieIjDZgIpm9FZIAFJkoHVtYh2LNfnofWSdU+rVp3Z5XaVRFHCQ7gEI7BhnOowQ3UoQkEHuEZXuHNeDJejHfjY9q6YBQze/BHxucPPbyUaw==</latexit>
78. ˜ T = I : Non-local basis : Local basis : Pooling : Convolution filters Deep Convolutional Framelets (Y, Han, Cha, SIIMS, 2018) ˜ T = I <latexit sha1_base64="nsw6gd0e+4ZO1ci14Ht6KPpot+8=">AAAB/XicbZDLSsNAFIZPvNZ6i5edm8EiuCqJCroRim50V6E3aGKZTCbt0MmFmYlQQ/FV3LhQxK3v4c63cdJmoa0/DHz85xzOmd9LOJPKsr6NhcWl5ZXV0lp5fWNza9vc2W3JOBWENknMY9HxsKScRbSpmOK0kwiKQ4/Ttje8zuvtByoki6OGGiXUDXE/YgEjWGmrZ+47dckcxbhPc7pvoEt02zMrVtWaCM2DXUAFCtV75pfjxyQNaaQIx1J2bStRboaFYoTTcdlJJU0wGeI+7WqMcEilm02uH6Mj7fgoiIV+kUIT9/dEhkMpR6GnO0OsBnK2lpv/1bqpCi7cjEVJqmhEpouClCMVozwK5DNBieIjDZgIpm9FZIAFJkoHVtYh2LNfnofWSdU+rVp3Z5XaVRFHCQ7gEI7BhnOowQ3UoQkEHuEZXuHNeDJejHfjY9q6YBQze/BHxucPPbyUaw==</latexit> Hd(f) Hd(f) = ˜ T ˜ T C C = T Hd(f) C = T (f ~ ) Encoder: Unlifting: f = (˜C) ~ ⌧(˜ ) convolution pooling un-pooling convolution Decoder:
79. Single Resolution Network Architecture
80. Multi-Resolution Network Architecture
81. ALOHA CNN k-Space Deep Learning Han et al, IEEE TMI,2019
82. K-space Deep Learning (Radial R=6) Ground-truth Acceleration Image learning CS K-space learning Han et al, IEEE TMI (in press)
83. K-space Deep Learning (Radial R=6) Ground-truth Acceleration Image learning CS K-space learning Han et al, IEEE TMI (in press)
84. EPI Ghost Artifact Correction Gx RO G y PE G z SS R F Ghost artifact image In EPI, Gradient is distorted by eddy currents and this causes phase shift Distorted gradient FT Even and odd echo mismatch causes ghost artifact! Phase shift
85. k-space Deep Learning for EPI Ghost Correction Image domain loss L2 loss is calculated on the image domain k-space (with Ghost) e eo o … e e o o … ALOHA IFT e e o o … Coil 1 … coil P Coil 1 … coil P Coil 1 … coil P Neural network k-space (with Ghost) e e o o … e e o o … IFT e e o o … Coil 1 … coil P Coil 1 … coil P Coil 1 … coil P k-space learning Network Input Network Label 34 Lee et al, MRM (in press), 2019
86. 7T EPI result (R=2) ALOHA Ghost image Half ROI learning With Reference PEC-SENSE Proposed (Full ROI) GSR : 10.48% GSR : 9.71% GSR : 15.04% GSR : 8.80% GSR : 4.92% 49 Lee et al, MRM (in press), 2019
87. Improving U-Net Ye et al, SIAM J. Imaging Science, 2018 Han et al, IEEE Trans. Medical Imaging, 2018 Yoo et al, SIAM J. Applied Math, 2019
88. Problem of U-net Pooling does NOT satisfy the frame condition JC Ye et al, SIAM Journal Imaging Sciences, 2018 Y. Han et al, TMI, 2018. ext > ext = I + > 6= I
89. Improving U-net using Deep Conv Framelets • Dual Frame U-net • Tight Frame U-net JC Ye et al, SIAM Journal Imaging Sciences, 2018 Y. Han and J. C. Ye, TMI, 2018
90. U-Net versus Dual Frame U-Net Y. Han and J. C. Ye, TMI, 2018; Yoo et al, SIJAM, 2018
91. Tight-Frame U-Net JC Ye et al, SIAM Journal Imaging Sciences, 2018
92. Denoising: U-Net vs. Tight-Frame U-Net
93. Inpainting: U-Net vs. Tight-Frame U-Net
94. Style Transfer : Power of Tight Frame U-net
99. Summary • Deep learning is a novel signal representation using combinatorial framelets • ReLUs generate multiple linear representation by partitioning the input space • Local Lipschitz controls the global Liptschiz continuity • Skipped connection improves the optimization landscape • Black-box nature of neural networks have been being unveiled.
100. Outlooks q End-to-End AI for radiological imaging Ø From AI-powered image acquisition to diagnosis for clear and rapid radiological imaging Existing AI Solutions: Diagnosis Our future: from acquisition to diagnosis
101. Acknowledgement • Daniel Rueckert (Imperial College) • Florian Knoll (NYU) • Fang Liu (Univ. of Wisconsin) • Mehmet Akcakaya (Univ. of Minnesota) • Dong Liang (SIAT, China) • Dinggang Shen (UNC) • Peder Larson (UCSF) • Grant – NRF of Korea – Ministry of Trade Industry and Energy • Hyunwook Park (KAIST) • Sung-hong Park (KAIST) • Jongho Lee (SNU) • Doshik Hwang (Yonsei Univ) • Won-Jin Moon (KonkukUniv Medical Center) • Eungyeop Kim (Gachon Univ. Medical Center) • Leonard Sunwoo (SNUBH) • Kyuhwan Jung (Vuno)
102. References 1.Ye, Jong Chul, and Woon Kyoung Sung. "Understanding Geometry of Encoder-Decoder CNNs." International Conference on Machine Learning (ICML), 2019. 2.Jong Chul Ye, Yoseob Han and Eunju Cha, "Deep convolutional framelets: a general deep learning framework for inverse problems", SIAM Journal on Imaging Sciences 11(2), 991–1048, 2018. 3. Yoseob Han and Jong Chul Ye,"Framing U-Net via Deep Convolutional Framelets: Application to Sparse-view CT", IEEE Trans. on Medical Imaging, vol. 37, no. 6, pp. 1418-1429, June, 2018. 4. Yoseob Han, Leonard Sunwoo, and Jong Chul Ye, "k-Space Deep Learning for Accelerated MRI", IEEE Trans. on Medical Imaging (in press), 2019 5. Juyoung Lee, Yoseob Han, Jae-Kyun Ryu, Jang-Yeon Park and Jong Chul Ye, "k-Space Deep Learning for Reference-free EPI Ghost Correction", Magnetic Resonance in Medicine (in press), 2019 6. Eunhee Kang, Won Chang, Jaejun Yoo, and Jong Chul Ye,"Deep Convolutional Framelet Denosing for Low-Dose CT via Wavelet Residual Network", IEEE Trans. on Medical Imaging, vol. 37, no.6, pp. 1358-1369, 2018. 7. Eunhee Kang, Junhong Min and Jong Chul Ye, " A Deep Convolutional Neural Network using Directional Wavelets for Low-dose X- ray CT Reconstruction", Medical Physics 44.10 (2017). 8.Jong Chul Ye, Jong Min Kim, Kyong Hwan Jin and Kiryung Lee, "Compressive sampling using annihilating filter-based low-rank interpolation", IEEE Trans. on Information Theory, vol. 63, no. 2, pp.777-801, Feb. 2017. 9. Kyong Hwan Jin, Dongwook Lee, and Jong Chul Ye. "A general framework for compressed sensing and parallel MRI using annihilating filter based low-rank Hankel matrix," IEEE Trans. on Computational Imaging, vol 2, no. 4, pp. 480 - 495, Dec. 2016.