Deep Learning for CT Reconstruction

Keynote Talk by Jong Chul Ye, SPIE Medical Imaging 2018, Feb, 24, 2018, Houston, USA

Transcript

1. Jong Chul Ye KAIST Bio-Imaging & Signal Processing Lab. Dept. Bio & Brain Engineering KAIST, Korea D De ee ep p L Le ea ar rn ni in ng g f fo or r C CT T R Re ec co on ns st tr ru uc ct ti io on n
2. DEEP LEARNING WAVE FOR RECON
4. * NMSE is wri9en at the corner. 2 2. .1 12 20 03 3e e- -1 1 X : Input (48) Ground truth (380th) Han and Ye, arxiv:1611.06391,2016, Jin et al, IEEE TIP, 2017 U-Net for Sparse View CT 6 6. .2 22 27 77 7e e- -3 3 U-Net
5. Low-dose CT with Adversarial loss
6. Sparse-view CT with Variational Network Chen et al, “LEARN”, arXiv:1707.09636
7. CT Filter design using Neural Network Würfl, Tobias, et al. 2016.
8. •  Successful demonstraMon of deep learning for various image reconstrucMon problems –  Low-dose x-ray CT (Kang et al, Chen et al, Wolterink et al) –  Sparse view CT (Jin et al, Han et al, Adler et al) –  Interior tomography (Han et al) –  Stationary CT for baggage inspection (Han et al) –  CS-MRI (Hammernik et al, Yang et al, Lee et al, Zhu et al) –  US imaging (Yoon et al ) –  Diffuse optical tomography (Yoo et al) •  Advantages –  Very fast reconstruction time –  Significantly improved results Other works
9. WHY DEEP LEARNING WORKS FOR RECON ? DOES IT CREATE ANY ARTIFICIAL FEATURES ?
10. Why Deep Learning works for recon ? Existing views 1: unfolded iteration •  Most prevailing views •  Direct connecMon to sparse recovery –  Cannot explain the filter channels Jin, arXiv:1611.03679
11. Why Deep Learning works for recon ? Existing views 2: generative model •  Image reconstruc5on as a distribu5on matching –  However, difficult to explain the role of black-box network Bora et al, Compressed Sensing using Generative Models, arXiv:1703.03208
12. Still Unresolved Problems.. •  What is the role of the nonlinearity such as rectified linear unit (ReLU) ? •  What is the role of the filter channels in convolutional layer ? •  Why do we need a pooling and unpooling in some architectures ? •  Why do some networks need fully connected layers whereas the others do not ? •  What is the role of by-pass connection or residual network ?
13. Our Proposal: Deep Learning == Deep Convolutional Framelets •  Ye et al, “Deep convolutional framelets: A general deep learning framework for inverse problems”, SIAM Journal Imaging Sciences (in press), 2018.
14. Matrix Representation of CNN Figure courtesy of Shoieb et al, 2016
15. Hankel Matrices
16. Why we are excited about Hankel matrix ? T -T 0 n1 -n1 0 * FRI Sampling theory (Ve9erlie et al) and compressed sensing
17. ︙ ︙ 1 2 3 4 5 6 7 8 9 -1 0 1 2 3 4 5 6 7 8 9 1 2 3 4 5 6 7 8 9 12 2 3 4 5 6 7 8 9 12 13 3 4 5 6 7 8 9 10 10 10 10 0 11 11 11 1 2 3 4 5 Finite length convolution Matrix Representation * ALOHA : Annihilating filter based LOw rank Hankel matrix Approach * Jin KH et al. IEEE TCI, 2016 * Jin KH et al.,IEEE TIP, 2015 * Ye JC et al. IEEE TIT, 2016 Annihilating filter-based low-rank Hankel matrix
18. Missing elements can be found by low rank Hankel structured matrix compleMon Nuclear norm Projec5on on sampling posi5ons min m kH ( m ) k⇤ subject to P⌦(b) = P⌦( f ) Rank H ( f ) = k * Jin KH et al IEEE TCI, 2016 * Jin KH et al.,IEEE TIP, 2015 * Ye JC et al., IEEE TIT, 2016 m Annihilating filter-based low-rank Hankel matrix
19. Image Inpainting Experiments 18. APR. 2015. 18 * * Jin KH et al.,IEEE TIP, 2015
20. 19 * Jin KH et al.,IEEE TIP, 2015
21. Low-rank Hankel matrix in Image
22. Key Observation Hankel matrix decomposition => Deep Learning
23. Deep Convolutional Framelets (Y, Han, Cha; 2018) H d( f ) H d( f ) = ˜ T ˜ T C C = Φ T H d( f ) C = Φ T ( f ~ ) Encoder: ˜ T = I ˜ = PR(V ) H d( f ) = U ⌃ V T Decoder: f = (˜ Φ C ) ~ ⌧ (˜ ) : Non-local basis : Local basis : Generalized pooling : CNN filters : Frame condition : Projection condition
24. Deep Convolutional Framelets (Y, Han, Cha; 2018)
25. Deep Convolutional Framelets (Y, Han,Cha; 2018)
26. Role of Insufficient Channels ? Truncated channel à Low rank Hankel matrix approximation
27. Compressed Sensing Hankel Structured Matrix Comple7on Deep Learning 26 From CS to Deep Learning: Coherent Theme
28. APPLICATION TO CT RECONSTRUCTION
29. Low-Dose CT •  To reduce the radiation exposure, sparse-view CT, low-dose CT and interior tomography. Sparse-view CT (Down-sampled View) Low-dose CT (Reduced X-ray dose) Interior Tomography (Truncated FOV)
30. WavResNet for low-dose CT (Kang, Yoo, Y; 2017) E. Kang et al, Medical Physics, vol. 44, no. 10, 2017. E. Kang et al, arXiv preprint arXiv:1707.09938
31. Signal Boosting with Multiple Framelet Expansions
32. WavResNet results
33. WavResNet results
34. MBIR Our latest Result C D WavResNet results
35. Full Dose Quarter dose
36. Full Dose WavResNet
37. Sparse-View CT •  To reduce the radiation exposure, sparse-view CT, low-dose CT and interior tomography. Sparse-view CT (Down-sampled View) Low-dose CT (Reduced X-ray dose) Interior Tomography (Truncated FOV)
38. Multi-resolution Analysis & Receptive Fields
39. Problem of U-net Pooling does NOT satisfy the frame condition JC Ye et al, SIAM Journal Imaging Sciences, 2018 Y. Han and J. C. Ye, arXiv preprint arXiv:1708.08333, 2017. ext > ext = I + > 6 = I
40. Improving U-net using Deep Conv Framelets •  Dual Frame U-net •  Tight Frame U-net JC Ye et al, SIAM Journal Imaging Sciences, 2018 Y. Han and J. C. Ye, arXiv preprint arXiv:1708.08333, 2017.
41. U-Net versus Dual Frame U-Net
42. Tight-Frame U-Net JC Ye et al, SIAM Journal Imaging Sciences, 2018
43. 90 view recon U-Net vs. Tight-Frame U-Net •  JC Ye et al, SIAM Journal Imaging Sciences, 2018 •  Y. Han and J. C. Ye, arXiv preprint arXiv:1708.08333, 2017.
46. •  Figures from internet 9 View Dual Energy CT for Baggage Screening
47. 9 View Dual Energy CT for Baggage Screening
48. 1st view 2nd view 3rd view 4th view 5th view 6th view 7th view 8th view 9th view
49. FBP
50. TV
51. Ours
52. Interior Tomography for ROI Reconstruction •  To reduce the radiation exposure, sparse-view CT, low-dose CT and interior tomography. Sparse-view CT (Down-sampled View) Low-dose CT (Reduced X-ray dose) Interior Tomography (Truncated FOV)
53. Deep Learning Interior Tomography Han et al, arXiv preprint arXiv:1712.10248, (2017).
54. Ground Truth FBP
55. TV Chord Line Ours 8~10 dB gain
56. Ground Truth Chord Line TV Ours
57. Conclusions: •  Deep learning has demonstrated significant performance improvement in CT reconstruction problems •  Deep convolutional framelets is a mathematical framework to understand deep learning for inverse problems –  Novel data-driven signal representation –  Structured low-rank shrinkage by controlling the no. of filter channels –  Frame condition is important •  Our analysis shows that “correctly designed” deep network is NOT a black-box and is SAFE for medical imaging reconstruction
58. THANK YOU