Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Deep Learning for CT Reconstruction

A3d61bc22cd700a92e7d4136a4d29e8f?s=47 Jong Chul Ye
February 24, 2018

Deep Learning for CT Reconstruction

Keynote Talk by Jong Chul Ye, SPIE Medical Imaging 2018, Feb, 24, 2018, Houston, USA

A3d61bc22cd700a92e7d4136a4d29e8f?s=128

Jong Chul Ye

February 24, 2018
Tweet

More Decks by Jong Chul Ye

Other Decks in Research

Transcript

  1. Jong Chul Ye KAIST Bio-Imaging & Signal Processing Lab. Dept.

    Bio & Brain Engineering KAIST, Korea D De ee ep p L Le ea ar rn ni in ng g f fo or r C CT T R Re ec co on ns st tr ru uc ct ti io on n
  2. DEEP LEARNING WAVE FOR RECON

  3. 2

  4. * NMSE is wri9en at the corner. 2 2. .1

    12 20 03 3e e- -1 1 X : Input (48) Ground truth (380th) Han and Ye, arxiv:1611.06391,2016, Jin et al, IEEE TIP, 2017 U-Net for Sparse View CT 6 6. .2 22 27 77 7e e- -3 3 U-Net
  5. Low-dose CT with Adversarial loss

  6. Sparse-view CT with Variational Network Chen et al, “LEARN”, arXiv:1707.09636

  7. CT Filter design using Neural Network Würfl, Tobias, et al.

    2016.
  8. •  Successful demonstraMon of deep learning for various image reconstrucMon

    problems –  Low-dose x-ray CT (Kang et al, Chen et al, Wolterink et al) –  Sparse view CT (Jin et al, Han et al, Adler et al) –  Interior tomography (Han et al) –  Stationary CT for baggage inspection (Han et al) –  CS-MRI (Hammernik et al, Yang et al, Lee et al, Zhu et al) –  US imaging (Yoon et al ) –  Diffuse optical tomography (Yoo et al) •  Advantages –  Very fast reconstruction time –  Significantly improved results Other works
  9. WHY DEEP LEARNING WORKS FOR RECON ? DOES IT CREATE

    ANY ARTIFICIAL FEATURES ?
  10. Why Deep Learning works for recon ? Existing views 1:

    unfolded iteration •  Most prevailing views •  Direct connecMon to sparse recovery –  Cannot explain the filter channels Jin, arXiv:1611.03679
  11. Why Deep Learning works for recon ? Existing views 2:

    generative model •  Image reconstruc5on as a distribu5on matching –  However, difficult to explain the role of black-box network Bora et al, Compressed Sensing using Generative Models, arXiv:1703.03208
  12. Still Unresolved Problems.. •  What is the role of the

    nonlinearity such as rectified linear unit (ReLU) ? •  What is the role of the filter channels in convolutional layer ? •  Why do we need a pooling and unpooling in some architectures ? •  Why do some networks need fully connected layers whereas the others do not ? •  What is the role of by-pass connection or residual network ?
  13. Our Proposal: Deep Learning == Deep Convolutional Framelets •  Ye

    et al, “Deep convolutional framelets: A general deep learning framework for inverse problems”, SIAM Journal Imaging Sciences (in press), 2018.
  14. Matrix Representation of CNN Figure courtesy of Shoieb et al,

    2016
  15. Hankel Matrices

  16. Why we are excited about Hankel matrix ? T -T

    0 n1 -n1 0 * FRI Sampling theory (Ve9erlie et al) and compressed sensing
  17. ︙ ︙ 1 2 3 4 5 6 7 8

    9 -1 0 1 2 3 4 5 6 7 8 9 1 2 3 4 5 6 7 8 9 12 2 3 4 5 6 7 8 9 12 13 3 4 5 6 7 8 9 10 10 10 10 0 11 11 11 1 2 3 4 5 Finite length convolution Matrix Representation * ALOHA : Annihilating filter based LOw rank Hankel matrix Approach * Jin KH et al. IEEE TCI, 2016 * Jin KH et al.,IEEE TIP, 2015 * Ye JC et al. IEEE TIT, 2016 Annihilating filter-based low-rank Hankel matrix
  18. Missing elements can be found by low rank Hankel structured

    matrix compleMon Nuclear norm Projec5on on sampling posi5ons min m kH ( m ) k⇤ subject to P⌦(b) = P⌦( f ) Rank H ( f ) = k * Jin KH et al IEEE TCI, 2016 * Jin KH et al.,IEEE TIP, 2015 * Ye JC et al., IEEE TIT, 2016 m Annihilating filter-based low-rank Hankel matrix
  19. Image Inpainting Experiments 18. APR. 2015. 18 * * Jin

    KH et al.,IEEE TIP, 2015
  20. 19 * Jin KH et al.,IEEE TIP, 2015

  21. Low-rank Hankel matrix in Image

  22. Key Observation Hankel matrix decomposition => Deep Learning

  23. Deep Convolutional Framelets (Y, Han, Cha; 2018) H d( f

    ) H d( f ) = ˜ T ˜ T C C = Φ T H d( f ) C = Φ T ( f ~ ) Encoder: ˜ T = I ˜ = PR(V ) H d( f ) = U ⌃ V T Decoder: f = (˜ Φ C ) ~ ⌧ (˜ ) : Non-local basis : Local basis : Generalized pooling : CNN filters : Frame condition : Projection condition
  24. Deep Convolutional Framelets (Y, Han, Cha; 2018)

  25. Deep Convolutional Framelets (Y, Han,Cha; 2018)

  26. Role of Insufficient Channels ? Truncated channel à Low rank

    Hankel matrix approximation
  27. Compressed Sensing Hankel Structured Matrix Comple7on Deep Learning 26 From

    CS to Deep Learning: Coherent Theme
  28. APPLICATION TO CT RECONSTRUCTION

  29. Low-Dose CT •  To reduce the radiation exposure, sparse-view CT,

    low-dose CT and interior tomography. Sparse-view CT (Down-sampled View) Low-dose CT (Reduced X-ray dose) Interior Tomography (Truncated FOV)
  30. WavResNet for low-dose CT (Kang, Yoo, Y; 2017) E. Kang

    et al, Medical Physics, vol. 44, no. 10, 2017. E. Kang et al, arXiv preprint arXiv:1707.09938
  31. Signal Boosting with Multiple Framelet Expansions

  32. WavResNet results

  33. WavResNet results

  34. MBIR Our latest Result C D WavResNet results

  35. Full Dose Quarter dose

  36. Full Dose WavResNet

  37. Sparse-View CT •  To reduce the radiation exposure, sparse-view CT,

    low-dose CT and interior tomography. Sparse-view CT (Down-sampled View) Low-dose CT (Reduced X-ray dose) Interior Tomography (Truncated FOV)
  38. Multi-resolution Analysis & Receptive Fields

  39. Problem of U-net Pooling does NOT satisfy the frame condition

    JC Ye et al, SIAM Journal Imaging Sciences, 2018 Y. Han and J. C. Ye, arXiv preprint arXiv:1708.08333, 2017. ext > ext = I + > 6 = I
  40. Improving U-net using Deep Conv Framelets •  Dual Frame U-net

    •  Tight Frame U-net JC Ye et al, SIAM Journal Imaging Sciences, 2018 Y. Han and J. C. Ye, arXiv preprint arXiv:1708.08333, 2017.
  41. U-Net versus Dual Frame U-Net

  42. Tight-Frame U-Net JC Ye et al, SIAM Journal Imaging Sciences,

    2018
  43. 90 view recon U-Net vs. Tight-Frame U-Net •  JC Ye

    et al, SIAM Journal Imaging Sciences, 2018 •  Y. Han and J. C. Ye, arXiv preprint arXiv:1708.08333, 2017.
  44. None
  45. None
  46. •  Figures from internet 9 View Dual Energy CT for

    Baggage Screening
  47. 9 View Dual Energy CT for Baggage Screening

  48. 1st view 2nd view 3rd view 4th view 5th view

    6th view 7th view 8th view 9th view
  49. FBP

  50. TV

  51. Ours

  52. Interior Tomography for ROI Reconstruction •  To reduce the radiation

    exposure, sparse-view CT, low-dose CT and interior tomography. Sparse-view CT (Down-sampled View) Low-dose CT (Reduced X-ray dose) Interior Tomography (Truncated FOV)
  53. Deep Learning Interior Tomography Han et al, arXiv preprint arXiv:1712.10248,

    (2017).
  54. Ground Truth FBP

  55. TV Chord Line Ours 8~10 dB gain

  56. Ground Truth Chord Line TV Ours

  57. Conclusions: •  Deep learning has demonstrated significant performance improvement in

    CT reconstruction problems •  Deep convolutional framelets is a mathematical framework to understand deep learning for inverse problems –  Novel data-driven signal representation –  Structured low-rank shrinkage by controlling the no. of filter channels –  Frame condition is important •  Our analysis shows that “correctly designed” deep network is NOT a black-box and is SAFE for medical imaging reconstruction
  58. THANK YOU