Pro Yearly is on sale from $80 to $50! »

Deep Convolutional Framelets: A general deep learning framework for inverse problems

A3d61bc22cd700a92e7d4136a4d29e8f?s=47 Jong Chul Ye
September 16, 2018

Deep Convolutional Framelets: A general deep learning framework for inverse problems

Keynote Talk by Jong Chul Ye, MLMIR - Machine Learning for Medical Image Reconstruction, MICCAI Workshop, Sept. 16th, 2018, Granada, Spain

A3d61bc22cd700a92e7d4136a4d29e8f?s=128

Jong Chul Ye

September 16, 2018
Tweet

Transcript

  1. Jong Chul Ye Deep Convolutional Framelets: A general deep learning

    framework for inverse problems Bio-Imaging, Signal Processing, & Learning (BISPL) Dept. Bio & Brain Engineering Dept. Mathematical Sciences KAIST, Korea
  2. • Successful demonstration of deep learning for various image reconstruction

    problems – Low-dose x-ray CT (Kang et al, Chen et al, Wolterink et al, Ye et al) – Sparse view CT (Jin et al, Han et al, Adler et al) – Interior tomography (Han et al) – Stationary CT for baggage inspection (Han et al) – CS-MRI (Hammernik et al, Schlemper et al, Yang et al, Lee et al, Zhu et al) – US imaging (Yoon et al ) – Diffuse optical tomography (Yoo et al) – Elastic tomography (Yoo et al) – Optical diffraction tomography (Kamilov et al) – etc • Advantages – Very fast reconstruction time – Significantly improved results Deep Learning for Inverse Problems
  3. None
  4. None
  5. None
  6. None
  7. 6

  8. WHY DEEP LEARNING WORKS FOR RECON ? DOES IT CREATE

    ANY ARTIFICIAL FEATURES ?
  9. Too Simple to Analyze..? Convolution & pooling à stone age

    tools of signal processing What do they do ?
  10. Paradox and Mysteries Residual Network Clean image Standard Network Zhang,

    K., et al, IEEE TIP, 2017.
  11. Dark Age of Applied Mathematics ?

  12. • What is the role of the nonlinearity such as

    rectified linear unit (ReLU) ? • Why do we need a pooling and unpooling in some architectures ? • Why do some networks need fully connected layers whereas the others do not ? • What is the role of by-pass connection or residual network ? • What is the role of the filter channels in convolutional layer ? Many Mysteries…
  13. Our Proposal: Deep Learning == Deep Convolutional Framelets • Ye

    et al, “Deep convolutional framelets: A general deep learning framework for inverse problems”, SIAM Journal Imaging Sciences, 11(2), 991-1048, 2018.
  14. Matrix Representation of CNN Figure courtesy of Shoieb et al,

    2016
  15. Hankel Matrix: Linear Lifting to Higher Dimensional Space

  16. Why we are excited about Hankel matrix ? T -T

    0 n1 -n1 0 * FRI Sampling theory (Vetterlie et al) and compressed sensing
  17.   1 2 3 4 5 6 7 8

    9 -1 0 1 2 3 4 5 6 7 8 9 1 2 3 4 5 6 7 8 9 12 2 3 4 5 6 7 8 9 12 13 3 4 5 6 7 8 9 10 10 10 10 0 11 11 11 1 2 3 4 5 Finite length convolution Matrix Representation * ALOHA : Annihilating filter based LOw rank Hankel matrix Approach * Jin KH et al. IEEE TCI, 2016 * Jin KH et al.,IEEE TIP, 2015 * Ye JC et al. IEEE TIT, 2016 Annihilating filter-based low-rank Hankel matrix
  18. Missing elements can be found by low rank Hankel structured

    matrix completion Nuclear norm Projection on sampling positions min m kH(m)k⇤ subject to P⌦(b) = P⌦(f) RankH(f) = k * Jin KH et al IEEE TCI, 2016 * Jin KH et al.,IEEE TIP, 2015 * Ye JC et al., IEEE TIT, 2016 m Annihilating filter-based low-rank Hankel matrix
  19. 18. APR. 2015. 18 * Image Inpainting Results Jin et

    al, IEEE TIP, 2015
  20. 19 Jin et al, IEEE TIP, 2015

  21. Key Observation Data-Driven Hankel matrix decomposition => Deep Learning •

    Ye et al, “Deep convolutional framelets: A general deep learning framework for inverse problems”, SIAM Journal Imaging Sciences, 11(2), 991-1048, 2018.
  22. Hd(f) = U⌃V T : Non-local basis : Local basis

    Convolution Framelets (Yin et al; 2017) > = I > = I Hd(f)
  23. Hd(f) Hd(f) = ˜ T ˜ T C C =

    T Hd(f) C = T (f ~ ) Encoder: ˜ T = I ˜ = PR(V ) Hd(f) = U⌃V T Unlifting: f = (˜C) ~ ⌧(˜ ) : Non-local basis : Local basis : Frame condition : rank condition convolution pooling un-pooling convolution : User-defined pooling : Learnable filters Hpi (gi) = X k,l [Ci]kl e Bkl i Decoder: Deep Convolutional Framelets (Y, Han, Cha; 2018)
  24. Single Resolution Network Architecture

  25. Multi-Resolution Network Architecture

  26. Conic fi [Ci]kl 0 Hpi (gi) = X k,l [Ci]kl

    e Bkl i Hpi (fi) ' Linear Lifting Geometry of CNN gi Linear Un-lifting Ci(fi) Ci(fi) 0 i ⇣i ⇣ e i ⌘
  27. fi [Ci]kl 0 Hpi (gi) = X k,l [Ci]kl e

    Bkl i Hpi (fi) ' Lifting Geometry of Residual CNN Ci(fi) Ci(fi) 0 i ⇣i ⇣ e i ⌘ gi Un-lifting
  28. Deep CNN Lifting Un-lifting Conic Lifting Un-lifting Lifting Un-lifting

  29. fi Nonlinear Lifting to Feature space Comparison with Kernel PCA

    gi Nonlinear Pre-Image calculation (fi) <latexit sha1_base64="E9VdeouKNx3eJ5UDAMrvyn9icnU=">AAAB+nicbVBNS8NAFHzxs9avWI9egkWol5KIoN6KXjxWMLbQhLDZbtqlm03Y3Ygl5K948aDi1V/izX/jps1BWwcWhpn3eLMTpoxKZdvfxsrq2vrGZm2rvr2zu7dvHjQeZJIJTFycsET0QyQJo5y4iipG+qkgKA4Z6YWTm9LvPRIhacLv1TQlfoxGnEYUI6WlwGx43TFteTFS4zDKoyKgp4HZtNv2DNYycSrShArdwPzyhgnOYsIVZkjKgWOnys+RUBQzUtS9TJIU4QkakYGmHMVE+vkse2GdaGVoRYnQjytrpv7eyFEs5TQO9WQZUi56pfifN8hUdOnnlKeZIhzPD0UZs1RilUVYQyoIVmyqCcKC6qwWHiOBsNJ11XUJzuKXl4l71r5qO3fnzc511UYNjuAYWuDABXTgFrrgAoYneIZXeDMK48V4Nz7moytGtXMIf2B8/gD8DZP1</latexit> <latexit sha1_base64="E9VdeouKNx3eJ5UDAMrvyn9icnU=">AAAB+nicbVBNS8NAFHzxs9avWI9egkWol5KIoN6KXjxWMLbQhLDZbtqlm03Y3Ygl5K948aDi1V/izX/jps1BWwcWhpn3eLMTpoxKZdvfxsrq2vrGZm2rvr2zu7dvHjQeZJIJTFycsET0QyQJo5y4iipG+qkgKA4Z6YWTm9LvPRIhacLv1TQlfoxGnEYUI6WlwGx43TFteTFS4zDKoyKgp4HZtNv2DNYycSrShArdwPzyhgnOYsIVZkjKgWOnys+RUBQzUtS9TJIU4QkakYGmHMVE+vkse2GdaGVoRYnQjytrpv7eyFEs5TQO9WQZUi56pfifN8hUdOnnlKeZIhzPD0UZs1RilUVYQyoIVmyqCcKC6qwWHiOBsNJ11XUJzuKXl4l71r5qO3fnzc511UYNjuAYWuDABXTgFrrgAoYneIZXeDMK48V4Nz7moytGtXMIf2B8/gD8DZP1</latexit> <latexit sha1_base64="E9VdeouKNx3eJ5UDAMrvyn9icnU=">AAAB+nicbVBNS8NAFHzxs9avWI9egkWol5KIoN6KXjxWMLbQhLDZbtqlm03Y3Ygl5K948aDi1V/izX/jps1BWwcWhpn3eLMTpoxKZdvfxsrq2vrGZm2rvr2zu7dvHjQeZJIJTFycsET0QyQJo5y4iipG+qkgKA4Z6YWTm9LvPRIhacLv1TQlfoxGnEYUI6WlwGx43TFteTFS4zDKoyKgp4HZtNv2DNYycSrShArdwPzyhgnOYsIVZkjKgWOnys+RUBQzUtS9TJIU4QkakYGmHMVE+vkse2GdaGVoRYnQjytrpv7eyFEs5TQO9WQZUi56pfifN8hUdOnnlKeZIhzPD0UZs1RilUVYQyoIVmyqCcKC6qwWHiOBsNJ11XUJzuKXl4l71r5qO3fnzc511UYNjuAYWuDABXTgFrrgAoYneIZXeDMK48V4Nz7moytGtXMIf2B8/gD8DZP1</latexit> <latexit sha1_base64="E9VdeouKNx3eJ5UDAMrvyn9icnU=">AAAB+nicbVBNS8NAFHzxs9avWI9egkWol5KIoN6KXjxWMLbQhLDZbtqlm03Y3Ygl5K948aDi1V/izX/jps1BWwcWhpn3eLMTpoxKZdvfxsrq2vrGZm2rvr2zu7dvHjQeZJIJTFycsET0QyQJo5y4iipG+qkgKA4Z6YWTm9LvPRIhacLv1TQlfoxGnEYUI6WlwGx43TFteTFS4zDKoyKgp4HZtNv2DNYycSrShArdwPzyhgnOYsIVZkjKgWOnys+RUBQzUtS9TJIU4QkakYGmHMVE+vkse2GdaGVoRYnQjytrpv7eyFEs5TQO9WQZUi56pfifN8hUdOnnlKeZIhzPD0UZs1RilUVYQyoIVmyqCcKC6qwWHiOBsNJ11XUJzuKXl4l71r5qO3fnzc511UYNjuAYWuDABXTgFrrgAoYneIZXeDMK48V4Nz7moytGtXMIf2B8/gD8DZP1</latexit> C = 1 N N X i=1 (fi) >(fi) <latexit sha1_base64="LKnrl556MzHt0DAUIt0n5VZ7wPI=">AAACK3icbZDLSsNAFIYnXmu9RV26GSxC3ZREBHVRKHbjqlQwttCkYTKdtEMnF2YmQgl5IDe+iiAurLj1PZy0WdjWHwZ+vnMOc87vxYwKaRhTbW19Y3Nru7RT3t3bPzjUj46fRJRwTCwcsYh3PSQIoyGxJJWMdGNOUOAx0vHGzbzeeSZc0Ch8lJOYOAEahtSnGEmFXL3ZrEPb5winZpa2MlskgZvSupn1W3Z7RKt2gOTI81M/c+lFTvq2jOJF7OoVo2bMBFeNWZgKKNR29Xd7EOEkIKHEDAnRM41YOinikmJGsrKdCBIjPEZD0lM2RAERTjo7NoPnigygH3H1Qgln9O9EigIhJoGnOvMlxXIth//Veon0b5yUhnEiSYjnH/kJgzKCeXJwQDnBkk2UQZhTtSvEI6SikyrfsgrBXD551ViXtdua+XBVadwVaZTAKTgDVWCCa9AA96ANLIDBC3gDn2CqvWof2pf2PW9d04qZE7Ag7ecX5JCoQA==</latexit> <latexit sha1_base64="LKnrl556MzHt0DAUIt0n5VZ7wPI=">AAACK3icbZDLSsNAFIYnXmu9RV26GSxC3ZREBHVRKHbjqlQwttCkYTKdtEMnF2YmQgl5IDe+iiAurLj1PZy0WdjWHwZ+vnMOc87vxYwKaRhTbW19Y3Nru7RT3t3bPzjUj46fRJRwTCwcsYh3PSQIoyGxJJWMdGNOUOAx0vHGzbzeeSZc0Ch8lJOYOAEahtSnGEmFXL3ZrEPb5winZpa2MlskgZvSupn1W3Z7RKt2gOTI81M/c+lFTvq2jOJF7OoVo2bMBFeNWZgKKNR29Xd7EOEkIKHEDAnRM41YOinikmJGsrKdCBIjPEZD0lM2RAERTjo7NoPnigygH3H1Qgln9O9EigIhJoGnOvMlxXIth//Veon0b5yUhnEiSYjnH/kJgzKCeXJwQDnBkk2UQZhTtSvEI6SikyrfsgrBXD551ViXtdua+XBVadwVaZTAKTgDVWCCa9AA96ANLIDBC3gDn2CqvWof2pf2PW9d04qZE7Ag7ecX5JCoQA==</latexit> <latexit sha1_base64="LKnrl556MzHt0DAUIt0n5VZ7wPI=">AAACK3icbZDLSsNAFIYnXmu9RV26GSxC3ZREBHVRKHbjqlQwttCkYTKdtEMnF2YmQgl5IDe+iiAurLj1PZy0WdjWHwZ+vnMOc87vxYwKaRhTbW19Y3Nru7RT3t3bPzjUj46fRJRwTCwcsYh3PSQIoyGxJJWMdGNOUOAx0vHGzbzeeSZc0Ch8lJOYOAEahtSnGEmFXL3ZrEPb5winZpa2MlskgZvSupn1W3Z7RKt2gOTI81M/c+lFTvq2jOJF7OoVo2bMBFeNWZgKKNR29Xd7EOEkIKHEDAnRM41YOinikmJGsrKdCBIjPEZD0lM2RAERTjo7NoPnigygH3H1Qgln9O9EigIhJoGnOvMlxXIth//Veon0b5yUhnEiSYjnH/kJgzKCeXJwQDnBkk2UQZhTtSvEI6SikyrfsgrBXD551ViXtdua+XBVadwVaZTAKTgDVWCCa9AA96ANLIDBC3gDn2CqvWof2pf2PW9d04qZE7Ag7ecX5JCoQA==</latexit> <latexit sha1_base64="LKnrl556MzHt0DAUIt0n5VZ7wPI=">AAACK3icbZDLSsNAFIYnXmu9RV26GSxC3ZREBHVRKHbjqlQwttCkYTKdtEMnF2YmQgl5IDe+iiAurLj1PZy0WdjWHwZ+vnMOc87vxYwKaRhTbW19Y3Nru7RT3t3bPzjUj46fRJRwTCwcsYh3PSQIoyGxJJWMdGNOUOAx0vHGzbzeeSZc0Ch8lJOYOAEahtSnGEmFXL3ZrEPb5winZpa2MlskgZvSupn1W3Z7RKt2gOTI81M/c+lFTvq2jOJF7OoVo2bMBFeNWZgKKNR29Xd7EOEkIKHEDAnRM41YOinikmJGsrKdCBIjPEZD0lM2RAERTjo7NoPnigygH3H1Qgln9O9EigIhJoGnOvMlxXIth//Veon0b5yUhnEiSYjnH/kJgzKCeXJwQDnBkk2UQZhTtSvEI6SikyrfsgrBXD551ViXtdua+XBVadwVaZTAKTgDVWCCa9AA96ANLIDBC3gDn2CqvWof2pf2PW9d04qZE7Ag7ecX5JCoQA==</latexit> PCA of • Nonlinear lifting & unlifting • Deterministic kernel • Difficulty in multilevel extension
  30. APPLICATION-DRIVEN EVIDENCES

  31. Problem of U-net Pooling does NOT satisfy the frame condition

    JC Ye et al, SIAM Journal Imaging Sciences, 2018 Y. Han et al, TMI, 2018. ext > ext = I + > 6= I
  32. Improving U-net using Deep Conv Framelets • Dual Frame U-net

    • Tight Frame U-net JC Ye et al, SIAM Journal Imaging Sciences, 2018 Y. Han and J. C. Ye, TMI, 2018
  33. U-Net versus Dual Frame U-Net

  34. Tight-Frame U-Net JC Ye et al, SIAM Journal Imaging Sciences,

    2018
  35. Denoising: U-Net vs. Tight-Frame U-Net

  36. Inpainting: U-Net vs. Tight-Frame U-Net

  37. Low-Dose CT • To reduce the radiation exposure, sparse-view CT,

    low-dose CT and interior tomography. Sparse-view CT (Down-sampled View) Low-dose CT (Reduced X-ray dose) Interior Tomography (Truncated FOV)
  38. WavResNet for low-dose CT (Kang, Yoo, Y; 2017)

  39. Signal Boosting with Multiple Framelet Expansions

  40. WavResNet results (Kang et al, TMI, 2018)

  41. WavResNet results (Kang et al, TMI, 2018)

  42. MBIR Our latest Result C D WavResNet results

  43. K-space Deep Learning for Accelerated MRI Han, Y., & Ye,

    J. C. (2018). k-Space Deep Learning for Accelerated MRI. arXiv preprint arXiv:1805.03779. Conventional Image Domain Learning
  44. K-space Deep Learning for Accelerated MRI Deep Neural Network IFT

    Han, Y., & Ye, J. C. (2018). k-Space Deep Learning for Accelerated MRI. arXiv preprint arXiv:1805.03779. Proposed k-space Deep Learning ALOHA : k-space interpolation à k-space interpolation using deep learning ? Yes
  45. ALOHA for Compressed Sensing MRI ALOHA: Annihilating filter-based low-rank Hankel

    matrix approach • Jin KH et al IEEE TCI, 2016 • Lee et al, MRM, 2015
  46. Single coil static MRI

  47. Parallel MRI

  48. ALOHA CNN

  49. k-Space Deep Learning for Accelerated MRI Han et al, arXiv:1805.03779

    ~3dB gain
  50. Improved Time-Resolved MRA using k-Space Deep Learning Eunju Cha ,

    Eung Yeop Kim and Jong Chul Ye 1 1 2 Dept. of Bio and Brain Engineering, KAIST, Dept. of Radiology, Gachon University Gil Mdeical Center 1 2 Motivation Ø To cover k-space data at different rate Ø Regular sampling pattern following view sharing of several temporal frames • Reconstruction using GRAPPA TWIST Fixed spatial resolution Limited temporal resolution How to reconstruct?
  51. Improved Time-Resolved MRA using k-Space Deep Learning Research Goal Ø

    To improve temporal resolution of TWIST imaging using deep k-space learning Ø To generate multiple reconstruction results with various spatial and temporal resolution using one network VS = 5 VS = 2 CNN
  52. k-Space Deep Learning for Parallel MRI Cha et al, arXiv:1806.00806

  53. WHAT IF WE DON’T HAVE REFERENCE ?

  54. Semi-Supervised Learning for low-dose CT 54 • Multiphase Cardiac CT

    denoising – Phase 1, 2: low-dose, Phase 3 ~ 10: normal dose – Goal: dynamic changes of heart structure – No reference available Kang et al, arXiv:1806.09748
  55. 55 • Cardiac CT denoising – Cycle Consistent Adversarial Denoising

    Network for Multiphase Coronary CT Angiography Semi-supervised Learning using Cyclic-GAN
  56. 56 Input: phase 1 Denoised output Target: phase 8 Input-

    output
  57. fbp admire recon 139 fbp-target input - admire input -

    recon AMC002_20180903 FBP(Phase1) FBP(Phase8) RECON ADMIRE Phase1 - ADMIRE Phase1 - RECON
  58. • Figures from internet 9 View CT for Baggage Screening

  59. 9 View CT for Baggage Screening

  60. 1st view 2nd view 3rd view 4th view 5th view

    6th view 7th view 8th view 9th view
  61. Semi-Supervised High Resolution View Synthesis Han et al, arXiv preprint

    arXiv:1712.10248, (2017). CT meeting 2018
  62. Semi-Supervised High Resolution View Synthesis 128 256 64 128 6

    4 256 256 512 512 512 1024 512 1024 512 256 512 256128 • Key idea • Training with measured views • Inference with non-measured views
  63. FBP

  64. TV

  65. Ours

  66. Summary • Deep learning for inverse problems • Significant performance

    gain • Has becomes mainstream topics • Deep convolutional framelets: • A new mathematical tool for understanding deep neural network for inverse problems • Biomedical image reconstruction • Key application for machine learning • Semi-supervised learning • New opportunities
  67. math Muchas gracias !