Overview of Machine Learning Methods for Reconstruction of Imaging Data

A3d61bc22cd700a92e7d4136a4d29e8f?s=47 Jong Chul Ye
October 26, 2018

Overview of Machine Learning Methods for Reconstruction of Imaging Data

Keynote Talk, ​ISMRM Workshop on Machine Learning, Part II, Oct 26, 2018, Washington DC, USA

A3d61bc22cd700a92e7d4136a4d29e8f?s=128

Jong Chul Ye

October 26, 2018
Tweet

Transcript

  1. Overview of Machine Learning Methods for Reconstruction of Imaging Data

    Jong Chul Ye, Ph.D Endowed Chair Professor Bio-Imaging, Signal Processing and Learning (BISPL) Group Dept. Bio & Brain Engineering Dept. Mathematical Science KAIST, Korea
  2. Unmet Needs in Medical Imaging q Medical imaging is essential

    for diagnosis of disease q Imaging systems with lower-dose of ionizing radiation ü Increasing risk of radiation dose from CT, PET, etc q Reducing contrast agents without losing diagnostic accuracy ü Severe side-effects from iodine, gadolinium contrast agents q Increasing the throughput of MR scanning ü Current MR scan time : ~ 40~50min/patient
  3. 3 The Solution q Deep learning-based recon • High Quality,

    Fast Reconstruction • Vendor-oriented Neural Network Training: a must-buy for customer • AI-driven hardware design: US systems, industrial CT • Solution Examples: • AI-powered low-dose CT reconstruction • AI-powered contrast synthesis for contrast agent reduction • AI-powered accelerated MRI • AI-powered US for low-power, ultrafast US imaging • AI-powered industrial CT
  4. None
  5. None
  6. None
  7. GANCS Mardani et al, IEEE TMI 2018

  8. None
  9. None
  10. QSMNet Yoon et al, NeuroImage, 2018

  11. 1 1

  12. WHY DEEP LEARNING WORKS FOR RECON ? DOES IT CREATE

    ANY ARTIFICIAL FEATURES ?
  13. WHAT IF WE DON’T HAVE LABEL DATA ?

  14. Too Simple to Analyze..? Convolution & pooling à stone age

    tools of signal processing What do they do ?
  15. Paradox and Mysteries Residual Network Clean image Standard Network Zhang,

    K., et al, IEEE TIP, 2017.
  16. Clue: Matrix Representation of CNN Figure courtesy of Shoieb et

    al, 2016
  17. Hankel Matrix: Linear Lifting to Higher Dimensional Space

  18. Missing elements can be found by low rank Hankel structured

    matrix completion Nuclear norm Projection on sampling positions min m kH(m)k⇤ subject to P⌦(b) = P⌦(f) RankH(f) = k * Jin KH et al IEEE TCI, 2016 * Jin KH et al.,IEEE TIP, 2015 * Ye JC et al., IEEE TIT, 2016 m Annihilating filter-based low-rank Hankel matrix (ALOHA)
  19. ALOHA for CS-MRI ALOHA: Annihilating filter-based low-rank Hankel matrix approach

    • Jin KH et al IEEE TCI, 2016 • Lee et al, MRM, 2015
  20. Single coil static MRI

  21. Key Observation Data-Driven Hankel matrix decomposition => Deep Learning •

    Ye et al, “Deep convolutional framelets: A general deep learning framework for inverse problems”, SIAM Journal Imaging Sciences, 11(2), 991-1048, 2018.
  22. Hd(f) = U⌃V T : Non-local basis : Local basis

    Convolution Framelets (Yin et al; 2017) > = I > = I Hd(f)
  23. Hd(f) Hd(f) = ˜ T ˜ T C C =

    T Hd(f) C = T (f ~ ) Encoder: ˜ T = I ˜ = PR(V ) Hd(f) = U⌃V T Unlifting: f = (˜C) ~ ⌧(˜ ) : Frame condition : rank condition convolution pooling un-pooling convolution : User-defined pooling : Learnable filters Hpi (gi) = X k,l [Ci]kl e Bkl i Decoder: Deep Convolutional Framelets (Y, Han, Cha; 2018)
  24. Single Resolution Network Architecture

  25. Multi-Resolution Network Architecture

  26. Conic fi [Ci]kl 0 Hpi (gi) = X k,l [Ci]kl

    e Bkl i Hpi (fi) ' Linear Lifting Geometry of CNN gi Linear Un-lifting Ci(fi) Ci(fi) 0 i ⇣i ⇣ e i ⌘
  27. fi [Ci]kl 0 Hpi (gi) = X k,l [Ci]kl e

    Bkl i Hpi (fi) ' Lifting Geometry of Residual CNN Ci(fi) Ci(fi) 0 i ⇣i ⇣ e i ⌘ gi Un-lifting
  28. Deep CNN Lifting Un-lifting Conic Lifting Un-lifting Lifting Un-lifting

  29. fi Nonlinear Lifting to Feature space Comparison with Kernel PCA

    gi Nonlinear Pre-Image calculation (fi) <latexit sha1_base64="E9VdeouKNx3eJ5UDAMrvyn9icnU=">AAAB+nicbVBNS8NAFHzxs9avWI9egkWol5KIoN6KXjxWMLbQhLDZbtqlm03Y3Ygl5K948aDi1V/izX/jps1BWwcWhpn3eLMTpoxKZdvfxsrq2vrGZm2rvr2zu7dvHjQeZJIJTFycsET0QyQJo5y4iipG+qkgKA4Z6YWTm9LvPRIhacLv1TQlfoxGnEYUI6WlwGx43TFteTFS4zDKoyKgp4HZtNv2DNYycSrShArdwPzyhgnOYsIVZkjKgWOnys+RUBQzUtS9TJIU4QkakYGmHMVE+vkse2GdaGVoRYnQjytrpv7eyFEs5TQO9WQZUi56pfifN8hUdOnnlKeZIhzPD0UZs1RilUVYQyoIVmyqCcKC6qwWHiOBsNJ11XUJzuKXl4l71r5qO3fnzc511UYNjuAYWuDABXTgFrrgAoYneIZXeDMK48V4Nz7moytGtXMIf2B8/gD8DZP1</latexit> <latexit sha1_base64="E9VdeouKNx3eJ5UDAMrvyn9icnU=">AAAB+nicbVBNS8NAFHzxs9avWI9egkWol5KIoN6KXjxWMLbQhLDZbtqlm03Y3Ygl5K948aDi1V/izX/jps1BWwcWhpn3eLMTpoxKZdvfxsrq2vrGZm2rvr2zu7dvHjQeZJIJTFycsET0QyQJo5y4iipG+qkgKA4Z6YWTm9LvPRIhacLv1TQlfoxGnEYUI6WlwGx43TFteTFS4zDKoyKgp4HZtNv2DNYycSrShArdwPzyhgnOYsIVZkjKgWOnys+RUBQzUtS9TJIU4QkakYGmHMVE+vkse2GdaGVoRYnQjytrpv7eyFEs5TQO9WQZUi56pfifN8hUdOnnlKeZIhzPD0UZs1RilUVYQyoIVmyqCcKC6qwWHiOBsNJ11XUJzuKXl4l71r5qO3fnzc511UYNjuAYWuDABXTgFrrgAoYneIZXeDMK48V4Nz7moytGtXMIf2B8/gD8DZP1</latexit> <latexit sha1_base64="E9VdeouKNx3eJ5UDAMrvyn9icnU=">AAAB+nicbVBNS8NAFHzxs9avWI9egkWol5KIoN6KXjxWMLbQhLDZbtqlm03Y3Ygl5K948aDi1V/izX/jps1BWwcWhpn3eLMTpoxKZdvfxsrq2vrGZm2rvr2zu7dvHjQeZJIJTFycsET0QyQJo5y4iipG+qkgKA4Z6YWTm9LvPRIhacLv1TQlfoxGnEYUI6WlwGx43TFteTFS4zDKoyKgp4HZtNv2DNYycSrShArdwPzyhgnOYsIVZkjKgWOnys+RUBQzUtS9TJIU4QkakYGmHMVE+vkse2GdaGVoRYnQjytrpv7eyFEs5TQO9WQZUi56pfifN8hUdOnnlKeZIhzPD0UZs1RilUVYQyoIVmyqCcKC6qwWHiOBsNJ11XUJzuKXl4l71r5qO3fnzc511UYNjuAYWuDABXTgFrrgAoYneIZXeDMK48V4Nz7moytGtXMIf2B8/gD8DZP1</latexit> <latexit sha1_base64="E9VdeouKNx3eJ5UDAMrvyn9icnU=">AAAB+nicbVBNS8NAFHzxs9avWI9egkWol5KIoN6KXjxWMLbQhLDZbtqlm03Y3Ygl5K948aDi1V/izX/jps1BWwcWhpn3eLMTpoxKZdvfxsrq2vrGZm2rvr2zu7dvHjQeZJIJTFycsET0QyQJo5y4iipG+qkgKA4Z6YWTm9LvPRIhacLv1TQlfoxGnEYUI6WlwGx43TFteTFS4zDKoyKgp4HZtNv2DNYycSrShArdwPzyhgnOYsIVZkjKgWOnys+RUBQzUtS9TJIU4QkakYGmHMVE+vkse2GdaGVoRYnQjytrpv7eyFEs5TQO9WQZUi56pfifN8hUdOnnlKeZIhzPD0UZs1RilUVYQyoIVmyqCcKC6qwWHiOBsNJ11XUJzuKXl4l71r5qO3fnzc511UYNjuAYWuDABXTgFrrgAoYneIZXeDMK48V4Nz7moytGtXMIf2B8/gD8DZP1</latexit> C = 1 N N X i=1 (fi) >(fi) <latexit sha1_base64="LKnrl556MzHt0DAUIt0n5VZ7wPI=">AAACK3icbZDLSsNAFIYnXmu9RV26GSxC3ZREBHVRKHbjqlQwttCkYTKdtEMnF2YmQgl5IDe+iiAurLj1PZy0WdjWHwZ+vnMOc87vxYwKaRhTbW19Y3Nru7RT3t3bPzjUj46fRJRwTCwcsYh3PSQIoyGxJJWMdGNOUOAx0vHGzbzeeSZc0Ch8lJOYOAEahtSnGEmFXL3ZrEPb5winZpa2MlskgZvSupn1W3Z7RKt2gOTI81M/c+lFTvq2jOJF7OoVo2bMBFeNWZgKKNR29Xd7EOEkIKHEDAnRM41YOinikmJGsrKdCBIjPEZD0lM2RAERTjo7NoPnigygH3H1Qgln9O9EigIhJoGnOvMlxXIth//Veon0b5yUhnEiSYjnH/kJgzKCeXJwQDnBkk2UQZhTtSvEI6SikyrfsgrBXD551ViXtdua+XBVadwVaZTAKTgDVWCCa9AA96ANLIDBC3gDn2CqvWof2pf2PW9d04qZE7Ag7ecX5JCoQA==</latexit> <latexit sha1_base64="LKnrl556MzHt0DAUIt0n5VZ7wPI=">AAACK3icbZDLSsNAFIYnXmu9RV26GSxC3ZREBHVRKHbjqlQwttCkYTKdtEMnF2YmQgl5IDe+iiAurLj1PZy0WdjWHwZ+vnMOc87vxYwKaRhTbW19Y3Nru7RT3t3bPzjUj46fRJRwTCwcsYh3PSQIoyGxJJWMdGNOUOAx0vHGzbzeeSZc0Ch8lJOYOAEahtSnGEmFXL3ZrEPb5winZpa2MlskgZvSupn1W3Z7RKt2gOTI81M/c+lFTvq2jOJF7OoVo2bMBFeNWZgKKNR29Xd7EOEkIKHEDAnRM41YOinikmJGsrKdCBIjPEZD0lM2RAERTjo7NoPnigygH3H1Qgln9O9EigIhJoGnOvMlxXIth//Veon0b5yUhnEiSYjnH/kJgzKCeXJwQDnBkk2UQZhTtSvEI6SikyrfsgrBXD551ViXtdua+XBVadwVaZTAKTgDVWCCa9AA96ANLIDBC3gDn2CqvWof2pf2PW9d04qZE7Ag7ecX5JCoQA==</latexit> <latexit sha1_base64="LKnrl556MzHt0DAUIt0n5VZ7wPI=">AAACK3icbZDLSsNAFIYnXmu9RV26GSxC3ZREBHVRKHbjqlQwttCkYTKdtEMnF2YmQgl5IDe+iiAurLj1PZy0WdjWHwZ+vnMOc87vxYwKaRhTbW19Y3Nru7RT3t3bPzjUj46fRJRwTCwcsYh3PSQIoyGxJJWMdGNOUOAx0vHGzbzeeSZc0Ch8lJOYOAEahtSnGEmFXL3ZrEPb5winZpa2MlskgZvSupn1W3Z7RKt2gOTI81M/c+lFTvq2jOJF7OoVo2bMBFeNWZgKKNR29Xd7EOEkIKHEDAnRM41YOinikmJGsrKdCBIjPEZD0lM2RAERTjo7NoPnigygH3H1Qgln9O9EigIhJoGnOvMlxXIth//Veon0b5yUhnEiSYjnH/kJgzKCeXJwQDnBkk2UQZhTtSvEI6SikyrfsgrBXD551ViXtdua+XBVadwVaZTAKTgDVWCCa9AA96ANLIDBC3gDn2CqvWof2pf2PW9d04qZE7Ag7ecX5JCoQA==</latexit> <latexit sha1_base64="LKnrl556MzHt0DAUIt0n5VZ7wPI=">AAACK3icbZDLSsNAFIYnXmu9RV26GSxC3ZREBHVRKHbjqlQwttCkYTKdtEMnF2YmQgl5IDe+iiAurLj1PZy0WdjWHwZ+vnMOc87vxYwKaRhTbW19Y3Nru7RT3t3bPzjUj46fRJRwTCwcsYh3PSQIoyGxJJWMdGNOUOAx0vHGzbzeeSZc0Ch8lJOYOAEahtSnGEmFXL3ZrEPb5winZpa2MlskgZvSupn1W3Z7RKt2gOTI81M/c+lFTvq2jOJF7OoVo2bMBFeNWZgKKNR29Xd7EOEkIKHEDAnRM41YOinikmJGsrKdCBIjPEZD0lM2RAERTjo7NoPnigygH3H1Qgln9O9EigIhJoGnOvMlxXIth//Veon0b5yUhnEiSYjnH/kJgzKCeXJwQDnBkk2UQZhTtSvEI6SikyrfsgrBXD551ViXtdua+XBVadwVaZTAKTgDVWCCa9AA96ANLIDBC3gDn2CqvWof2pf2PW9d04qZE7Ag7ecX5JCoQA==</latexit> PCA of • Nonlinear lifting & unlifting • Deterministic kernel • Difficulty in multilevel extension
  30. Insights obtained from the analysis q Pooling layer à network

    architecture ü Frame condition qWhere to design neural network ? ü Where Hankel structured matrix is low-ranked q Other insights ü Concatenation layers ü Residual or direction mapping ?
  31. APPLICATION-DRIVEN EVIDENCES

  32. ALOHA CNN k-Space Deep Learning for Accelerated MRI

  33. k-Space Deep Learning for Accelerated MRI Han et al, arXiv:1805.03779

    ~3dB gain
  34. Research Goal Ø To improve temporal resolution of TWIST imaging

    using deep k-space learning Ø To generate multiple reconstruction results with various spatial and temporal resolution using one network VS = 5 VS = 2 CNN K-space Deep Learning for Time-resolved MRI
  35. K-space Deep Learning Results Conventional Solution  Solution

  36. Low-Dose CT • To reduce the radiation exposure, sparse-view CT,

    low-dose CT and interior tomography. Sparse-view CT (Down-sampled View) Low-dose CT (Reduced X-ray dose) Interior Tomography (Truncated FOV)
  37. WavResNet for low-dose CT (Kang et al, Medical Phyhsics, 2017,

    IEEE TMI 2018)
  38.  Solution Conventional Solution Quater dose Full-dose

  39.  Solution Conventional Solution Quater dose Full-dose

  40. MBIR Our method C D

  41. WHAT IF WE DON’T HAVE REFERENCE ?

  42. Low-dose CT with Adversarial loss

  43. Un-Supervised Learning for low-dose CT 43 • Multiphase Cardiac CT

    denoising – Phase 1, 2: low-dose, Phase 3 ~ 10: normal dose – Goal: dynamic changes of heart structure – No reference available Kang et al, Medical Physics (in press), 2018
  44. 44 (a) (b) (c) (d) (e) (f) (g) (h) (b)

    (c) (d) (f) (g) (h) GAN
  45. 45 • Cardiac CT denoising – Cycle Consistent Adversarial Denoising

    Network for Multiphase Coronary CT Angiography Un-supervised Learning using Cyclic-GAN
  46. 46 (a) (b) (c) (d) (e) (f) (g) (h) (b)

    (c) (d) (f) (g) (h) GAN
  47. 47 Input: phase 1 Denoised output Target: phase 8 Input-

    output
  48. FBP Proposed ADMIRE

  49. Unsupervised / supervised / ground-truth

  50. 1st view 2nd view 3rd view 4th view 5th view

    6th view 7th view 8th view 9th view q AI-powered 9-View CT for Security Scanning Semi-supervised Learning Han et al, arXiv preprint arXiv:1712.10248, (2017). CT meeting 2018
  51. Conventional Solution  Solution

  52. Semi-Supervised High Resolution View Synthesis 128 256 64 128 6

    4 256 256 512 512 512 1024 512 1024 512 256 512 256128 • Key idea • Training with measured views • Generate stacks of x-y images • Neural network training with theta-z images
  53. Outlooks q End-to-End AI for radiological imaging Ø From AI-powered

    image acquisition to diagnosis for clear and rapid radiological imaging Existing AI Solutions: Diagnosis Our solution: from acquisition to diagnosis
  54. math Thank You