Overview of Machine Learning Methods for Reconstruction of Imaging Data

A3d61bc22cd700a92e7d4136a4d29e8f?s=47 Jong Chul Ye
October 26, 2018

Overview of Machine Learning Methods for Reconstruction of Imaging Data

Keynote Talk, ​ISMRM Workshop on Machine Learning, Part II, Oct 26, 2018, Washington DC, USA


Jong Chul Ye

October 26, 2018


  1. Overview of Machine Learning Methods for Reconstruction of Imaging Data

    Jong Chul Ye, Ph.D Endowed Chair Professor Bio-Imaging, Signal Processing and Learning (BISPL) Group Dept. Bio & Brain Engineering Dept. Mathematical Science KAIST, Korea
  2. Unmet Needs in Medical Imaging q Medical imaging is essential

    for diagnosis of disease q Imaging systems with lower-dose of ionizing radiation ü Increasing risk of radiation dose from CT, PET, etc q Reducing contrast agents without losing diagnostic accuracy ü Severe side-effects from iodine, gadolinium contrast agents q Increasing the throughput of MR scanning ü Current MR scan time : ~ 40~50min/patient
  3. 3 The Solution q Deep learning-based recon • High Quality,

    Fast Reconstruction • Vendor-oriented Neural Network Training: a must-buy for customer • AI-driven hardware design: US systems, industrial CT • Solution Examples: • AI-powered low-dose CT reconstruction • AI-powered contrast synthesis for contrast agent reduction • AI-powered accelerated MRI • AI-powered US for low-power, ultrafast US imaging • AI-powered industrial CT
  4. None
  5. None
  6. None
  7. GANCS Mardani et al, IEEE TMI 2018

  8. None
  9. None
  10. QSMNet Yoon et al, NeuroImage, 2018

  11. 1 1



  14. Too Simple to Analyze..? Convolution & pooling à stone age

    tools of signal processing What do they do ?
  15. Paradox and Mysteries Residual Network Clean image Standard Network Zhang,

    K., et al, IEEE TIP, 2017.
  16. Clue: Matrix Representation of CNN Figure courtesy of Shoieb et

    al, 2016
  17. Hankel Matrix: Linear Lifting to Higher Dimensional Space

  18. Missing elements can be found by low rank Hankel structured

    matrix completion Nuclear norm Projection on sampling positions min m kH(m)k⇤ subject to P⌦(b) = P⌦(f) RankH(f) = k * Jin KH et al IEEE TCI, 2016 * Jin KH et al.,IEEE TIP, 2015 * Ye JC et al., IEEE TIT, 2016 m Annihilating filter-based low-rank Hankel matrix (ALOHA)
  19. ALOHA for CS-MRI ALOHA: Annihilating filter-based low-rank Hankel matrix approach

    • Jin KH et al IEEE TCI, 2016 • Lee et al, MRM, 2015
  20. Single coil static MRI

  21. Key Observation Data-Driven Hankel matrix decomposition => Deep Learning •

    Ye et al, “Deep convolutional framelets: A general deep learning framework for inverse problems”, SIAM Journal Imaging Sciences, 11(2), 991-1048, 2018.
  22. Hd(f) = U⌃V T : Non-local basis : Local basis

    Convolution Framelets (Yin et al; 2017) > = I > = I Hd(f)
  23. Hd(f) Hd(f) = ˜ T ˜ T C C =

    T Hd(f) C = T (f ~ ) Encoder: ˜ T = I ˜ = PR(V ) Hd(f) = U⌃V T Unlifting: f = (˜C) ~ ⌧(˜ ) : Frame condition : rank condition convolution pooling un-pooling convolution : User-defined pooling : Learnable filters Hpi (gi) = X k,l [Ci]kl e Bkl i Decoder: Deep Convolutional Framelets (Y, Han, Cha; 2018)
  24. Single Resolution Network Architecture

  25. Multi-Resolution Network Architecture

  26. Conic fi [Ci]kl 0 Hpi (gi) = X k,l [Ci]kl

    e Bkl i Hpi (fi) ' Linear Lifting Geometry of CNN gi Linear Un-lifting Ci(fi) Ci(fi) 0 i ⇣i ⇣ e i ⌘
  27. fi [Ci]kl 0 Hpi (gi) = X k,l [Ci]kl e

    Bkl i Hpi (fi) ' Lifting Geometry of Residual CNN Ci(fi) Ci(fi) 0 i ⇣i ⇣ e i ⌘ gi Un-lifting
  28. Deep CNN Lifting Un-lifting Conic Lifting Un-lifting Lifting Un-lifting

  29. fi Nonlinear Lifting to Feature space Comparison with Kernel PCA

    gi Nonlinear Pre-Image calculation (fi) <latexit sha1_base64="E9VdeouKNx3eJ5UDAMrvyn9icnU=">AAAB+nicbVBNS8NAFHzxs9avWI9egkWol5KIoN6KXjxWMLbQhLDZbtqlm03Y3Ygl5K948aDi1V/izX/jps1BWwcWhpn3eLMTpoxKZdvfxsrq2vrGZm2rvr2zu7dvHjQeZJIJTFycsET0QyQJo5y4iipG+qkgKA4Z6YWTm9LvPRIhacLv1TQlfoxGnEYUI6WlwGx43TFteTFS4zDKoyKgp4HZtNv2DNYycSrShArdwPzyhgnOYsIVZkjKgWOnys+RUBQzUtS9TJIU4QkakYGmHMVE+vkse2GdaGVoRYnQjytrpv7eyFEs5TQO9WQZUi56pfifN8hUdOnnlKeZIhzPD0UZs1RilUVYQyoIVmyqCcKC6qwWHiOBsNJ11XUJzuKXl4l71r5qO3fnzc511UYNjuAYWuDABXTgFrrgAoYneIZXeDMK48V4Nz7moytGtXMIf2B8/gD8DZP1</latexit> <latexit sha1_base64="E9VdeouKNx3eJ5UDAMrvyn9icnU=">AAAB+nicbVBNS8NAFHzxs9avWI9egkWol5KIoN6KXjxWMLbQhLDZbtqlm03Y3Ygl5K948aDi1V/izX/jps1BWwcWhpn3eLMTpoxKZdvfxsrq2vrGZm2rvr2zu7dvHjQeZJIJTFycsET0QyQJo5y4iipG+qkgKA4Z6YWTm9LvPRIhacLv1TQlfoxGnEYUI6WlwGx43TFteTFS4zDKoyKgp4HZtNv2DNYycSrShArdwPzyhgnOYsIVZkjKgWOnys+RUBQzUtS9TJIU4QkakYGmHMVE+vkse2GdaGVoRYnQjytrpv7eyFEs5TQO9WQZUi56pfifN8hUdOnnlKeZIhzPD0UZs1RilUVYQyoIVmyqCcKC6qwWHiOBsNJ11XUJzuKXl4l71r5qO3fnzc511UYNjuAYWuDABXTgFrrgAoYneIZXeDMK48V4Nz7moytGtXMIf2B8/gD8DZP1</latexit> <latexit sha1_base64="E9VdeouKNx3eJ5UDAMrvyn9icnU=">AAAB+nicbVBNS8NAFHzxs9avWI9egkWol5KIoN6KXjxWMLbQhLDZbtqlm03Y3Ygl5K948aDi1V/izX/jps1BWwcWhpn3eLMTpoxKZdvfxsrq2vrGZm2rvr2zu7dvHjQeZJIJTFycsET0QyQJo5y4iipG+qkgKA4Z6YWTm9LvPRIhacLv1TQlfoxGnEYUI6WlwGx43TFteTFS4zDKoyKgp4HZtNv2DNYycSrShArdwPzyhgnOYsIVZkjKgWOnys+RUBQzUtS9TJIU4QkakYGmHMVE+vkse2GdaGVoRYnQjytrpv7eyFEs5TQO9WQZUi56pfifN8hUdOnnlKeZIhzPD0UZs1RilUVYQyoIVmyqCcKC6qwWHiOBsNJ11XUJzuKXl4l71r5qO3fnzc511UYNjuAYWuDABXTgFrrgAoYneIZXeDMK48V4Nz7moytGtXMIf2B8/gD8DZP1</latexit> <latexit sha1_base64="E9VdeouKNx3eJ5UDAMrvyn9icnU=">AAAB+nicbVBNS8NAFHzxs9avWI9egkWol5KIoN6KXjxWMLbQhLDZbtqlm03Y3Ygl5K948aDi1V/izX/jps1BWwcWhpn3eLMTpoxKZdvfxsrq2vrGZm2rvr2zu7dvHjQeZJIJTFycsET0QyQJo5y4iipG+qkgKA4Z6YWTm9LvPRIhacLv1TQlfoxGnEYUI6WlwGx43TFteTFS4zDKoyKgp4HZtNv2DNYycSrShArdwPzyhgnOYsIVZkjKgWOnys+RUBQzUtS9TJIU4QkakYGmHMVE+vkse2GdaGVoRYnQjytrpv7eyFEs5TQO9WQZUi56pfifN8hUdOnnlKeZIhzPD0UZs1RilUVYQyoIVmyqCcKC6qwWHiOBsNJ11XUJzuKXl4l71r5qO3fnzc511UYNjuAYWuDABXTgFrrgAoYneIZXeDMK48V4Nz7moytGtXMIf2B8/gD8DZP1</latexit> C = 1 N N X i=1 (fi) >(fi) <latexit sha1_base64="LKnrl556MzHt0DAUIt0n5VZ7wPI=">AAACK3icbZDLSsNAFIYnXmu9RV26GSxC3ZREBHVRKHbjqlQwttCkYTKdtEMnF2YmQgl5IDe+iiAurLj1PZy0WdjWHwZ+vnMOc87vxYwKaRhTbW19Y3Nru7RT3t3bPzjUj46fRJRwTCwcsYh3PSQIoyGxJJWMdGNOUOAx0vHGzbzeeSZc0Ch8lJOYOAEahtSnGEmFXL3ZrEPb5winZpa2MlskgZvSupn1W3Z7RKt2gOTI81M/c+lFTvq2jOJF7OoVo2bMBFeNWZgKKNR29Xd7EOEkIKHEDAnRM41YOinikmJGsrKdCBIjPEZD0lM2RAERTjo7NoPnigygH3H1Qgln9O9EigIhJoGnOvMlxXIth//Veon0b5yUhnEiSYjnH/kJgzKCeXJwQDnBkk2UQZhTtSvEI6SikyrfsgrBXD551ViXtdua+XBVadwVaZTAKTgDVWCCa9AA96ANLIDBC3gDn2CqvWof2pf2PW9d04qZE7Ag7ecX5JCoQA==</latexit> <latexit sha1_base64="LKnrl556MzHt0DAUIt0n5VZ7wPI=">AAACK3icbZDLSsNAFIYnXmu9RV26GSxC3ZREBHVRKHbjqlQwttCkYTKdtEMnF2YmQgl5IDe+iiAurLj1PZy0WdjWHwZ+vnMOc87vxYwKaRhTbW19Y3Nru7RT3t3bPzjUj46fRJRwTCwcsYh3PSQIoyGxJJWMdGNOUOAx0vHGzbzeeSZc0Ch8lJOYOAEahtSnGEmFXL3ZrEPb5winZpa2MlskgZvSupn1W3Z7RKt2gOTI81M/c+lFTvq2jOJF7OoVo2bMBFeNWZgKKNR29Xd7EOEkIKHEDAnRM41YOinikmJGsrKdCBIjPEZD0lM2RAERTjo7NoPnigygH3H1Qgln9O9EigIhJoGnOvMlxXIth//Veon0b5yUhnEiSYjnH/kJgzKCeXJwQDnBkk2UQZhTtSvEI6SikyrfsgrBXD551ViXtdua+XBVadwVaZTAKTgDVWCCa9AA96ANLIDBC3gDn2CqvWof2pf2PW9d04qZE7Ag7ecX5JCoQA==</latexit> <latexit sha1_base64="LKnrl556MzHt0DAUIt0n5VZ7wPI=">AAACK3icbZDLSsNAFIYnXmu9RV26GSxC3ZREBHVRKHbjqlQwttCkYTKdtEMnF2YmQgl5IDe+iiAurLj1PZy0WdjWHwZ+vnMOc87vxYwKaRhTbW19Y3Nru7RT3t3bPzjUj46fRJRwTCwcsYh3PSQIoyGxJJWMdGNOUOAx0vHGzbzeeSZc0Ch8lJOYOAEahtSnGEmFXL3ZrEPb5winZpa2MlskgZvSupn1W3Z7RKt2gOTI81M/c+lFTvq2jOJF7OoVo2bMBFeNWZgKKNR29Xd7EOEkIKHEDAnRM41YOinikmJGsrKdCBIjPEZD0lM2RAERTjo7NoPnigygH3H1Qgln9O9EigIhJoGnOvMlxXIth//Veon0b5yUhnEiSYjnH/kJgzKCeXJwQDnBkk2UQZhTtSvEI6SikyrfsgrBXD551ViXtdua+XBVadwVaZTAKTgDVWCCa9AA96ANLIDBC3gDn2CqvWof2pf2PW9d04qZE7Ag7ecX5JCoQA==</latexit> <latexit sha1_base64="LKnrl556MzHt0DAUIt0n5VZ7wPI=">AAACK3icbZDLSsNAFIYnXmu9RV26GSxC3ZREBHVRKHbjqlQwttCkYTKdtEMnF2YmQgl5IDe+iiAurLj1PZy0WdjWHwZ+vnMOc87vxYwKaRhTbW19Y3Nru7RT3t3bPzjUj46fRJRwTCwcsYh3PSQIoyGxJJWMdGNOUOAx0vHGzbzeeSZc0Ch8lJOYOAEahtSnGEmFXL3ZrEPb5winZpa2MlskgZvSupn1W3Z7RKt2gOTI81M/c+lFTvq2jOJF7OoVo2bMBFeNWZgKKNR29Xd7EOEkIKHEDAnRM41YOinikmJGsrKdCBIjPEZD0lM2RAERTjo7NoPnigygH3H1Qgln9O9EigIhJoGnOvMlxXIth//Veon0b5yUhnEiSYjnH/kJgzKCeXJwQDnBkk2UQZhTtSvEI6SikyrfsgrBXD551ViXtdua+XBVadwVaZTAKTgDVWCCa9AA96ANLIDBC3gDn2CqvWof2pf2PW9d04qZE7Ag7ecX5JCoQA==</latexit> PCA of • Nonlinear lifting & unlifting • Deterministic kernel • Difficulty in multilevel extension
  30. Insights obtained from the analysis q Pooling layer à network

    architecture ü Frame condition qWhere to design neural network ? ü Where Hankel structured matrix is low-ranked q Other insights ü Concatenation layers ü Residual or direction mapping ?

  32. ALOHA CNN k-Space Deep Learning for Accelerated MRI

  33. k-Space Deep Learning for Accelerated MRI Han et al, arXiv:1805.03779

    ~3dB gain
  34. Research Goal Ø To improve temporal resolution of TWIST imaging

    using deep k-space learning Ø To generate multiple reconstruction results with various spatial and temporal resolution using one network VS = 5 VS = 2 CNN K-space Deep Learning for Time-resolved MRI
  35. K-space Deep Learning Results Conventional Solution  Solution

  36. Low-Dose CT • To reduce the radiation exposure, sparse-view CT,

    low-dose CT and interior tomography. Sparse-view CT (Down-sampled View) Low-dose CT (Reduced X-ray dose) Interior Tomography (Truncated FOV)
  37. WavResNet for low-dose CT (Kang et al, Medical Phyhsics, 2017,

    IEEE TMI 2018)
  38.  Solution Conventional Solution Quater dose Full-dose

  39.  Solution Conventional Solution Quater dose Full-dose

  40. MBIR Our method C D


  42. Low-dose CT with Adversarial loss

  43. Un-Supervised Learning for low-dose CT 43 • Multiphase Cardiac CT

    denoising – Phase 1, 2: low-dose, Phase 3 ~ 10: normal dose – Goal: dynamic changes of heart structure – No reference available Kang et al, Medical Physics (in press), 2018
  44. 44 (a) (b) (c) (d) (e) (f) (g) (h) (b)

    (c) (d) (f) (g) (h) GAN
  45. 45 • Cardiac CT denoising – Cycle Consistent Adversarial Denoising

    Network for Multiphase Coronary CT Angiography Un-supervised Learning using Cyclic-GAN
  46. 46 (a) (b) (c) (d) (e) (f) (g) (h) (b)

    (c) (d) (f) (g) (h) GAN
  47. 47 Input: phase 1 Denoised output Target: phase 8 Input-

  48. FBP Proposed ADMIRE

  49. Unsupervised / supervised / ground-truth

  50. 1st view 2nd view 3rd view 4th view 5th view

    6th view 7th view 8th view 9th view q AI-powered 9-View CT for Security Scanning Semi-supervised Learning Han et al, arXiv preprint arXiv:1712.10248, (2017). CT meeting 2018
  51. Conventional Solution  Solution

  52. Semi-Supervised High Resolution View Synthesis 128 256 64 128 6

    4 256 256 512 512 512 1024 512 1024 512 256 512 256128 • Key idea • Training with measured views • Generate stacks of x-y images • Neural network training with theta-z images
  53. Outlooks q End-to-End AI for radiological imaging Ø From AI-powered

    image acquisition to diagnosis for clear and rapid radiological imaging Existing AI Solutions: Diagnosis Our solution: from acquisition to diagnosis
  54. math Thank You