Tweet
Tweet

Slide 1

Slide 1 text

Overview of Machine Learning Methods for Reconstruction of Imaging Data Jong Chul Ye, Ph.D Endowed Chair Professor Bio-Imaging, Signal Processing and Learning (BISPL) Group Dept. Bio & Brain Engineering Dept. Mathematical Science KAIST, Korea

Slide 2

Slide 2 text

Unmet Needs in Medical Imaging q Medical imaging is essential for diagnosis of disease q Imaging systems with lower-dose of ionizing radiation ü Increasing risk of radiation dose from CT, PET, etc q Reducing contrast agents without losing diagnostic accuracy ü Severe side-effects from iodine, gadolinium contrast agents q Increasing the throughput of MR scanning ü Current MR scan time : ~ 40~50min/patient

Slide 3

Slide 3 text

3 The Solution q Deep learning-based recon • High Quality, Fast Reconstruction • Vendor-oriented Neural Network Training: a must-buy for customer • AI-driven hardware design: US systems, industrial CT • Solution Examples: • AI-powered low-dose CT reconstruction • AI-powered contrast synthesis for contrast agent reduction • AI-powered accelerated MRI • AI-powered US for low-power, ultrafast US imaging • AI-powered industrial CT

Slide 4

Slide 4 text

No content

Slide 5

Slide 5 text

No content

Slide 6

Slide 6 text

No content

Slide 7

Slide 7 text

GANCS Mardani et al, IEEE TMI 2018

Slide 8

Slide 8 text

No content

Slide 9

Slide 9 text

No content

Slide 10

Slide 10 text

QSMNet Yoon et al, NeuroImage, 2018

Slide 11

Slide 11 text

1 1

Slide 12

Slide 12 text

WHY DEEP LEARNING WORKS FOR RECON ? DOES IT CREATE ANY ARTIFICIAL FEATURES ?

Slide 13

Slide 13 text

WHAT IF WE DON’T HAVE LABEL DATA ?

Slide 14

Slide 14 text

Too Simple to Analyze..? Convolution & pooling à stone age tools of signal processing What do they do ?

Slide 15

Slide 15 text

Paradox and Mysteries Residual Network Clean image Standard Network Zhang, K., et al, IEEE TIP, 2017.

Slide 16

Slide 16 text

Clue: Matrix Representation of CNN Figure courtesy of Shoieb et al, 2016

Slide 17

Slide 17 text

Hankel Matrix: Linear Lifting to Higher Dimensional Space

Slide 18

Slide 18 text

Missing elements can be found by low rank Hankel structured matrix completion Nuclear norm Projection on sampling positions min m kH(m)k⇤ subject to P⌦(b) = P⌦(f) RankH(f) = k * Jin KH et al IEEE TCI, 2016 * Jin KH et al.,IEEE TIP, 2015 * Ye JC et al., IEEE TIT, 2016 m Annihilating filter-based low-rank Hankel matrix (ALOHA)

Slide 19

Slide 19 text

ALOHA for CS-MRI ALOHA: Annihilating filter-based low-rank Hankel matrix approach • Jin KH et al IEEE TCI, 2016 • Lee et al, MRM, 2015

Slide 20

Slide 20 text

Single coil static MRI

Slide 21

Slide 21 text

Key Observation Data-Driven Hankel matrix decomposition => Deep Learning • Ye et al, “Deep convolutional framelets: A general deep learning framework for inverse problems”, SIAM Journal Imaging Sciences, 11(2), 991-1048, 2018.

Slide 22

Slide 22 text

Hd(f) = U⌃V T : Non-local basis : Local basis Convolution Framelets (Yin et al; 2017) > = I > = I Hd(f)

Slide 23

Slide 23 text

Hd(f) Hd(f) = ˜ T ˜ T C C = T Hd(f) C = T (f ~ ) Encoder: ˜ T = I ˜ = PR(V ) Hd(f) = U⌃V T Unlifting: f = (˜C) ~ ⌧(˜ ) : Frame condition : rank condition convolution pooling un-pooling convolution : User-defined pooling : Learnable filters Hpi (gi) = X k,l [Ci]kl e Bkl i Decoder: Deep Convolutional Framelets (Y, Han, Cha; 2018)

Slide 24

Slide 24 text

Single Resolution Network Architecture

Slide 25

Slide 25 text

Multi-Resolution Network Architecture

Slide 26

Slide 26 text

Conic fi [Ci]kl 0 Hpi (gi) = X k,l [Ci]kl e Bkl i Hpi (fi) ' Linear Lifting Geometry of CNN gi Linear Un-lifting Ci(fi) Ci(fi) 0 i ⇣i ⇣ e i ⌘

Slide 27

Slide 27 text

fi [Ci]kl 0 Hpi (gi) = X k,l [Ci]kl e Bkl i Hpi (fi) ' Lifting Geometry of Residual CNN Ci(fi) Ci(fi) 0 i ⇣i ⇣ e i ⌘ gi Un-lifting

Slide 28

Slide 28 text

Deep CNN Lifting Un-lifting Conic Lifting Un-lifting Lifting Un-lifting

Slide 29

Slide 29 text

fi Nonlinear Lifting to Feature space Comparison with Kernel PCA gi Nonlinear Pre-Image calculation (fi) AAAB+nicbVBNS8NAFHzxs9avWI9egkWol5KIoN6KXjxWMLbQhLDZbtqlm03Y3Ygl5K948aDi1V/izX/jps1BWwcWhpn3eLMTpoxKZdvfxsrq2vrGZm2rvr2zu7dvHjQeZJIJTFycsET0QyQJo5y4iipG+qkgKA4Z6YWTm9LvPRIhacLv1TQlfoxGnEYUI6WlwGx43TFteTFS4zDKoyKgp4HZtNv2DNYycSrShArdwPzyhgnOYsIVZkjKgWOnys+RUBQzUtS9TJIU4QkakYGmHMVE+vkse2GdaGVoRYnQjytrpv7eyFEs5TQO9WQZUi56pfifN8hUdOnnlKeZIhzPD0UZs1RilUVYQyoIVmyqCcKC6qwWHiOBsNJ11XUJzuKXl4l71r5qO3fnzc511UYNjuAYWuDABXTgFrrgAoYneIZXeDMK48V4Nz7moytGtXMIf2B8/gD8DZP1 AAAB+nicbVBNS8NAFHzxs9avWI9egkWol5KIoN6KXjxWMLbQhLDZbtqlm03Y3Ygl5K948aDi1V/izX/jps1BWwcWhpn3eLMTpoxKZdvfxsrq2vrGZm2rvr2zu7dvHjQeZJIJTFycsET0QyQJo5y4iipG+qkgKA4Z6YWTm9LvPRIhacLv1TQlfoxGnEYUI6WlwGx43TFteTFS4zDKoyKgp4HZtNv2DNYycSrShArdwPzyhgnOYsIVZkjKgWOnys+RUBQzUtS9TJIU4QkakYGmHMVE+vkse2GdaGVoRYnQjytrpv7eyFEs5TQO9WQZUi56pfifN8hUdOnnlKeZIhzPD0UZs1RilUVYQyoIVmyqCcKC6qwWHiOBsNJ11XUJzuKXl4l71r5qO3fnzc511UYNjuAYWuDABXTgFrrgAoYneIZXeDMK48V4Nz7moytGtXMIf2B8/gD8DZP1 AAAB+nicbVBNS8NAFHzxs9avWI9egkWol5KIoN6KXjxWMLbQhLDZbtqlm03Y3Ygl5K948aDi1V/izX/jps1BWwcWhpn3eLMTpoxKZdvfxsrq2vrGZm2rvr2zu7dvHjQeZJIJTFycsET0QyQJo5y4iipG+qkgKA4Z6YWTm9LvPRIhacLv1TQlfoxGnEYUI6WlwGx43TFteTFS4zDKoyKgp4HZtNv2DNYycSrShArdwPzyhgnOYsIVZkjKgWOnys+RUBQzUtS9TJIU4QkakYGmHMVE+vkse2GdaGVoRYnQjytrpv7eyFEs5TQO9WQZUi56pfifN8hUdOnnlKeZIhzPD0UZs1RilUVYQyoIVmyqCcKC6qwWHiOBsNJ11XUJzuKXl4l71r5qO3fnzc511UYNjuAYWuDABXTgFrrgAoYneIZXeDMK48V4Nz7moytGtXMIf2B8/gD8DZP1 AAAB+nicbVBNS8NAFHzxs9avWI9egkWol5KIoN6KXjxWMLbQhLDZbtqlm03Y3Ygl5K948aDi1V/izX/jps1BWwcWhpn3eLMTpoxKZdvfxsrq2vrGZm2rvr2zu7dvHjQeZJIJTFycsET0QyQJo5y4iipG+qkgKA4Z6YWTm9LvPRIhacLv1TQlfoxGnEYUI6WlwGx43TFteTFS4zDKoyKgp4HZtNv2DNYycSrShArdwPzyhgnOYsIVZkjKgWOnys+RUBQzUtS9TJIU4QkakYGmHMVE+vkse2GdaGVoRYnQjytrpv7eyFEs5TQO9WQZUi56pfifN8hUdOnnlKeZIhzPD0UZs1RilUVYQyoIVmyqCcKC6qwWHiOBsNJ11XUJzuKXl4l71r5qO3fnzc511UYNjuAYWuDABXTgFrrgAoYneIZXeDMK48V4Nz7moytGtXMIf2B8/gD8DZP1 C = 1 N N X i=1 (fi) >(fi) AAACK3icbZDLSsNAFIYnXmu9RV26GSxC3ZREBHVRKHbjqlQwttCkYTKdtEMnF2YmQgl5IDe+iiAurLj1PZy0WdjWHwZ+vnMOc87vxYwKaRhTbW19Y3Nru7RT3t3bPzjUj46fRJRwTCwcsYh3PSQIoyGxJJWMdGNOUOAx0vHGzbzeeSZc0Ch8lJOYOAEahtSnGEmFXL3ZrEPb5winZpa2MlskgZvSupn1W3Z7RKt2gOTI81M/c+lFTvq2jOJF7OoVo2bMBFeNWZgKKNR29Xd7EOEkIKHEDAnRM41YOinikmJGsrKdCBIjPEZD0lM2RAERTjo7NoPnigygH3H1Qgln9O9EigIhJoGnOvMlxXIth//Veon0b5yUhnEiSYjnH/kJgzKCeXJwQDnBkk2UQZhTtSvEI6SikyrfsgrBXD551ViXtdua+XBVadwVaZTAKTgDVWCCa9AA96ANLIDBC3gDn2CqvWof2pf2PW9d04qZE7Ag7ecX5JCoQA== AAACK3icbZDLSsNAFIYnXmu9RV26GSxC3ZREBHVRKHbjqlQwttCkYTKdtEMnF2YmQgl5IDe+iiAurLj1PZy0WdjWHwZ+vnMOc87vxYwKaRhTbW19Y3Nru7RT3t3bPzjUj46fRJRwTCwcsYh3PSQIoyGxJJWMdGNOUOAx0vHGzbzeeSZc0Ch8lJOYOAEahtSnGEmFXL3ZrEPb5winZpa2MlskgZvSupn1W3Z7RKt2gOTI81M/c+lFTvq2jOJF7OoVo2bMBFeNWZgKKNR29Xd7EOEkIKHEDAnRM41YOinikmJGsrKdCBIjPEZD0lM2RAERTjo7NoPnigygH3H1Qgln9O9EigIhJoGnOvMlxXIth//Veon0b5yUhnEiSYjnH/kJgzKCeXJwQDnBkk2UQZhTtSvEI6SikyrfsgrBXD551ViXtdua+XBVadwVaZTAKTgDVWCCa9AA96ANLIDBC3gDn2CqvWof2pf2PW9d04qZE7Ag7ecX5JCoQA== AAACK3icbZDLSsNAFIYnXmu9RV26GSxC3ZREBHVRKHbjqlQwttCkYTKdtEMnF2YmQgl5IDe+iiAurLj1PZy0WdjWHwZ+vnMOc87vxYwKaRhTbW19Y3Nru7RT3t3bPzjUj46fRJRwTCwcsYh3PSQIoyGxJJWMdGNOUOAx0vHGzbzeeSZc0Ch8lJOYOAEahtSnGEmFXL3ZrEPb5winZpa2MlskgZvSupn1W3Z7RKt2gOTI81M/c+lFTvq2jOJF7OoVo2bMBFeNWZgKKNR29Xd7EOEkIKHEDAnRM41YOinikmJGsrKdCBIjPEZD0lM2RAERTjo7NoPnigygH3H1Qgln9O9EigIhJoGnOvMlxXIth//Veon0b5yUhnEiSYjnH/kJgzKCeXJwQDnBkk2UQZhTtSvEI6SikyrfsgrBXD551ViXtdua+XBVadwVaZTAKTgDVWCCa9AA96ANLIDBC3gDn2CqvWof2pf2PW9d04qZE7Ag7ecX5JCoQA== AAACK3icbZDLSsNAFIYnXmu9RV26GSxC3ZREBHVRKHbjqlQwttCkYTKdtEMnF2YmQgl5IDe+iiAurLj1PZy0WdjWHwZ+vnMOc87vxYwKaRhTbW19Y3Nru7RT3t3bPzjUj46fRJRwTCwcsYh3PSQIoyGxJJWMdGNOUOAx0vHGzbzeeSZc0Ch8lJOYOAEahtSnGEmFXL3ZrEPb5winZpa2MlskgZvSupn1W3Z7RKt2gOTI81M/c+lFTvq2jOJF7OoVo2bMBFeNWZgKKNR29Xd7EOEkIKHEDAnRM41YOinikmJGsrKdCBIjPEZD0lM2RAERTjo7NoPnigygH3H1Qgln9O9EigIhJoGnOvMlxXIth//Veon0b5yUhnEiSYjnH/kJgzKCeXJwQDnBkk2UQZhTtSvEI6SikyrfsgrBXD551ViXtdua+XBVadwVaZTAKTgDVWCCa9AA96ANLIDBC3gDn2CqvWof2pf2PW9d04qZE7Ag7ecX5JCoQA== PCA of • Nonlinear lifting & unlifting • Deterministic kernel • Difficulty in multilevel extension

Slide 30

Slide 30 text

Insights obtained from the analysis q Pooling layer à network architecture ü Frame condition qWhere to design neural network ? ü Where Hankel structured matrix is low-ranked q Other insights ü Concatenation layers ü Residual or direction mapping ?

Slide 31

Slide 31 text

APPLICATION-DRIVEN EVIDENCES

Slide 32

Slide 32 text

ALOHA CNN k-Space Deep Learning for Accelerated MRI

Slide 33

Slide 33 text

k-Space Deep Learning for Accelerated MRI Han et al, arXiv:1805.03779 ~3dB gain

Slide 34

Slide 34 text

Research Goal Ø To improve temporal resolution of TWIST imaging using deep k-space learning Ø To generate multiple reconstruction results with various spatial and temporal resolution using one network VS = 5 VS = 2 CNN K-space Deep Learning for Time-resolved MRI

Slide 35

Slide 35 text

K-space Deep Learning Results Conventional Solution  Solution

Slide 36

Slide 36 text

Low-Dose CT • To reduce the radiation exposure, sparse-view CT, low-dose CT and interior tomography. Sparse-view CT (Down-sampled View) Low-dose CT (Reduced X-ray dose) Interior Tomography (Truncated FOV)

Slide 37

Slide 37 text

WavResNet for low-dose CT (Kang et al, Medical Phyhsics, 2017, IEEE TMI 2018)

Slide 38

Slide 38 text

 Solution Conventional Solution Quater dose Full-dose

Slide 39

Slide 39 text

 Solution Conventional Solution Quater dose Full-dose

Slide 40

Slide 40 text

MBIR Our method C D

Slide 41

Slide 41 text

WHAT IF WE DON’T HAVE REFERENCE ?

Slide 42

Slide 42 text

Low-dose CT with Adversarial loss

Slide 43

Slide 43 text

Un-Supervised Learning for low-dose CT 43 • Multiphase Cardiac CT denoising – Phase 1, 2: low-dose, Phase 3 ~ 10: normal dose – Goal: dynamic changes of heart structure – No reference available Kang et al, Medical Physics (in press), 2018

Slide 44

Slide 44 text

44 (a) (b) (c) (d) (e) (f) (g) (h) (b) (c) (d) (f) (g) (h) GAN

Slide 45

Slide 45 text

45 • Cardiac CT denoising – Cycle Consistent Adversarial Denoising Network for Multiphase Coronary CT Angiography Un-supervised Learning using Cyclic-GAN

Slide 46

Slide 46 text

46 (a) (b) (c) (d) (e) (f) (g) (h) (b) (c) (d) (f) (g) (h) GAN

Slide 47

Slide 47 text

47 Input: phase 1 Denoised output Target: phase 8 Input- output

Slide 48

Slide 48 text

FBP Proposed ADMIRE

Slide 49

Slide 49 text

Unsupervised / supervised / ground-truth

Slide 50

Slide 50 text

1st view 2nd view 3rd view 4th view 5th view 6th view 7th view 8th view 9th view q AI-powered 9-View CT for Security Scanning Semi-supervised Learning Han et al, arXiv preprint arXiv:1712.10248, (2017). CT meeting 2018

Slide 51

Slide 51 text

Conventional Solution  Solution

Slide 52

Slide 52 text

Semi-Supervised High Resolution View Synthesis 128 256 64 128 6 4 256 256 512 512 512 1024 512 1024 512 256 512 256128 • Key idea • Training with measured views • Generate stacks of x-y images • Neural network training with theta-z images

Slide 53

Slide 53 text

Outlooks q End-to-End AI for radiological imaging Ø From AI-powered image acquisition to diagnosis for clear and rapid radiological imaging Existing AI Solutions: Diagnosis Our solution: from acquisition to diagnosis

Slide 54

Slide 54 text

math Thank You