Jong Chul Ye KAIST Bio-Imaging & Signal Processing Lab. Dept. Bio & Brain Engineering KAIST, Korea D De ee ep p L Le ea ar rn ni in ng g f fo or r C CT T R Re ec co on ns st tr ru uc ct ti io on n
* NMSE is wri9en at the corner. 2 2. .1 12 20 03 3e e- -1 1 X : Input (48) Ground truth (380th) Han and Ye, arxiv:1611.06391,2016, Jin et al, IEEE TIP, 2017 U-Net for Sparse View CT 6 6. .2 22 27 77 7e e- -3 3 U-Net
• Successful demonstraMon of deep learning for various image reconstrucMon problems – Low-dose x-ray CT (Kang et al, Chen et al, Wolterink et al) – Sparse view CT (Jin et al, Han et al, Adler et al) – Interior tomography (Han et al) – Stationary CT for baggage inspection (Han et al) – CS-MRI (Hammernik et al, Yang et al, Lee et al, Zhu et al) – US imaging (Yoon et al ) – Diffuse optical tomography (Yoo et al) • Advantages – Very fast reconstruction time – Significantly improved results Other works
Why Deep Learning works for recon ? Existing views 1: unfolded iteration • Most prevailing views • Direct connecMon to sparse recovery – Cannot explain the filter channels Jin, arXiv:1611.03679
Why Deep Learning works for recon ? Existing views 2: generative model • Image reconstruc5on as a distribu5on matching – However, difficult to explain the role of black-box network Bora et al, Compressed Sensing using Generative Models, arXiv:1703.03208
Still Unresolved Problems.. • What is the role of the nonlinearity such as rectified linear unit (ReLU) ? • What is the role of the filter channels in convolutional layer ? • Why do we need a pooling and unpooling in some architectures ? • Why do some networks need fully connected layers whereas the others do not ? • What is the role of by-pass connection or residual network ?
Our Proposal: Deep Learning == Deep Convolutional Framelets • Ye et al, “Deep convolutional framelets: A general deep learning framework for inverse problems”, SIAM Journal Imaging Sciences (in press), 2018.
Missing elements can be found by low rank Hankel structured matrix compleMon Nuclear norm Projec5on on sampling posi5ons min m kH ( m ) k⇤ subject to P⌦(b) = P⌦( f ) Rank H ( f ) = k * Jin KH et al IEEE TCI, 2016 * Jin KH et al.,IEEE TIP, 2015 * Ye JC et al., IEEE TIT, 2016 m Annihilating filter-based low-rank Hankel matrix
Deep Convolutional Framelets (Y, Han, Cha; 2018) H d( f ) H d( f ) = ˜ T ˜ T C C = Φ T H d( f ) C = Φ T ( f ~ ) Encoder: ˜ T = I ˜ = PR(V ) H d( f ) = U ⌃ V T Decoder: f = (˜ Φ C ) ~ ⌧ (˜ ) : Non-local basis : Local basis : Generalized pooling : CNN filters : Frame condition : Projection condition
Problem of U-net Pooling does NOT satisfy the frame condition JC Ye et al, SIAM Journal Imaging Sciences, 2018 Y. Han and J. C. Ye, arXiv preprint arXiv:1708.08333, 2017. ext > ext = I + > 6 = I
Improving U-net using Deep Conv Framelets • Dual Frame U-net • Tight Frame U-net JC Ye et al, SIAM Journal Imaging Sciences, 2018 Y. Han and J. C. Ye, arXiv preprint arXiv:1708.08333, 2017.
90 view recon U-Net vs. Tight-Frame U-Net • JC Ye et al, SIAM Journal Imaging Sciences, 2018 • Y. Han and J. C. Ye, arXiv preprint arXiv:1708.08333, 2017.
Conclusions: • Deep learning has demonstrated significant performance improvement in CT reconstruction problems • Deep convolutional framelets is a mathematical framework to understand deep learning for inverse problems – Novel data-driven signal representation – Structured low-rank shrinkage by controlling the no. of filter channels – Frame condition is important • Our analysis shows that “correctly designed” deep network is NOT a black-box and is SAFE for medical imaging reconstruction
jongcye
ryou0634
akihikowatanabe
bollegala
kuri8ive
eumesy
kyoun
inoichan
kuribayashi4
schulta
kentaitakura
kosuken
searls
jasonvnalue
keathley
sstephenson
tammielis
cherdarchuk
chriscoyier
sachag
davidbonilla
dotmariusz
michaelherold
bkeepers