Share
Embed
Deep Learning for CT Reconstruction: From Concept to Practices
Refresher Course by Jong Chul Ye, CT Meeting 2018, The Fifth International Conference on Image Formation in X-Ray Computed Tomography, May 20-23, 2018, Salt Lake City, USA
Jong Chul Ye
May 23, 2018
Transcript
91 slides
-
Jong Chul Ye Bio-Imaging & Signal Processing Lab. Dept. Bio
& Brain Engineering Dept. Mathematical Sciences KAIST, Korea Refresher Course Deep Learning for CT Reconstruction: From Concept to Practices This can be downloaded from http://bispl.weebly.com -
Course Overview • Introduction • CNN review • Deep learning:
biological origin • Deep learning: mathematical origin • Applications to CT • Programming Example -
Deep Learning Age • Deep learning has been successfully used
for classification, low-level computer vision, etc • Even outperforms human observers -
• Successful demonstra5on of deep learning for various image reconstruc5on
problems – Low-dose x-ray CT (Kang et al, Chen et al, Wolterink et al) – Sparse view CT (Jin et al, Han et al, Adler et al) – Interior tomography (Han et al) – Stationary CT for baggage inspection (Han et al) – CS-MRI (Hammernik et al, Yang et al, Lee et al, Zhu et al) – US imaging (Yoon et al ) – Diffuse optical tomography (Yoo et al) – Elastic tomography (Yoo et al) – etc • Advantages – Very fast reconstruction time – Significantly improved results Other works -
WHY DEEP LEARNING WORKS FOR RECON ? DOES IT CREATE
ANY ARTIFICIAL FEATURES ? -
Too Simple to Analyze.. Convolution & pooling à stone age
tools of signal processing What do they do ? -
Visual Information Processing in Brain 17 Kravitz et al, Trends
in Cognitive Sciences January 2013, Vol. 17, No. 1 -
Retina, V1 Layer 18 Receptive fields of two ganglion cells
in retina à convolution Orientation column in V1 http://darioprandi.com/docs/talks/image-reconstruction-recognition/graphics/pinwheels.jpg Figure courtesy by distillery.com -
“The Jennifer Anniston Cell” 19 Quiroga et al, Nature, Vol
435, 24, June 2005 -
Why Deep Learning works for recon ? Existing views 1:
unfolded iteration • Most prevailing views • Direct connec5on to sparse recovery – Cannot explain the filter channels Jin, arXiv:1611.03679 -
Why Deep Learning works for recon ? Existing views 2:
generative model • Image reconstruc5on as a distribu5on matching – However, difficult to explain the role of black-box network Bora et al, Compressed Sensing using Generative Models, arXiv:1703.03208 -
Our Proposal: Deep Learning == Deep Convolutional Framelets • Ye
et al, “Deep convolutional framelets: A general deep learning framework for inverse problems”, SIAM Journal Imaging Sciences, 11(2), 991-1048, 2018. -
Why we are excited about Hankel matrix ? T -T
0 n1 -n1 0 * FRI Sampling theory (VeEerlie et al) and compressed sensing -
︙ ︙ 1 2 3 4 5 6 7 8
9 -1 0 1 2 3 4 5 6 7 8 9 1 2 3 4 5 6 7 8 9 12 2 3 4 5 6 7 8 9 12 13 3 4 5 6 7 8 9 10 10 10 10 0 11 11 11 1 2 3 4 5 Finite length convolution Matrix Representation * ALOHA : Annihilating filter based LOw rank Hankel matrix Approach * Jin KH et al. IEEE TCI, 2016 * Jin KH et al.,IEEE TIP, 2015 * Ye JC et al. IEEE TIT, 2016 Annihilating filter-based low-rank Hankel matrix -
Missing elements can be found by low rank Hankel structured
matrix comple5on Nuclear norm Projec5on on sampling posi5ons min m kH ( m ) k⇤ subject to P⌦(b) = P⌦( f ) RankH(f) = k * Jin KH et al IEEE TCI, 2016 * Jin KH et al.,IEEE TIP, 2015 * Ye JC et al., IEEE TIT, 2016 m Annihilating filter-based low-rank Hankel matrix -
Key Observation Data-Driven Hankel matrix decomposition => Deep Learning •
Ye et al, “Deep convolutional framelets: A general deep learning framework for inverse problems”, SIAM Journal Imaging Sciences, 11(2), 991-1048, 2018. -
H d(f) H d(f) = ˜ T ˜ T C
C = T H d(f) C = T (f ~ ) Encoder: ˜ T = I ˜ = PR(V ) H d(f) = U⌃V T Unlifting: f = (˜C) ~ ⌧(˜ ) : Non-local basis : Local basis : Frame condition : Low-rank condition convolution pooling un-pooling convolution : User-defined pooling : Learnable filters H pi (gi) = X k,l [Ci]kl e Bkl i Decoder: Deep Convolutional Framelets (Y, Han, Cha; 2018) -
Role of ReLU: Conic encoding D. D. Lee and H.
S. Seung, Nature, 1999 ReLU: positive framelet coefficients Conic encoding à part by representation similar to visual cortex -
https://www.youtube.com/watch?v=DdC0QN6f3G4 Relativity: Lifting to the Space-time à linear trajectory High
Dimensional Lifting in Relativity Falling apple Apple at rest Universal law of gravity: 3-D Space à curved trajectory Falling apple Apple at rest -
Compressed Sensing Hankel Structured Matrix Comple7on Deep Learning 49 From
CS to Deep Learning: Coherent Theme -
Low-Dose CT • To reduce the radiation exposure, sparse-view CT,
low-dose CT and interior tomography. Sparse-view CT (Down-sampled View) Low-dose CT (Reduced X-ray dose) Interior Tomography (Truncated FOV) -
52 Wavelet transform level 2 level 1 level 3 level
4 Wavelet recomposition + Residual learning : Low-resolution image bypass High SNR band CNN (Kang, et al, Medical Physics 44(10)) -
Sparse-View CT • To reduce the radiation exposure, sparse-view CT,
low-dose CT and interior tomography. Sparse-view CT (Down-sampled View) Low-dose CT (Reduced X-ray dose) Interior Tomography (Truncated FOV) -
Problem of U-net Pooling does NOT satisfy the frame condition
JC Ye et al, SIAM Journal Imaging Sciences, 2018 Y. Han et al, TMI, 2018. ext > ext = I + > 6= I -
Improving U-net using Deep Conv Framelets • Dual Frame U-net
• Tight Frame U-net JC Ye et al, SIAM Journal Imaging Sciences, 2018 Y. Han and J. C. Ye, TMI, 2018 -
Tight-Frame U-Net JC Ye et al, SIAM Journal Imaging Sciences,
2018 Han et al, TMI, 2018 -
90 view recon U-Net vs. Tight-Frame U-Net • JC Ye
et al, SIAM Journal Imaging Sciences, 2018 • Y. Han and J. C. Ye, TMI, 2018 -
1st view 2nd view 3rd view 4th view 5th view
6th view 7th view 8th view 9th view -
ROI Reconstruction • To reduce the radiation exposure, sparse-view CT,
low-dose CT and interior tomography. Sparse-view CT (Down-sampled View) Low-dose CT (Reduced X-ray dose) Interior Tomography (Truncated FOV) -
Deep Learning Interior Tomography Han et al, arXiv preprint arXiv:1712.10248,
(2017): CT meeting 2018. -
Still Unresolved Problems.. • Cascaded geometry of deep neural network
• Generalization capability • Optimization landscape • Training procedure • Extension to classification problems -
Dataset of Natural Images • MNIST (http://yann.lecun.com/exdb/mnist/) – Handwritten digits.
• SVHN (http://ufldl.stanford.edu/housenumbers/) – House numbers from Google Street View. • ImageNet (http://www.image-net.org/) – The de-facto image dataset. • LSUN (http://lsun.cs.princeton.edu/2016/) – Large-scale scene understanding challenge. • Pascal VOC ( http://host.robots.ox.ac.uk/pascal/VOC/) – Standardized image dataset. • MS COCO (http://cocodataset.org/#home) – Common Objects in Context. • CIFAR-10 / -100 (https://www.cs.utoronto.ca/~kriz/cifar.html) – Tiny images data set. • BSDS300 / 500 ( https://www2.eecs.berkeley.edu/Research/ Projects/CS/vision/grouping/resources.html) – Contour detection and image Segmentation resources. https://en.wikipedia.org/wiki/List_of_datasets_for_machine_learning_research https://www.kaggle.com/datasets -
Dataset for Medical Images • HCP (https://www.humanconnectome.org/) – Behavioral and
3T / 7T MR imaging dataset. • MRI Data (http://mridata.org/) – Raw k-space dataset acquired on a GE clinical 3T scanner. • LUNA (https://luna16.grand-challenge.org/data/) – Lung Nodule analysis dataset acquired on a CT scanner. • Data Science Bowl (https://www.kaggle.com/c/data-science-bowl-2017) – A thousand low-dose CT images. • NIH Chest X-rays (https://nihcc.app.box.com/v/ChestXray-NIHCC) – X-ray images with disease labels. http://www.cancerimagingarchive.net/ https://www.kaggle.com/datasets • TCIA Collections (http://www.cancerimagingarchive.net/) • De-identifies and hosts a large archive of medical images of cancer accessible for public download. • The data are organized as “Collections”, typically patients related by a common disease, image modality (MRI, CT, etc). -
Libraries for Deep learning • TensorFlow (https://www.tensorflow.org/) – Python •
Theano ( http://deeplearning.net/software/theano/) – Python • Keras (https://keras.io/) – Python • Caffe (http://caffe.berkeleyvision.org/) – Python • Torch (or PyTorch) (http://torch.ch/) – C / C++ (or Python) • Deeplearning4J ( https://deeplearning4j.org/) – Java • Microsoft Cognitive Toolkit (CNTK) ( https://www.microsoft.com/en-us/cognitive- toolkit/) – Python / C / C++ • MatConvNet ( http://www.vlfeat.org/matconvnet/) – Matlab -
Step 1: Compile the toolbox 1. Unzip the MatConvNet toolbox.
2. Open ‘vl_compilenn.m’ in Matlab. -
Step 1: Compile the toolbox (cont.) 3. Check the options
such as enableGpu and enableCudnn. 4. Run the ‘vl_compilenn.m’. * To use GPU processing (false true), you must have CUDA installed. ( https://developer.nvidia.com/cuda-90-download-archive ) ** To use cuDNN library (false true), you must have cuDNN installed. ( https://developer.nvidia.com/cudnn ) -
Step 2: Prepare dataset As a classification example, MNIST consists
of as follows, Images % struct-type Data % handwritten digit image labels % [1, …, 10] set % [1, 2, 3], 1, 2, and 3 indicate train, valid, and test set, respectively. Data Labels 6 -
Step 2: Prepare dataset (cont.) • As a segmentation example,
U-Net dataset consists of as follows, – Images % struct-type Ø data % Cell image Ø labels % [1, 2], Mask image. 1 and 2 indicate back- and for-ground, respectively. Ø set % [1, 2, 3] Data Labels -
Step 3: Implementation of the network architecture • Developers only
need to program the network architecture code because MatConvNet supports the network training framework. Support famous network architectures, such as alexnet, vggnet, resnet, inceptionent, and so on. -
Step 3: Implementation of the architecture (cont.) – The implementation
details of U-Net U-Net can be implemented, recursively. Stage 0 Stage 1 Stage 2 Stage 3 Stage 4 -
Step 3: Implementation of the architecture (cont.) 1. Create objects
of network and layers. Encoder Part Skip + Concat Part Decoder Part • The structure of Stage 0 Network Part -
Step 3: Implementation of the architecture (cont.) 2. Connect each
layers. • The structure of Stage 0 Layer Name ( string-type ) Layer object ( object ) Input Name ( string-type ) Output Name ( string-type ) Parameters Name ( string-type ) All objects and names must be unique. -
Step 3: Implementation of the architecture (cont.) 3. Implement recurrently
the each stages and add a loss function. Previous parts (3.1 and 3.2) become functional as ‘add_block_unet’. -
Step 4: Network hyper-parameter set up • MatConvNet supports the
default hyper-parameters as follows, Refer the cnn_train.m ( or cnn_train_dag.m ) The supported hyper-parameters 1. The size of mini-batch 2. The number of epochs 3. Learning rate 4. Weight decay factor 5. Solvers such as SGD, AdaDelta, AdaGrad, Adam, and RMS The kind of Optimization Solvers -
Step 5: Run the training script 1. Training script 2.
Training loss 3. Training loss graph • Blue : train • Orange : valid -
Acknowledgements CT Team • Yoseob Han • Eunhee Kang •
Jawook Goo US Team • Shujaat Khan • Jaeyong Hur MR Team • Dongwook Lee • Juyoung Lee • Eunju Cha • Byung-hoon Kim Image Analysis Team • Boa Kim • Junyoung Kim Optics Team • Sungjun Lim • Junyoung Kim • Jungsol Kim • Taesung Kwon