Share
Embed
Deep Convolutional Neural Networks in industrial applications
Deep Learning is a new (and at the same time old) paradigm in machine learning which allows to extract features directly from huge amounts of raw data with a minimum of human interaction. This talk gives an introduction about deep learning in general and focuses an important application of deep convolutional neural networks (CNN) in industrial settings: optical quality control.
Munich DataGeeks
October 08, 2016
Transcript
50 slides
-
Deep Convolutional Neural Networks in industrial applications - Daniel Weimer
- Datageeks Data Day, 8th October 2016 – Munich -
Agenda 1. The machine learning pipeline 2. Deep Learning: A
new paradigm in AI 3. Deep Conv Nets for industrial inspection 4. Experiments, Results and Limits 5. Conclusion Daniel Weimer -
1. The machine learning pipeline Daniel Weimer Traditional Feature Extractor
Classifier Inspired by Y. LeCun -
1. The machine learning pipeline Daniel Weimer Traditional HoG (People
Detection) Viola&Jones (Face recognition) SIFT (Matching) Feature point histogram (3D Matching) Feature Extractor Classifier -
1. The machine learning pipeline Daniel Weimer Feature Extractor Classifier
Traditional Better Feature Extractor Classifier Mid-level Features • Hand crafted feature representations • Deep domain-specific knowledge for feature description is required (expert driven) • Take a lot of time to develop such features Inspired by Y. LeCun -
1. The machine learning pipeline Daniel Weimer Feature Extractor Classifier
Traditional Better Feature Extractor Classifier Mid-level Features Deep Learning Low-level Features Classifier Mid-level Features High-level Features Training / Learning Inspired by Y. LeCun -
1. The machine learning pipeline Daniel Weimer What might be
the ultimate result of an ML algorithm giving that image? -
1. The machine learning pipeline Daniel Weimer Person Ball Grass
Automated feature generation with rising complexity „4 persons playing football in a stadium“ -
1. The machine learning pipeline Daniel Weimer Can we learn
powerful features directly from data? Without the need of human interaction! Deep Machine Learning -
2. Deep Learning: A new paradigm in AI Why do
we need a new paradigm in AI? Daniel Weimer -
2. Deep Learning: A new paradigm in AI Why do
we need a new paradigm in AI? Daniel Weimer [Ng 2016] -
2. Deep Learning: A new paradigm in AI A Renaissance
of neural architectures: The basic motivation of deep learning are traditional neural networks x1 x2 x3 . . . xn y {0,1} +1 h(x) feature vector x neuron weights Daniel Weimer Bias -
2. Deep Learning: A new paradigm in AI Motivation: Raw
data will be transformed to complex features by stacking together non-linear feature extraction stages x1 x2 x3 . . . xn y {0,1} h(x) ... Daniel Weimer pixel vector x -
2. Deep Learning: A new paradigm in AI Training of
deep neural networks: Standard back-propagation of errors (backprop) x1 x2 x3 . . . xn y {0,1} h(x) y = h(x) = 1 0.8 0.2 error = ... Daniel Weimer pixel vector x -
2. Deep Learning: A new paradigm in AI Training of
deep neural networks: Standard back-propagation of errors (backprop) x1 x2 x3 . . . xn y {0,1} ... h(x) y = h(x) = 1 0.8 0.2 error = pixel vector x -
2. Deep Learning: A new paradigm in AI Training of
deep neural networks: Standard back-propagation of errors (backprop) x1 x2 x3 . . . xn y {0,1} ... h(x) y = h(x) = 1 0.8 0.2 error = pixel vector x Daniel Weimer Error become smaller smaller smaller smaller ...... Vanishing gradient problem -
2. Deep Learning: A new paradigm in AI Training of
deep neural networks: Method 1: Stacked autoencoder learning (around 2006) ... Daniel Weimer IN IN=OUT -
2. Deep Learning: A new paradigm in AI Training of
deep neural networks: Method 1: Stacked autoencoder learning (around 2006) ... Daniel Weimer + + fine tuning -
2. Deep Learning: A new paradigm in AI Training of
deep neural networks: Method 2: Dropout, Regularization and Activation x1 x2 x3 . . . xn y {0,1} ... Daniel Weimer active dropout -
2. Deep Learning: A new paradigm in AI Fully connected
layers are a huge problem by applying deep nets to images: Daniel Weimer -
2. Deep Learning: A new paradigm in AI Fully connected
layers are a huge problem by applying deep nets to images: Daniel Weimer -
2. Deep Learning: A new paradigm in AI Fully connected
layers are a huge problem by applying deep nets to images: Daniel Weimer -
2. Deep Learning: A new paradigm in AI From deep
neural networks to deep convolutional neural networks (ConvNets) Reducing the number of neuron weights by receptive field approach layer-1 layer-2 FC-ANN [LeCun et.al, 1989] features model Daniel Weimer -
2. Deep Learning: A new paradigm in AI From deep
neural networks to deep convolutional neural networks (ConvNets) Reducing the number of neuron weights by receptive field approach Activation Convolution Pooling layer-1 layer-2 FC-ANN <Conv> <Pool> <ReLU> [LeCun et.al, 1989] Daniel Weimer -
2. Deep Learning: A new paradigm in AI From deep
neural networks to deep convolutional neural networks (ConvNets) Reducing the number of neuron weights by receptive field approach Activation Convolution Pooling layer-1 layer-2 FC-ANN • K neurons learn n×n filter per layer • Step width of S pixel per filter • Number of layers: m • Padding in border region of the image Design- Hyperparameter In this work: n×n = 3×3 * *[Simonyan u. Zisserman 2014] Daniel Weimer -
2. Deep Learning: A new paradigm in AI From deep
neural networks to deep convolutional neural networks (ConvNets) Reducing the number of neuron weights by receptive field approach Activation Convolution Pooling layer-1 layer-2 FC-ANN • Pooling of max(2×2) neighboaring pixel • Step width of 1 pixel Reducing image information by a factor of 2 in width and height Design- Hyperparameter Daniel Weimer -
2. Deep Learning: A new paradigm in AI From deep
neural networks to deep convolutional neural networks (ConvNets) Reducing the number of neuron weights by receptive field approach Activation Convolution Pooling layer-1 layer-2 FC-ANN Daniel Weimer -
3. Deep Conv Nets for industrial inspection Defects on surfaces
are varying with respect to • Size • Geometry • Orientation • Different background textures Daniel Weimer -
3. Deep Conv Nets for industrial inspection How to apply
deep learning in industrial inspection? Daniel Weimer -
3. Deep Conv Nets for industrial inspection Daniel Weimer Defect
detection is a classification problem: good/bad classification example x2 Classification problem 1 x1 -
3. Deep Conv Nets for industrial inspection Daniel Weimer Defect
detection is a classification problem: good/bad classification example x2 Classification problem 1 x1 -
3. Deep Conv Nets for industrial inspection Daniel Weimer Defect
detection is a classification problem: Hyperplane, model ? x1 x2 good/bad classification example Classification problem 1 Supervised training 2 Training Examples (historical data) Modell offline online defect -
1. Data representation and augmentation Block extraction Augmentation Shuffle the
data Dimensionality of training data: px × py × 1.299.200 3. Deep Conv Nets for industrial inspection Daniel Weimer -
4. Experiments, results and limits 2. Network architectures Basis Tiefe-1
Tiefe-2 Tiefe-3 Stufe-1 conv3-NK1 max-pooling conv3-NK1 conv3-NK1 max-Pooling conv3-NK1 conv3-NK1 pad max-Pooling conv3-NK1 conv3-NK1 max-Pooling Stufe-2 conv3-NK2 max-pooling conv3-NK2 conv3-NK2 max-Pooling conv3-NK2 conv3-NK2 conv3-NK2 max-Pooling conv3-NK2 conv3-NK2 conv3-NK2 max-Pooling Stufe-3 conv3-NK3 max-pooling conv3-NK3 max-pooling conv3-NK3 conv3-NK3 max-pooling conv3-NK3 conv3-NK3 conv3-NK3 conv3-NK3 Max-pooling NN-1024 NN-1024 NN-12 Softmax Zusätzliche <Conv> <Pool> <ReLU> Schichten Tiefe Zusätzliche Neuronen pro Schicht Breite Number of neurons NK1 NK2 NK3 Basis 6 16 120 Width-1 16 32 256 Width-2 32 64 512 Width-3 64 128 256 Daniel Weimer -
4. Experiments, results and limits 2. Network architectures Basis Depth-1
Tiefe-2 Tiefe-3 Stufe-1 conv3-NK1 max-pooling conv3-NK1 conv3-NK1 max-pooling conv3-NK1 conv3-NK1 pad max-Pooling conv3-NK1 conv3-NK1 max-Pooling Stufe-2 conv3-NK2 max-pooling conv3-NK2 conv3-NK2 max-pooling conv3-NK2 conv3-NK2 conv3-NK2 max-Pooling conv3-NK2 conv3-NK2 conv3-NK2 max-Pooling Stufe-3 conv3-NK3 max-pooling conv3-NK3 max-pooling conv3-NK3 conv3-NK3 max-pooling conv3-NK3 conv3-NK3 conv3-NK3 conv3-NK3 Max-pooling NN-1024 NN-1024 NN-12 Softmax Zusätzliche <Conv> <Pool> <ReLU> Schichten Tiefe Zusätzliche Neuronen pro Schicht Breite Number of neurons NK1 NK2 NK3 Basis 6 16 120 Width-1 16 32 256 Width-2 32 64 512 Width-3 64 128 256 Daniel Weimer -
4. Experiments, results and limits 2. Network architectures Basis Depth-1
Depth-2 Depth-3 Stufe-1 conv3-NK1 max-pooling conv3-NK1 conv3-NK1 max-pooling conv3-NK1 conv3-NK1 pad max-pooling conv3-NK1 conv3-NK1 max-pooling Stufe-2 conv3-NK2 max-pooling conv3-NK2 conv3-NK2 max-pooling conv3-NK2 conv3-NK2 conv3-NK2 max-pooling conv3-NK2 conv3-NK2 conv3-NK2 max-pooling Stufe-3 conv3-NK3 max-pooling conv3-NK3 max-pooling conv3-NK3 conv3-NK3 max-pooling conv3-NK3 conv3-NK3 conv3-NK3 conv3-NK3 max-pooling NN-1024 NN-1024 NN-12 Softmax Additional <Conv> <Pool> <ReLU> layer Depth Additional neurons per layer Width Number of neurons NK1 NK2 NK3 Basis 6 16 120 Width-1 16 32 256 Width-2 32 64 512 Width-3 64 128 256 Daniel Weimer -
4. Experiments, results and limits Influence of number of layers
and neurons per layer Depth Highest precision Number of parameter (Million) Feature dimension Runtime per block Basis 95,68% 6,98 NK3 × 12 × 4 0,83ms Depth-1 96,44% 5,14 NK3 × 11 × 3 0,78ms Depth-2 96,87% 2,32 NK3 × 9 × 1 0,69ms Depth-3 97,03% 3,98 NK3 × 7 × 3 1,01ms Daniel Weimer -
4. Experiments, results and limits Influence of number of layers
and neurons per layer 1,4% 95,68% 97,03% ~ 1 - Depth Highest precision Number of parameter (Million) Feature dimension Runtime per block Basis 95,68% 6,98 NK3 × 12 × 4 0,83ms Depth-1 96,44% 5,14 NK3 × 11 × 3 0,78ms Depth-2 96,87% 2,32 NK3 × 9 × 1 0,69ms Depth-3 97,03% 3,98 NK3 × 7 × 3 1,01ms Daniel Weimer -
4. Experiments, results and limits ~1,2% Influence of number of
layers and neurons per layer 1,4% ~ Depth Highest precision Number of parameter (Million) Feature dimension Runtime per block Basis 95,68% 6,98 NK3 × 12 × 4 0,83ms Depth-1 96,44% 5,14 NK3 × 11 × 3 0,78ms Depth-2 96,87% 2,32 NK3 × 9 × 1 0,69ms Depth-3 97,03% 3,98 NK3 × 7 × 3 1,01ms • Runtime is driven by depth of layer • As more layers, as more complex (and compressed) the features Daniel Weimer -
4. Experiments, results and limits • Runtime is driven by
depth of layer • As more layers, as more complex (and compressed) the features Influence of number of layers and neurons per layer Just a small step in precision, despite additional layers • Limit of ConvNets with respect to depth • Assumption: limit of geometric complexity leads to saturation effects in high layers ~1,2% 1,4% ~ Depth Highest precision Number of parameter (Million) Feature dimension Runtime per block Basis 95,68% 6,98 NK3 × 12 × 4 0,83ms Depth-1 96,44% 5,14 NK3 × 11 × 3 0,78ms Depth-2 96,87% 2,32 NK3 × 9 × 1 0,69ms Depth-3 97,03% 3,98 NK3 × 7 × 3 1,01ms Daniel Weimer -
4. Experiments, results and limits Highest precision and time/block Width
NK Depth-2 Depth-3 Basis {6,16,120} 96,87% @ 0,69ms 97,03% @ 1,01ms Width-1 {16,32,256} 96,94% @ 0,98ms 97,19% @ 1,60ms Width-2 {32,64,512} 97,13% @ 1,65ms 97,36% @ 3,25ms Width-3 {64,128,256} 97,19% @ 1,42ms 97,37% @ 2,00ms ~3,3% ~5,1% Influence of number of layers and neurons per layer Depth Highest precision Number of parameter (Million) Feature dimension Runtime per block Basis 95,68% 6,98 NK3 × 12 × 4 0,83ms Depth-1 96,44% 5,14 NK3 × 11 × 3 0,78ms Depth-2 96,87% 2,32 NK3 × 9 × 1 0,69ms Depth-3 97,03% 3,98 NK3 × 7 × 3 1,01ms Daniel Weimer -
4. Experiments, results and limits Benchmark with DAGM dataset Class
Deep Learning Stat. features + SVM SIFT + NN** Weibull features*** 1 100 99,5 98,9 87 2 100 94,3 95,7 - 3 95,5 99,5 98,5 99,8 4 100 92,5 - - 5 98,8 96,9 98,2 97,2 6 100 100 99,8 94,9 1 100 99,7 100 98 2 97,3 80 91,3 - 3 100 100 100 100 4 98,7 96,1 - - 5 100 96,1 100 100 6 99,5 96,2 100 100 TPR TNR ***[Timm u. Barth 2011] **[Siebel u. Sommer 2008] *[Scholz-Reiter et. al 2012] Daniel Weimer -
5. Conclusion Conclusion • Deep Learning pushes Machine Learning to
the next level: – OLD: Design features on raw data level (pixel, time steps, letters, ….. ) – NEW: Design of architectures that automatically process data (we don‘t touch raw data) • When to choose Deep Learning: You have enough training data (with or without label) Massively parallel hardware support (for training) Your problem can be represented in evolutionary stages Manual feature representation is expensive or saturated Daniel Weimer -
Daniel Weimer 27 Literatur LeCun, Y. ; Boser, B. ;
Denker, J.S. ; Henderson, D. ; Howard, R.E. ; Hubbard, W. ; Jackel, D.L.: Backpropagation applied to hand-written zip code recogition. In: Neural Computation 1 (1989), Nr. 4, S. 541–551 [LeCun et. al, 1989] Xie, X.: A Review of Recent Advances in Surface Defect Detection using Texture analysis Techniques. In: Electronic Letters on Computer Vision and Image Analysis 7 (2008), Nr. 3. – ISSN 1577–5097 [Xie, 2008] Siebel, N.T ; Sommer, G.: Learning Defect Classifiers for Visual Inspection Images by Neuro- Evolution using Weakly Labelled Training Data. In: Evolutionary Computation, 2008, S. 3925 – 3931 [Siebel u. Sommer 2008] Timm, F. ; Barth, E.: Non-parametric texture defect detection using Weibull features. In: IS&T/SPIE Electronic Imaging, International Society for Optics and Photonics, 2011 [Timm u. Barth 2011] Ng, A.: Deep Learning. http://cs229.stanford.edu/materials/CS229-DeepLearning.pdf Lecture at Stanford University [Ng 2015] Karimi, M.H. ; Asemani, D.: Surface defect detection in tiling Industries using digital image processing methods: Analysis and evaluation. In: ISA Transactions 53 (2014), Nr. 3, S. 834– 844. – ISSN 0019–0578 [Karimi u. Asemani 2014] Simonyan, K. ; Zisserman, A.: Very Deep Convolutional Networks for Large-Scale Image Recognition. In: CoRR abs/1409.1556 (2014) [Simonyan u. Zisserman 2014]