Deep Convolutional Neural Networks in industrial applications

Deep Learning is a new (and at the same time old) paradigm in machine learning which allows to extract features directly from huge amounts of raw data with a minimum of human interaction. This talk gives an introduction about deep learning in general and focuses an important application of deep convolutional neural networks (CNN) in industrial settings: optical quality control.

Transcript

1. Deep Convolutional Neural Networks in industrial applications - Daniel Weimer - Datageeks Data Day, 8th October 2016 – Munich
2. Agenda 1. The machine learning pipeline 2. Deep Learning: A new paradigm in AI 3. Deep Conv Nets for industrial inspection 4. Experiments, Results and Limits 5. Conclusion Daniel Weimer
3. 1. The machine learning pipeline Daniel Weimer Traditional Feature Extractor Classifier Inspired by Y. LeCun
4. 1. The machine learning pipeline Daniel Weimer Traditional HoG (People Detection) Viola&Jones (Face recognition) SIFT (Matching) Feature point histogram (3D Matching) Feature Extractor Classifier
5. 1. The machine learning pipeline Daniel Weimer Feature Extractor Classifier Traditional Better Feature Extractor Classifier Mid-level Features • Hand crafted feature representations • Deep domain-specific knowledge for feature description is required (expert driven) • Take a lot of time to develop such features Inspired by Y. LeCun
7. 1. The machine learning pipeline Daniel Weimer Feature Extractor Classifier Traditional Better Feature Extractor Classifier Mid-level Features Deep Learning Low-level Features Classifier Mid-level Features High-level Features Training / Learning Inspired by Y. LeCun
8. 1. The machine learning pipeline Daniel Weimer What might be the ultimate result of an ML algorithm giving that image?
9. 1. The machine learning pipeline Daniel Weimer Person Ball Grass Automated feature generation with rising complexity „4 persons playing football in a stadium“
10. 1. The machine learning pipeline Daniel Weimer Can we learn powerful features directly from data? Without the need of human interaction! Deep Machine Learning
11. 2. Deep Learning: A new paradigm in AI Why do we need a new paradigm in AI? Daniel Weimer
12. 2. Deep Learning: A new paradigm in AI Why do we need a new paradigm in AI? Daniel Weimer [Ng 2016]
13. 2. Deep Learning: A new paradigm in AI A Renaissance of neural architectures: The basic motivation of deep learning are traditional neural networks x1 x2 x3 . . . xn y {0,1} +1 h(x) feature vector x neuron weights Daniel Weimer Bias
14. 2. Deep Learning: A new paradigm in AI Motivation: Raw data will be transformed to complex features by stacking together non-linear feature extraction stages x1 x2 x3 . . . xn y {0,1} h(x) ... Daniel Weimer pixel vector x
15. 2. Deep Learning: A new paradigm in AI Training of deep neural networks: Standard back-propagation of errors (backprop) x1 x2 x3 . . . xn y {0,1} h(x) y = h(x) = 1 0.8 0.2 error = ... Daniel Weimer pixel vector x
16. 2. Deep Learning: A new paradigm in AI Training of deep neural networks: Standard back-propagation of errors (backprop) x1 x2 x3 . . . xn y {0,1} ... h(x) y = h(x) = 1 0.8 0.2 error = pixel vector x
17. 2. Deep Learning: A new paradigm in AI Training of deep neural networks: Standard back-propagation of errors (backprop) x1 x2 x3 . . . xn y {0,1} ... h(x) y = h(x) = 1 0.8 0.2 error = pixel vector x Daniel Weimer Error become smaller smaller smaller smaller ...... Vanishing gradient problem
18. 2. Deep Learning: A new paradigm in AI Training of deep neural networks: Method 1: Stacked autoencoder learning (around 2006) ... Daniel Weimer IN IN=OUT
19. 2. Deep Learning: A new paradigm in AI Training of deep neural networks: Method 1: Stacked autoencoder learning (around 2006) ... Daniel Weimer + + fine tuning
20. 2. Deep Learning: A new paradigm in AI Training of deep neural networks: Method 2: Dropout, Regularization and Activation x1 x2 x3 . . . xn y {0,1} ... Daniel Weimer active dropout
21. 2. Deep Learning: A new paradigm in AI Fully connected layers are a huge problem by applying deep nets to images: Daniel Weimer
22. 2. Deep Learning: A new paradigm in AI Fully connected layers are a huge problem by applying deep nets to images: Daniel Weimer
23. 2. Deep Learning: A new paradigm in AI Fully connected layers are a huge problem by applying deep nets to images: Daniel Weimer
24. 2. Deep Learning: A new paradigm in AI From deep neural networks to deep convolutional neural networks (ConvNets) Reducing the number of neuron weights by receptive field approach layer-1 layer-2 FC-ANN [LeCun et.al, 1989] features model Daniel Weimer
25. 2. Deep Learning: A new paradigm in AI From deep neural networks to deep convolutional neural networks (ConvNets) Reducing the number of neuron weights by receptive field approach Activation Convolution Pooling layer-1 layer-2 FC-ANN <Conv> <Pool> <ReLU> [LeCun et.al, 1989] Daniel Weimer
26. 2. Deep Learning: A new paradigm in AI From deep neural networks to deep convolutional neural networks (ConvNets) Reducing the number of neuron weights by receptive field approach Activation Convolution Pooling layer-1 layer-2 FC-ANN • K neurons learn n×n filter per layer • Step width of S pixel per filter • Number of layers: m • Padding in border region of the image Design- Hyperparameter In this work: n×n = 3×3 * *[Simonyan u. Zisserman 2014] Daniel Weimer
27. 2. Deep Learning: A new paradigm in AI From deep neural networks to deep convolutional neural networks (ConvNets) Reducing the number of neuron weights by receptive field approach Activation Convolution Pooling layer-1 layer-2 FC-ANN • Pooling of max(2×2) neighboaring pixel • Step width of 1 pixel  Reducing image information by a factor of 2 in width and height Design- Hyperparameter Daniel Weimer
28. 2. Deep Learning: A new paradigm in AI From deep neural networks to deep convolutional neural networks (ConvNets) Reducing the number of neuron weights by receptive field approach Activation Convolution Pooling layer-1 layer-2 FC-ANN Daniel Weimer
29. 3. Deep Conv Nets for industrial inspection Daniel Weimer
30. 3. Deep Conv Nets for industrial inspection Daniel Weimer [DAGM dataset 2006]
31. 3. Deep Conv Nets for industrial inspection Daniel Weimer [DAGM dataset 2006]
32. 3. Deep Conv Nets for industrial inspection Defects on surfaces are varying with respect to • Size • Geometry • Orientation • Different background textures Daniel Weimer
33. 3. Deep Conv Nets for industrial inspection How to apply deep learning in industrial inspection? Daniel Weimer
34. 3. Deep Conv Nets for industrial inspection Daniel Weimer Defect detection is a classification problem: good/bad classification example x2 Classification problem 1 x1
35. 3. Deep Conv Nets for industrial inspection Daniel Weimer Defect detection is a classification problem: good/bad classification example x2 Classification problem 1 x1
36. 3. Deep Conv Nets for industrial inspection Daniel Weimer Defect detection is a classification problem: Hyperplane, model ? x1 x2 good/bad classification example Classification problem 1 Supervised training 2 Training Examples (historical data) Modell offline online defect
37. 1. Data representation and augmentation Block extraction Augmentation Shuffle the data Dimensionality of training data: px × py × 1.299.200 3. Deep Conv Nets for industrial inspection Daniel Weimer
38. 4. Experiments, results and limits 2. Network architectures Basis Tiefe-1 Tiefe-2 Tiefe-3 Stufe-1 conv3-NK1 max-pooling conv3-NK1 conv3-NK1 max-Pooling conv3-NK1 conv3-NK1 pad max-Pooling conv3-NK1 conv3-NK1 max-Pooling Stufe-2 conv3-NK2 max-pooling conv3-NK2 conv3-NK2 max-Pooling conv3-NK2 conv3-NK2 conv3-NK2 max-Pooling conv3-NK2 conv3-NK2 conv3-NK2 max-Pooling Stufe-3 conv3-NK3 max-pooling conv3-NK3 max-pooling conv3-NK3 conv3-NK3 max-pooling conv3-NK3 conv3-NK3 conv3-NK3 conv3-NK3 Max-pooling NN-1024 NN-1024 NN-12 Softmax Zusätzliche <Conv> <Pool> <ReLU> Schichten  Tiefe Zusätzliche Neuronen pro Schicht  Breite Number of neurons NK1 NK2 NK3 Basis 6 16 120 Width-1 16 32 256 Width-2 32 64 512 Width-3 64 128 256 Daniel Weimer
39. 4. Experiments, results and limits 2. Network architectures Basis Depth-1 Tiefe-2 Tiefe-3 Stufe-1 conv3-NK1 max-pooling conv3-NK1 conv3-NK1 max-pooling conv3-NK1 conv3-NK1 pad max-Pooling conv3-NK1 conv3-NK1 max-Pooling Stufe-2 conv3-NK2 max-pooling conv3-NK2 conv3-NK2 max-pooling conv3-NK2 conv3-NK2 conv3-NK2 max-Pooling conv3-NK2 conv3-NK2 conv3-NK2 max-Pooling Stufe-3 conv3-NK3 max-pooling conv3-NK3 max-pooling conv3-NK3 conv3-NK3 max-pooling conv3-NK3 conv3-NK3 conv3-NK3 conv3-NK3 Max-pooling NN-1024 NN-1024 NN-12 Softmax Zusätzliche <Conv> <Pool> <ReLU> Schichten  Tiefe Zusätzliche Neuronen pro Schicht  Breite Number of neurons NK1 NK2 NK3 Basis 6 16 120 Width-1 16 32 256 Width-2 32 64 512 Width-3 64 128 256 Daniel Weimer
40. 4. Experiments, results and limits 2. Network architectures Basis Depth-1 Depth-2 Depth-3 Stufe-1 conv3-NK1 max-pooling conv3-NK1 conv3-NK1 max-pooling conv3-NK1 conv3-NK1 pad max-pooling conv3-NK1 conv3-NK1 max-pooling Stufe-2 conv3-NK2 max-pooling conv3-NK2 conv3-NK2 max-pooling conv3-NK2 conv3-NK2 conv3-NK2 max-pooling conv3-NK2 conv3-NK2 conv3-NK2 max-pooling Stufe-3 conv3-NK3 max-pooling conv3-NK3 max-pooling conv3-NK3 conv3-NK3 max-pooling conv3-NK3 conv3-NK3 conv3-NK3 conv3-NK3 max-pooling NN-1024 NN-1024 NN-12 Softmax Additional <Conv> <Pool> <ReLU> layer  Depth Additional neurons per layer  Width Number of neurons NK1 NK2 NK3 Basis 6 16 120 Width-1 16 32 256 Width-2 32 64 512 Width-3 64 128 256 Daniel Weimer
41. 4. Experiments, results and limits Influence of number of layers and neurons per layer Depth Highest precision Number of parameter (Million) Feature dimension Runtime per block Basis 95,68% 6,98 NK3 × 12 × 4 0,83ms Depth-1 96,44% 5,14 NK3 × 11 × 3 0,78ms Depth-2 96,87% 2,32 NK3 × 9 × 1 0,69ms Depth-3 97,03% 3,98 NK3 × 7 × 3 1,01ms Daniel Weimer
42. 4. Experiments, results and limits Influence of number of layers and neurons per layer 1,4% 95,68% 97,03% ~ 1 - Depth Highest precision Number of parameter (Million) Feature dimension Runtime per block Basis 95,68% 6,98 NK3 × 12 × 4 0,83ms Depth-1 96,44% 5,14 NK3 × 11 × 3 0,78ms Depth-2 96,87% 2,32 NK3 × 9 × 1 0,69ms Depth-3 97,03% 3,98 NK3 × 7 × 3 1,01ms Daniel Weimer
43. 4. Experiments, results and limits ~1,2% Influence of number of layers and neurons per layer 1,4% ~ Depth Highest precision Number of parameter (Million) Feature dimension Runtime per block Basis 95,68% 6,98 NK3 × 12 × 4 0,83ms Depth-1 96,44% 5,14 NK3 × 11 × 3 0,78ms Depth-2 96,87% 2,32 NK3 × 9 × 1 0,69ms Depth-3 97,03% 3,98 NK3 × 7 × 3 1,01ms • Runtime is driven by depth of layer • As more layers, as more complex (and compressed) the features Daniel Weimer
44. 4. Experiments, results and limits • Runtime is driven by depth of layer • As more layers, as more complex (and compressed) the features Influence of number of layers and neurons per layer Just a small step in precision, despite additional layers • Limit of ConvNets with respect to depth • Assumption: limit of geometric complexity leads to saturation effects in high layers ~1,2% 1,4% ~ Depth Highest precision Number of parameter (Million) Feature dimension Runtime per block Basis 95,68% 6,98 NK3 × 12 × 4 0,83ms Depth-1 96,44% 5,14 NK3 × 11 × 3 0,78ms Depth-2 96,87% 2,32 NK3 × 9 × 1 0,69ms Depth-3 97,03% 3,98 NK3 × 7 × 3 1,01ms Daniel Weimer
45. 4. Experiments, results and limits Highest precision and time/block Width NK Depth-2 Depth-3 Basis {6,16,120} 96,87% @ 0,69ms 97,03% @ 1,01ms Width-1 {16,32,256} 96,94% @ 0,98ms 97,19% @ 1,60ms Width-2 {32,64,512} 97,13% @ 1,65ms 97,36% @ 3,25ms Width-3 {64,128,256} 97,19% @ 1,42ms 97,37% @ 2,00ms ~3,3% ~5,1% Influence of number of layers and neurons per layer Depth Highest precision Number of parameter (Million) Feature dimension Runtime per block Basis 95,68% 6,98 NK3 × 12 × 4 0,83ms Depth-1 96,44% 5,14 NK3 × 11 × 3 0,78ms Depth-2 96,87% 2,32 NK3 × 9 × 1 0,69ms Depth-3 97,03% 3,98 NK3 × 7 × 3 1,01ms Daniel Weimer
46. 4. Experiments, results and limits Benchmark with DAGM dataset Class Deep Learning Stat. features + SVM SIFT + NN** Weibull features*** 1 100 99,5 98,9 87 2 100 94,3 95,7 - 3 95,5 99,5 98,5 99,8 4 100 92,5 - - 5 98,8 96,9 98,2 97,2 6 100 100 99,8 94,9 1 100 99,7 100 98 2 97,3 80 91,3 - 3 100 100 100 100 4 98,7 96,1 - - 5 100 96,1 100 100 6 99,5 96,2 100 100 TPR TNR ***[Timm u. Barth 2011] **[Siebel u. Sommer 2008] *[Scholz-Reiter et. al 2012] Daniel Weimer
47. 4. Experiments, results and limits Detection results Daniel Weimer
48. 5. Conclusion Conclusion • Deep Learning pushes Machine Learning to the next level: – OLD: Design features on raw data level (pixel, time steps, letters, ….. ) – NEW: Design of architectures that automatically process data (we don‘t touch raw data) • When to choose Deep Learning:  You have enough training data (with or without label)  Massively parallel hardware support (for training)  Your problem can be represented in evolutionary stages  Manual feature representation is expensive or saturated Daniel Weimer
49. Daniel Weimer Thank you very much! feel free to contact me: daniel.weimer@volkswagen.de
50. Daniel Weimer 27 Literatur LeCun, Y. ; Boser, B. ; Denker, J.S. ; Henderson, D. ; Howard, R.E. ; Hubbard, W. ; Jackel, D.L.: Backpropagation applied to hand-written zip code recogition. In: Neural Computation 1 (1989), Nr. 4, S. 541–551 [LeCun et. al, 1989] Xie, X.: A Review of Recent Advances in Surface Defect Detection using Texture analysis Techniques. In: Electronic Letters on Computer Vision and Image Analysis 7 (2008), Nr. 3. – ISSN 1577–5097 [Xie, 2008] Siebel, N.T ; Sommer, G.: Learning Defect Classifiers for Visual Inspection Images by Neuro- Evolution using Weakly Labelled Training Data. In: Evolutionary Computation, 2008, S. 3925 – 3931 [Siebel u. Sommer 2008] Timm, F. ; Barth, E.: Non-parametric texture defect detection using Weibull features. In: IS&T/SPIE Electronic Imaging, International Society for Optics and Photonics, 2011 [Timm u. Barth 2011] Ng, A.: Deep Learning. http://cs229.stanford.edu/materials/CS229-DeepLearning.pdf Lecture at Stanford University [Ng 2015] Karimi, M.H. ; Asemani, D.: Surface defect detection in tiling Industries using digital image processing methods: Analysis and evaluation. In: ISA Transactions 53 (2014), Nr. 3, S. 834– 844. – ISSN 0019–0578 [Karimi u. Asemani 2014] Simonyan, K. ; Zisserman, A.: Very Deep Convolutional Networks for Large-Scale Image Recognition. In: CoRR abs/1409.1556 (2014) [Simonyan u. Zisserman 2014]