Deep Convolutional Neural Networks in industrial applications

Deep Convolutional Neural Networks in industrial applications

Deep Learning is a new (and at the same time old) paradigm in machine learning which allows to extract features directly from huge amounts of raw data with a minimum of human interaction. This talk gives an introduction about deep learning in general and focuses an important application of deep convolutional neural networks (CNN) in industrial settings: optical quality control.

3c3f3f18c25ea5283640ebd23553e7c6?s=128

MunichDataGeeks

October 08, 2016
Tweet

Transcript

  1. Deep Convolutional Neural Networks in industrial applications - Daniel Weimer

    - Datageeks Data Day, 8th October 2016 – Munich
  2. Agenda 1. The machine learning pipeline 2. Deep Learning: A

    new paradigm in AI 3. Deep Conv Nets for industrial inspection 4. Experiments, Results and Limits 5. Conclusion Daniel Weimer
  3. 1. The machine learning pipeline Daniel Weimer Traditional Feature Extractor

    Classifier Inspired by Y. LeCun
  4. 1. The machine learning pipeline Daniel Weimer Traditional HoG (People

    Detection) Viola&Jones (Face recognition) SIFT (Matching) Feature point histogram (3D Matching) Feature Extractor Classifier
  5. 1. The machine learning pipeline Daniel Weimer Feature Extractor Classifier

    Traditional Better Feature Extractor Classifier Mid-level Features • Hand crafted feature representations • Deep domain-specific knowledge for feature description is required (expert driven) • Take a lot of time to develop such features Inspired by Y. LeCun
  6. None
  7. 1. The machine learning pipeline Daniel Weimer Feature Extractor Classifier

    Traditional Better Feature Extractor Classifier Mid-level Features Deep Learning Low-level Features Classifier Mid-level Features High-level Features Training / Learning Inspired by Y. LeCun
  8. 1. The machine learning pipeline Daniel Weimer What might be

    the ultimate result of an ML algorithm giving that image?
  9. 1. The machine learning pipeline Daniel Weimer Person Ball Grass

    Automated feature generation with rising complexity „4 persons playing football in a stadium“
  10. 1. The machine learning pipeline Daniel Weimer Can we learn

    powerful features directly from data? Without the need of human interaction! Deep Machine Learning
  11. 2. Deep Learning: A new paradigm in AI Why do

    we need a new paradigm in AI? Daniel Weimer
  12. 2. Deep Learning: A new paradigm in AI Why do

    we need a new paradigm in AI? Daniel Weimer [Ng 2016]
  13. 2. Deep Learning: A new paradigm in AI A Renaissance

    of neural architectures: The basic motivation of deep learning are traditional neural networks x1 x2 x3 . . . xn y {0,1} +1 h(x) feature vector x neuron weights Daniel Weimer Bias
  14. 2. Deep Learning: A new paradigm in AI Motivation: Raw

    data will be transformed to complex features by stacking together non-linear feature extraction stages x1 x2 x3 . . . xn y {0,1} h(x) ... Daniel Weimer pixel vector x
  15. 2. Deep Learning: A new paradigm in AI Training of

    deep neural networks: Standard back-propagation of errors (backprop) x1 x2 x3 . . . xn y {0,1} h(x) y = h(x) = 1 0.8 0.2 error = ... Daniel Weimer pixel vector x
  16. 2. Deep Learning: A new paradigm in AI Training of

    deep neural networks: Standard back-propagation of errors (backprop) x1 x2 x3 . . . xn y {0,1} ... h(x) y = h(x) = 1 0.8 0.2 error = pixel vector x
  17. 2. Deep Learning: A new paradigm in AI Training of

    deep neural networks: Standard back-propagation of errors (backprop) x1 x2 x3 . . . xn y {0,1} ... h(x) y = h(x) = 1 0.8 0.2 error = pixel vector x Daniel Weimer Error become smaller smaller smaller smaller ...... Vanishing gradient problem
  18. 2. Deep Learning: A new paradigm in AI Training of

    deep neural networks: Method 1: Stacked autoencoder learning (around 2006) ... Daniel Weimer IN IN=OUT
  19. 2. Deep Learning: A new paradigm in AI Training of

    deep neural networks: Method 1: Stacked autoencoder learning (around 2006) ... Daniel Weimer + + fine tuning
  20. 2. Deep Learning: A new paradigm in AI Training of

    deep neural networks: Method 2: Dropout, Regularization and Activation x1 x2 x3 . . . xn y {0,1} ... Daniel Weimer active dropout
  21. 2. Deep Learning: A new paradigm in AI Fully connected

    layers are a huge problem by applying deep nets to images: Daniel Weimer
  22. 2. Deep Learning: A new paradigm in AI Fully connected

    layers are a huge problem by applying deep nets to images: Daniel Weimer
  23. 2. Deep Learning: A new paradigm in AI Fully connected

    layers are a huge problem by applying deep nets to images: Daniel Weimer
  24. 2. Deep Learning: A new paradigm in AI From deep

    neural networks to deep convolutional neural networks (ConvNets) Reducing the number of neuron weights by receptive field approach layer-1 layer-2 FC-ANN [LeCun et.al, 1989] features model Daniel Weimer
  25. 2. Deep Learning: A new paradigm in AI From deep

    neural networks to deep convolutional neural networks (ConvNets) Reducing the number of neuron weights by receptive field approach Activation Convolution Pooling layer-1 layer-2 FC-ANN <Conv> <Pool> <ReLU> [LeCun et.al, 1989] Daniel Weimer
  26. 2. Deep Learning: A new paradigm in AI From deep

    neural networks to deep convolutional neural networks (ConvNets) Reducing the number of neuron weights by receptive field approach Activation Convolution Pooling layer-1 layer-2 FC-ANN • K neurons learn n×n filter per layer • Step width of S pixel per filter • Number of layers: m • Padding in border region of the image Design- Hyperparameter In this work: n×n = 3×3 * *[Simonyan u. Zisserman 2014] Daniel Weimer
  27. 2. Deep Learning: A new paradigm in AI From deep

    neural networks to deep convolutional neural networks (ConvNets) Reducing the number of neuron weights by receptive field approach Activation Convolution Pooling layer-1 layer-2 FC-ANN • Pooling of max(2×2) neighboaring pixel • Step width of 1 pixel  Reducing image information by a factor of 2 in width and height Design- Hyperparameter Daniel Weimer
  28. 2. Deep Learning: A new paradigm in AI From deep

    neural networks to deep convolutional neural networks (ConvNets) Reducing the number of neuron weights by receptive field approach Activation Convolution Pooling layer-1 layer-2 FC-ANN Daniel Weimer
  29. 3. Deep Conv Nets for industrial inspection Daniel Weimer

  30. 3. Deep Conv Nets for industrial inspection Daniel Weimer [DAGM

    dataset 2006]
  31. 3. Deep Conv Nets for industrial inspection Daniel Weimer [DAGM

    dataset 2006]
  32. 3. Deep Conv Nets for industrial inspection Defects on surfaces

    are varying with respect to • Size • Geometry • Orientation • Different background textures Daniel Weimer
  33. 3. Deep Conv Nets for industrial inspection How to apply

    deep learning in industrial inspection? Daniel Weimer
  34. 3. Deep Conv Nets for industrial inspection Daniel Weimer Defect

    detection is a classification problem: good/bad classification example x2 Classification problem 1 x1
  35. 3. Deep Conv Nets for industrial inspection Daniel Weimer Defect

    detection is a classification problem: good/bad classification example x2 Classification problem 1 x1
  36. 3. Deep Conv Nets for industrial inspection Daniel Weimer Defect

    detection is a classification problem: Hyperplane, model ? x1 x2 good/bad classification example Classification problem 1 Supervised training 2 Training Examples (historical data) Modell offline online defect
  37. 1. Data representation and augmentation Block extraction Augmentation Shuffle the

    data Dimensionality of training data: px × py × 1.299.200 3. Deep Conv Nets for industrial inspection Daniel Weimer
  38. 4. Experiments, results and limits 2. Network architectures Basis Tiefe-1

    Tiefe-2 Tiefe-3 Stufe-1 conv3-NK1 max-pooling conv3-NK1 conv3-NK1 max-Pooling conv3-NK1 conv3-NK1 pad max-Pooling conv3-NK1 conv3-NK1 max-Pooling Stufe-2 conv3-NK2 max-pooling conv3-NK2 conv3-NK2 max-Pooling conv3-NK2 conv3-NK2 conv3-NK2 max-Pooling conv3-NK2 conv3-NK2 conv3-NK2 max-Pooling Stufe-3 conv3-NK3 max-pooling conv3-NK3 max-pooling conv3-NK3 conv3-NK3 max-pooling conv3-NK3 conv3-NK3 conv3-NK3 conv3-NK3 Max-pooling NN-1024 NN-1024 NN-12 Softmax Zusätzliche <Conv> <Pool> <ReLU> Schichten  Tiefe Zusätzliche Neuronen pro Schicht  Breite Number of neurons NK1 NK2 NK3 Basis 6 16 120 Width-1 16 32 256 Width-2 32 64 512 Width-3 64 128 256 Daniel Weimer
  39. 4. Experiments, results and limits 2. Network architectures Basis Depth-1

    Tiefe-2 Tiefe-3 Stufe-1 conv3-NK1 max-pooling conv3-NK1 conv3-NK1 max-pooling conv3-NK1 conv3-NK1 pad max-Pooling conv3-NK1 conv3-NK1 max-Pooling Stufe-2 conv3-NK2 max-pooling conv3-NK2 conv3-NK2 max-pooling conv3-NK2 conv3-NK2 conv3-NK2 max-Pooling conv3-NK2 conv3-NK2 conv3-NK2 max-Pooling Stufe-3 conv3-NK3 max-pooling conv3-NK3 max-pooling conv3-NK3 conv3-NK3 max-pooling conv3-NK3 conv3-NK3 conv3-NK3 conv3-NK3 Max-pooling NN-1024 NN-1024 NN-12 Softmax Zusätzliche <Conv> <Pool> <ReLU> Schichten  Tiefe Zusätzliche Neuronen pro Schicht  Breite Number of neurons NK1 NK2 NK3 Basis 6 16 120 Width-1 16 32 256 Width-2 32 64 512 Width-3 64 128 256 Daniel Weimer
  40. 4. Experiments, results and limits 2. Network architectures Basis Depth-1

    Depth-2 Depth-3 Stufe-1 conv3-NK1 max-pooling conv3-NK1 conv3-NK1 max-pooling conv3-NK1 conv3-NK1 pad max-pooling conv3-NK1 conv3-NK1 max-pooling Stufe-2 conv3-NK2 max-pooling conv3-NK2 conv3-NK2 max-pooling conv3-NK2 conv3-NK2 conv3-NK2 max-pooling conv3-NK2 conv3-NK2 conv3-NK2 max-pooling Stufe-3 conv3-NK3 max-pooling conv3-NK3 max-pooling conv3-NK3 conv3-NK3 max-pooling conv3-NK3 conv3-NK3 conv3-NK3 conv3-NK3 max-pooling NN-1024 NN-1024 NN-12 Softmax Additional <Conv> <Pool> <ReLU> layer  Depth Additional neurons per layer  Width Number of neurons NK1 NK2 NK3 Basis 6 16 120 Width-1 16 32 256 Width-2 32 64 512 Width-3 64 128 256 Daniel Weimer
  41. 4. Experiments, results and limits Influence of number of layers

    and neurons per layer Depth Highest precision Number of parameter (Million) Feature dimension Runtime per block Basis 95,68% 6,98 NK3 × 12 × 4 0,83ms Depth-1 96,44% 5,14 NK3 × 11 × 3 0,78ms Depth-2 96,87% 2,32 NK3 × 9 × 1 0,69ms Depth-3 97,03% 3,98 NK3 × 7 × 3 1,01ms Daniel Weimer
  42. 4. Experiments, results and limits Influence of number of layers

    and neurons per layer 1,4% 95,68% 97,03% ~ 1 - Depth Highest precision Number of parameter (Million) Feature dimension Runtime per block Basis 95,68% 6,98 NK3 × 12 × 4 0,83ms Depth-1 96,44% 5,14 NK3 × 11 × 3 0,78ms Depth-2 96,87% 2,32 NK3 × 9 × 1 0,69ms Depth-3 97,03% 3,98 NK3 × 7 × 3 1,01ms Daniel Weimer
  43. 4. Experiments, results and limits ~1,2% Influence of number of

    layers and neurons per layer 1,4% ~ Depth Highest precision Number of parameter (Million) Feature dimension Runtime per block Basis 95,68% 6,98 NK3 × 12 × 4 0,83ms Depth-1 96,44% 5,14 NK3 × 11 × 3 0,78ms Depth-2 96,87% 2,32 NK3 × 9 × 1 0,69ms Depth-3 97,03% 3,98 NK3 × 7 × 3 1,01ms • Runtime is driven by depth of layer • As more layers, as more complex (and compressed) the features Daniel Weimer
  44. 4. Experiments, results and limits • Runtime is driven by

    depth of layer • As more layers, as more complex (and compressed) the features Influence of number of layers and neurons per layer Just a small step in precision, despite additional layers • Limit of ConvNets with respect to depth • Assumption: limit of geometric complexity leads to saturation effects in high layers ~1,2% 1,4% ~ Depth Highest precision Number of parameter (Million) Feature dimension Runtime per block Basis 95,68% 6,98 NK3 × 12 × 4 0,83ms Depth-1 96,44% 5,14 NK3 × 11 × 3 0,78ms Depth-2 96,87% 2,32 NK3 × 9 × 1 0,69ms Depth-3 97,03% 3,98 NK3 × 7 × 3 1,01ms Daniel Weimer
  45. 4. Experiments, results and limits Highest precision and time/block Width

    NK Depth-2 Depth-3 Basis {6,16,120} 96,87% @ 0,69ms 97,03% @ 1,01ms Width-1 {16,32,256} 96,94% @ 0,98ms 97,19% @ 1,60ms Width-2 {32,64,512} 97,13% @ 1,65ms 97,36% @ 3,25ms Width-3 {64,128,256} 97,19% @ 1,42ms 97,37% @ 2,00ms ~3,3% ~5,1% Influence of number of layers and neurons per layer Depth Highest precision Number of parameter (Million) Feature dimension Runtime per block Basis 95,68% 6,98 NK3 × 12 × 4 0,83ms Depth-1 96,44% 5,14 NK3 × 11 × 3 0,78ms Depth-2 96,87% 2,32 NK3 × 9 × 1 0,69ms Depth-3 97,03% 3,98 NK3 × 7 × 3 1,01ms Daniel Weimer
  46. 4. Experiments, results and limits Benchmark with DAGM dataset Class

    Deep Learning Stat. features + SVM SIFT + NN** Weibull features*** 1 100 99,5 98,9 87 2 100 94,3 95,7 - 3 95,5 99,5 98,5 99,8 4 100 92,5 - - 5 98,8 96,9 98,2 97,2 6 100 100 99,8 94,9 1 100 99,7 100 98 2 97,3 80 91,3 - 3 100 100 100 100 4 98,7 96,1 - - 5 100 96,1 100 100 6 99,5 96,2 100 100 TPR TNR ***[Timm u. Barth 2011] **[Siebel u. Sommer 2008] *[Scholz-Reiter et. al 2012] Daniel Weimer
  47. 4. Experiments, results and limits Detection results Daniel Weimer

  48. 5. Conclusion Conclusion • Deep Learning pushes Machine Learning to

    the next level: – OLD: Design features on raw data level (pixel, time steps, letters, ….. ) – NEW: Design of architectures that automatically process data (we don‘t touch raw data) • When to choose Deep Learning:  You have enough training data (with or without label)  Massively parallel hardware support (for training)  Your problem can be represented in evolutionary stages  Manual feature representation is expensive or saturated Daniel Weimer
  49. Daniel Weimer Thank you very much! feel free to contact

    me: daniel.weimer@volkswagen.de
  50. Daniel Weimer 27 Literatur LeCun, Y. ; Boser, B. ;

    Denker, J.S. ; Henderson, D. ; Howard, R.E. ; Hubbard, W. ; Jackel, D.L.: Backpropagation applied to hand-written zip code recogition. In: Neural Computation 1 (1989), Nr. 4, S. 541–551 [LeCun et. al, 1989] Xie, X.: A Review of Recent Advances in Surface Defect Detection using Texture analysis Techniques. In: Electronic Letters on Computer Vision and Image Analysis 7 (2008), Nr. 3. – ISSN 1577–5097 [Xie, 2008] Siebel, N.T ; Sommer, G.: Learning Defect Classifiers for Visual Inspection Images by Neuro- Evolution using Weakly Labelled Training Data. In: Evolutionary Computation, 2008, S. 3925 – 3931 [Siebel u. Sommer 2008] Timm, F. ; Barth, E.: Non-parametric texture defect detection using Weibull features. In: IS&T/SPIE Electronic Imaging, International Society for Optics and Photonics, 2011 [Timm u. Barth 2011] Ng, A.: Deep Learning. http://cs229.stanford.edu/materials/CS229-DeepLearning.pdf Lecture at Stanford University [Ng 2015] Karimi, M.H. ; Asemani, D.: Surface defect detection in tiling Industries using digital image processing methods: Analysis and evaluation. In: ISA Transactions 53 (2014), Nr. 3, S. 834– 844. – ISSN 0019–0578 [Karimi u. Asemani 2014] Simonyan, K. ; Zisserman, A.: Very Deep Convolutional Networks for Large-Scale Image Recognition. In: CoRR abs/1409.1556 (2014) [Simonyan u. Zisserman 2014]