Upgrade to Pro — share decks privately, control downloads, hide ads and more …

An Intro to Deep Learning

An Intro to Deep Learning

Presentation give at Neurospin (NeuroBreakfast)

Olivier Grisel

July 26, 2017
Tweet

More Decks by Olivier Grisel

Other Decks in Technology

Transcript

  1. An Intro to
    Deep Learning
    Olivier Grisel - Neurospin 2017

    View Slide

  2. Outline
    • ML, DL & Artificial Intelligence
    • Deep Learning
    • Computer Vision
    • Natural Language Understanding and Machine
    Translation
    • Other possible applications

    View Slide

  3. Machine Learning,

    Deep Learning and
    Artificial Intelligence

    View Slide

  4. Artificial Intelligence
    Predictive Modeling
    (Data Analytics)

    View Slide

  5. Artificial Intelligence
    Predictive Modeling
    (Data Analytics)
    Self-driving cars
    IBM Watson
    Movie
    recommendations
    Predictive
    Maintenance

    View Slide

  6. Artificial Intelligence
    Hand-crafted
    symbolic
    reasoning
    systems
    Predictive Modeling
    (Data Analytics)

    View Slide

  7. Artificial Intelligence
    Hand-crafted
    symbolic
    reasoning
    systems
    Machine Learning
    Predictive Modeling
    (Data Analytics)

    View Slide

  8. Artificial Intelligence
    Hand-crafted
    symbolic
    reasoning
    systems
    Machine Learning
    Deep
    Learning
    Predictive Modeling
    (Data Analytics)

    View Slide

  9. Artificial Intelligence
    Hand-crafted
    symbolic
    reasoning
    systems
    Machine Learning
    Deep
    Learning
    Predictive Modeling
    (Data Analytics)

    View Slide

  10. Deep Learning
    • Neural Networks from the 90’s rebranded in 2006+
    • « Neuron » is a loose inspiration (not important)
    • Stacked architecture of modules that compute
    internal abstract representations from the data
    • Parameters are tuned from labeled examples

    View Slide

  11. Deep Learning in the 90’s
    sources: LeNet5 & Stanford Deep Learning Tutorial

    View Slide

  12. x = Input Vector
    h1 = Hidden Activations
    h2 = Hidden Activations
    f1(x, w1) = max(dot(x, w1), 0)
    y = Output Vector
    f3(h2, w3) = softmax(dot(h2, w3))
    f2(h1, w2) = max(dot(h1, w2), 0)
    w1
    w2
    f1
    f2
    f3 w3

    View Slide

  13. • All modules are differentiables
    • w.r.t. module inputs
    • w.r.t. module parameters
    • Training by (Stochastic) Gradient Descent
    • Chain rule: backpropagation algorithm
    • Tune parameters to minimize classification loss

    View Slide

  14. Recent success
    • 2009: state of the art acoustic model for speech
    recognition
    • 2011: state of the art road sign classification
    • 2012: state of the art object classification
    • 2013/14: end-to-end speech recognition, object
    detection
    • 2014/15: state of the art machine translation, getting
    closer for Natural Language Understanding in general

    View Slide

  15. Why now?
    • More labeled data
    • More compute power (optimized BLAS and GPUs)
    • Improvements to algorithms

    View Slide

  16. source: Alec Radford on RNNs

    View Slide

  17. Deep Learning for
    Computer Vision

    View Slide

  18. Deep Learning in the 90’s
    • Yann LeCun invented Convolutional Networks
    • First NN successfully trained with many layers

    View Slide

  19. Early success at OCR

    View Slide

  20. Natural image classification
    until 2012
    Feature
    Extractions
    Classification
    Data
    independent
    Supervised
    Learning
    dog

    View Slide

  21. Natural image classification
    until 2012
    Feature
    Extractions
    Classification
    Data
    independent
    Supervised
    Learning
    dog
    cat

    View Slide

  22. Natural image classification
    until 2012
    Feature
    Extractions
    Classification
    Data
    independent
    Supervised
    Learning
    cat

    View Slide

  23. NN
    Layer
    Supervised
    Learning
    dog
    Supervised
    Learning
    Supervised
    Learning
    NN
    Layer
    NN
    Layer
    Image classification today

    View Slide

  24. Image classification today
    NN
    Layer
    Supervised
    Learning
    Supervised
    Learning
    Supervised
    Learning
    NN
    Layer
    NN
    Layer
    dog
    cat

    View Slide

  25. Image classification today
    NN
    Layer
    Supervised
    Learning
    Supervised
    Learning
    Supervised
    Learning
    NN
    Layer
    NN
    Layer
    dog
    cat

    View Slide

  26. Image classification today
    NN
    Layer
    Supervised
    Learning
    Supervised
    Learning
    Supervised
    Learning
    NN
    Layer
    NN
    Layer
    dog
    cat

    View Slide

  27. ImageNet Challenge 2012
    • 1.2M images labeled with 1000 object categories
    • AlexNet from the deep learning team of U. of
    Toronto wins with 15% error rate vs 26% for the
    second (traditional CV pipeline)

    View Slide

  28. View Slide

  29. ImageNet Challenge 2013
    • Clarifai ConvNet model wins at 11% error rate
    • Many other participants used ConvNets

    View Slide

  30. View Slide

  31. ImageNet Challenge 2014
    • Monster model: GoogLeNet at
    6.7% error rate

    View Slide

  32. GoogLeNet vs Andrej
    • Andrej Karpathy evaluated human performance
    (himself): ~5% error rate
    • "It is clear that humans will soon only be able to
    outperform state of the art image classification
    models by use of significant effort, expertise, and
    time.”
    source: What I learned from competing against a ConvNet on ImageNet

    View Slide

  33. ImageNet Challenge 2015
    • Microsoft Research Asia wins
    with networks with depths
    ranging from 34 to 152 layers
    • New record: 3.6% error rate

    View Slide

  34. source: https://www.eff.org/files/AI-progress-metrics.html

    View Slide

  35. source: https://github.com/facebookresearch/deepmask

    View Slide

  36. source: https://github.com/Cadene/vqa.pytorch

    View Slide

  37. source: https://github.com/Cadene/vqa.pytorch

    View Slide

  38. Recurrent
    Neural Networks

    View Slide

  39. source: The Unreasonable Effectiveness of RNNs

    View Slide

  40. Applications of RNNs
    • Natural Language Processing

    (e.g. Language Modeling, Sentiment Analysis)
    • Machine Translation

    (e.g. English to French)
    • Speech recognition: audio to text
    • Speech synthesis: text to audio
    • Biological sequence modeling (DNA, Proteins)

    View Slide

  41. Language modeling
    source: The Unreasonable Effectiveness of RNNs

    View Slide

  42. Shakespeare
    source: The Unreasonable Effectiveness of RNNs

    View Slide

  43. Linux source code

    View Slide

  44. Attentional architectures
    for Machine Translation

    View Slide

  45. Neural MT
    source: From language modeling to machine translation

    View Slide

  46. Attentional Neural MT
    source: From language modeling to machine translation

    View Slide

  47. source: Google's Neural Machine Translation System: Bridging
    the Gap between Human and Machine Translation

    View Slide

  48. Attention == Alignment
    source: Neural MT by Jointly Learning to Align and Translate

    View Slide

  49. source: Show, Attend and Tell

    View Slide

  50. Learning to answer
    questions

    View Slide

  51. Paraphrases from web news

    View Slide

  52. source: Teaching Machines to Read and Comprehend

    View Slide

  53. source: Teaching Machines to Read and Comprehend

    View Slide

  54. View Slide

  55. Medical Imaging

    View Slide

  56. View Slide

  57. View Slide

  58. Challenges for
    NeuroImaging
    • DL need many labeled images
    • Few subjects per studies (costly)
    • Poor labels: low inter-agreement (e.g. autism)
    • fMRI: low SNR of input data it-self
    • 3D data: huge GPU memory requirements

    View Slide

  59. Conclusion
    • ML and DL progress is fast paced
    • Many applications already in production (e.g.
    speech, image indexing, translation, face
    recognition)
    • Machine Learning is now moving from pattern
    recognition to higher level reasoning
    • Lack of high quality labeled data still a limitation for
    some applications

    View Slide

  60. Thank you!
    http://twitter.com/ogrisel
    http://speakerdeck.com/ogrisel
    Online DL class: http://www.fast.ai/
    Keras examples: https://keras.io/
    DL Book: http://www.deeplearningbook.org/
    UPS DL class: https://github.com/m2dsupsdlclass/lectures-labs

    View Slide