An Intro to Deep Learning

Presentation give at Neurospin (NeuroBreakfast)

Transcript

1. An Intro to
   Deep Learning
   Olivier Grisel - Neurospin 2017
   

   View Slide

2. Outline
   • ML, DL & Artificial Intelligence
   • Deep Learning
   • Computer Vision
   • Natural Language Understanding and Machine
   Translation
   • Other possible applications
   

   View Slide

3. Machine Learning,

   Deep Learning and
   Artificial Intelligence
   

   View Slide

4. Artificial Intelligence
   Predictive Modeling
   (Data Analytics)
   

   View Slide

5. Artificial Intelligence
   Predictive Modeling
   (Data Analytics)
   Self-driving cars
   IBM Watson
   Movie
   recommendations
   Predictive
   Maintenance
   

   View Slide

6. Artificial Intelligence
   Hand-crafted
   symbolic
   reasoning
   systems
   Predictive Modeling
   (Data Analytics)
   

   View Slide

7. Artificial Intelligence
   Hand-crafted
   symbolic
   reasoning
   systems
   Machine Learning
   Predictive Modeling
   (Data Analytics)
   

   View Slide

8. Artificial Intelligence
   Hand-crafted
   symbolic
   reasoning
   systems
   Machine Learning
   Deep
   Learning
   Predictive Modeling
   (Data Analytics)
   

   View Slide

9. Artificial Intelligence
   Hand-crafted
   symbolic
   reasoning
   systems
   Machine Learning
   Deep
   Learning
   Predictive Modeling
   (Data Analytics)
   

   View Slide

10. Deep Learning
    • Neural Networks from the 90’s rebranded in 2006+
    • « Neuron » is a loose inspiration (not important)
    • Stacked architecture of modules that compute
    internal abstract representations from the data
    • Parameters are tuned from labeled examples
    

    View Slide

11. Deep Learning in the 90’s
    sources: LeNet5 & Stanford Deep Learning Tutorial
    

    View Slide

12. x = Input Vector
    h1 = Hidden Activations
    h2 = Hidden Activations
    f1(x, w1) = max(dot(x, w1), 0)
    y = Output Vector
    f3(h2, w3) = softmax(dot(h2, w3))
    f2(h1, w2) = max(dot(h1, w2), 0)
    w1
    w2
    f1
    f2
    f3 w3
    

    View Slide

13. • All modules are differentiables
    • w.r.t. module inputs
    • w.r.t. module parameters
    • Training by (Stochastic) Gradient Descent
    • Chain rule: backpropagation algorithm
    • Tune parameters to minimize classification loss
    

    View Slide

14. Recent success
    • 2009: state of the art acoustic model for speech
    recognition
    • 2011: state of the art road sign classification
    • 2012: state of the art object classification
    • 2013/14: end-to-end speech recognition, object
    detection
    • 2014/15: state of the art machine translation, getting
    closer for Natural Language Understanding in general
    

    View Slide

15. Why now?
    • More labeled data
    • More compute power (optimized BLAS and GPUs)
    • Improvements to algorithms
    

    View Slide

16. source: Alec Radford on RNNs
    

    View Slide

17. Deep Learning for
    Computer Vision
    

    View Slide

18. Deep Learning in the 90’s
    • Yann LeCun invented Convolutional Networks
    • First NN successfully trained with many layers
    

    View Slide

19. Early success at OCR
    

    View Slide

20. Natural image classification
    until 2012
    Feature
    Extractions
    Classification
    Data
    independent
    Supervised
    Learning
    dog
    

    View Slide

21. Natural image classification
    until 2012
    Feature
    Extractions
    Classification
    Data
    independent
    Supervised
    Learning
    dog
    cat
    

    View Slide

22. Natural image classification
    until 2012
    Feature
    Extractions
    Classification
    Data
    independent
    Supervised
    Learning
    cat
    

    View Slide

23. NN
    Layer
    Supervised
    Learning
    dog
    Supervised
    Learning
    Supervised
    Learning
    NN
    Layer
    NN
    Layer
    Image classification today
    

    View Slide

24. Image classification today
    NN
    Layer
    Supervised
    Learning
    Supervised
    Learning
    Supervised
    Learning
    NN
    Layer
    NN
    Layer
    dog
    cat
    

    View Slide

25. Image classification today
    NN
    Layer
    Supervised
    Learning
    Supervised
    Learning
    Supervised
    Learning
    NN
    Layer
    NN
    Layer
    dog
    cat
    

    View Slide

26. Image classification today
    NN
    Layer
    Supervised
    Learning
    Supervised
    Learning
    Supervised
    Learning
    NN
    Layer
    NN
    Layer
    dog
    cat
    

    View Slide

27. ImageNet Challenge 2012
    • 1.2M images labeled with 1000 object categories
    • AlexNet from the deep learning team of U. of
    Toronto wins with 15% error rate vs 26% for the
    second (traditional CV pipeline)
    

    View Slide

28. View Slide

29. ImageNet Challenge 2013
    • Clarifai ConvNet model wins at 11% error rate
    • Many other participants used ConvNets
    

    View Slide

30. View Slide

31. ImageNet Challenge 2014
    • Monster model: GoogLeNet at
    6.7% error rate
    

    View Slide

32. GoogLeNet vs Andrej
    • Andrej Karpathy evaluated human performance
    (himself): ~5% error rate
    • "It is clear that humans will soon only be able to
    outperform state of the art image classification
    models by use of significant effort, expertise, and
    time.”
    source: What I learned from competing against a ConvNet on ImageNet
    

    View Slide

33. ImageNet Challenge 2015
    • Microsoft Research Asia wins
    with networks with depths
    ranging from 34 to 152 layers
    • New record: 3.6% error rate
    

    View Slide

34. source: https://www.eff.org/files/AI-progress-metrics.html
    

    View Slide

35. source: https://github.com/facebookresearch/deepmask
    

    View Slide

36. source: https://github.com/Cadene/vqa.pytorch
    

    View Slide

37. source: https://github.com/Cadene/vqa.pytorch
    

    View Slide

38. Recurrent
    Neural Networks
    

    View Slide

39. source: The Unreasonable Effectiveness of RNNs
    

    View Slide

40. Applications of RNNs
    • Natural Language Processing

    (e.g. Language Modeling, Sentiment Analysis)
    • Machine Translation

    (e.g. English to French)
    • Speech recognition: audio to text
    • Speech synthesis: text to audio
    • Biological sequence modeling (DNA, Proteins)
    

    View Slide

41. Language modeling
    source: The Unreasonable Effectiveness of RNNs
    

    View Slide

42. Shakespeare
    source: The Unreasonable Effectiveness of RNNs
    

    View Slide

43. Linux source code
    

    View Slide

44. Attentional architectures
    for Machine Translation
    

    View Slide

45. Neural MT
    source: From language modeling to machine translation
    

    View Slide

46. Attentional Neural MT
    source: From language modeling to machine translation
    

    View Slide

47. source: Google's Neural Machine Translation System: Bridging
    the Gap between Human and Machine Translation
    

    View Slide

48. Attention == Alignment
    source: Neural MT by Jointly Learning to Align and Translate
    

    View Slide

49. source: Show, Attend and Tell
    

    View Slide

50. Learning to answer
    questions
    

    View Slide

51. Paraphrases from web news
    

    View Slide

52. source: Teaching Machines to Read and Comprehend
    

    View Slide

53. source: Teaching Machines to Read and Comprehend
    

    View Slide

54. View Slide

55. Medical Imaging
    

    View Slide

56. View Slide

57. View Slide

58. Challenges for
    NeuroImaging
    • DL need many labeled images
    • Few subjects per studies (costly)
    • Poor labels: low inter-agreement (e.g. autism)
    • fMRI: low SNR of input data it-self
    • 3D data: huge GPU memory requirements
    

    View Slide

59. Conclusion
    • ML and DL progress is fast paced
    • Many applications already in production (e.g.
    speech, image indexing, translation, face
    recognition)
    • Machine Learning is now moving from pattern
    recognition to higher level reasoning
    • Lack of high quality labeled data still a limitation for
    some applications
    

    View Slide

60. Thank you!
    http://twitter.com/ogrisel
    http://speakerdeck.com/ogrisel
    Online DL class: http://www.fast.ai/
    Keras examples: https://keras.io/
    DL Book: http://www.deeplearningbook.org/
    UPS DL class: https://github.com/m2dsupsdlclass/lectures-labs
    

    View Slide