Upgrade to Pro — share decks privately, control downloads, hide ads and more …

An Intro to Deep Learning

An Intro to Deep Learning

Presentation give at Neurospin (NeuroBreakfast)

Olivier Grisel

July 26, 2017

More Decks by Olivier Grisel

Other Decks in Technology


  1. Outline • ML, DL & Artificial Intelligence • Deep Learning

    • Computer Vision • Natural Language Understanding and Machine Translation • Other possible applications
  2. Deep Learning • Neural Networks from the 90’s rebranded in

    2006+ • « Neuron » is a loose inspiration (not important) • Stacked architecture of modules that compute internal abstract representations from the data • Parameters are tuned from labeled examples
  3. x = Input Vector h1 = Hidden Activations h2 =

    Hidden Activations f1(x, w1) = max(dot(x, w1), 0) y = Output Vector f3(h2, w3) = softmax(dot(h2, w3)) f2(h1, w2) = max(dot(h1, w2), 0) w1 w2 f1 f2 f3 w3
  4. • All modules are differentiables • w.r.t. module inputs •

    w.r.t. module parameters • Training by (Stochastic) Gradient Descent • Chain rule: backpropagation algorithm • Tune parameters to minimize classification loss
  5. Recent success • 2009: state of the art acoustic model

    for speech recognition • 2011: state of the art road sign classification • 2012: state of the art object classification • 2013/14: end-to-end speech recognition, object detection • 2014/15: state of the art machine translation, getting closer for Natural Language Understanding in general
  6. Why now? • More labeled data • More compute power

    (optimized BLAS and GPUs) • Improvements to algorithms
  7. Deep Learning in the 90’s • Yann LeCun invented Convolutional

    Networks • First NN successfully trained with many layers
  8. ImageNet Challenge 2012 • 1.2M images labeled with 1000 object

    categories • AlexNet from the deep learning team of U. of Toronto wins with 15% error rate vs 26% for the second (traditional CV pipeline)
  9. ImageNet Challenge 2013 • Clarifai ConvNet model wins at 11%

    error rate • Many other participants used ConvNets
  10. GoogLeNet vs Andrej • Andrej Karpathy evaluated human performance (himself):

    ~5% error rate • "It is clear that humans will soon only be able to outperform state of the art image classification models by use of significant effort, expertise, and time.” source: What I learned from competing against a ConvNet on ImageNet
  11. ImageNet Challenge 2015 • Microsoft Research Asia wins with networks

    with depths ranging from 34 to 152 layers • New record: 3.6% error rate
  12. Applications of RNNs • Natural Language Processing
 (e.g. Language Modeling,

    Sentiment Analysis) • Machine Translation
 (e.g. English to French) • Speech recognition: audio to text • Speech synthesis: text to audio • Biological sequence modeling (DNA, Proteins)
  13. Challenges for NeuroImaging • DL need many labeled images •

    Few subjects per studies (costly) • Poor labels: low inter-agreement (e.g. autism) • fMRI: low SNR of input data it-self • 3D data: huge GPU memory requirements
  14. Conclusion • ML and DL progress is fast paced •

    Many applications already in production (e.g. speech, image indexing, translation, face recognition) • Machine Learning is now moving from pattern recognition to higher level reasoning • Lack of high quality labeled data still a limitation for some applications
  15. Thank you! http://twitter.com/ogrisel http://speakerdeck.com/ogrisel Online DL class: http://www.fast.ai/ Keras examples:

    https://keras.io/ DL Book: http://www.deeplearningbook.org/ UPS DL class: https://github.com/m2dsupsdlclass/lectures-labs