Predictive Modeling and Deep Learning

Predictive Modeling and Deep Learning

Entretiens du Nouveau Monde Industriel 2015, Paris.

Aee56554ec30edfd680e1c937ed4e54d?s=128

Olivier Grisel

December 15, 2015
Tweet

Transcript

  1. Predictive Modeling & Deep Learning Olivier Grisel - ENMI -

    Paris 2015
  2. Outline • Predictive Modeling & Artificial Intelligence • Deep Learning

    • Computer Vision • Natural Language Understanding and Machine Translation • Learning to Reason and Answer Questions
  3. Predictive Modeling

  4. type (category) # rooms (int) surface (float m2) public trans

    (boolean) Apartment 3 50 TRUE House 5 254 FALSE Duplex 4 68 TRUE Apartment 2 32 TRUE sold (float k€) 450 430 712 234 features target samples (train)
  5. type (category) # rooms (int) surface (float m2) public trans

    (boolean) Apartment 3 50 TRUE House 5 254 FALSE Duplex 4 68 TRUE Apartment 2 32 TRUE sold (float k€) 450 430 712 234 features target samples (train) Apartment 2 33 TRUE House 4 210 TRUE samples (test) ? ?
  6. Training text docs images sounds transactions Labels Machine Learning Algorithm

    Model Predictive Modeling Data Flow Feature vectors
  7. New text doc image sound transaction Model Expected Label Predictive

    Modeling Data Flow Feature vector Training text docs images sounds transactions Labels Machine Learning Algorithm Feature vectors
  8. Inventory forecasting & trends detection Predictive modeling examples Personalized radios

    Fraud detection Virality and readers engagement Predictive maintenance Personality matching
  9. Artificial Intelligence Predictive Modeling (Data Analytics)

  10. Artificial Intelligence Predictive Modeling (Data Analytics) Self-driving cars IBM Watson

    Movie recommendations Predictive Maintenance
  11. Artificial Intelligence Hand-crafted symbolic reasoning systems Predictive Modeling (Data Analytics)

  12. Artificial Intelligence Hand-crafted symbolic reasoning systems Machine Learning Predictive Modeling

    (Data Analytics)
  13. Artificial Intelligence Hand-crafted symbolic reasoning systems Machine Learning Deep Learning

    Predictive Modeling (Data Analytics)
  14. Artificial Intelligence Hand-crafted symbolic reasoning systems Machine Learning Deep Learning

    Predictive Modeling (Data Analytics)
  15. Deep Learning • Neural Networks from the 90’s rebranded in

    2006+ • « Neuron » is a loose inspiration (not important) • Stacked architecture of modules that compute internal abstract representations from the data • Parameters are tuned from labeled examples
  16. Deep Learning for Computer Vision

  17. Deep Learning in the 90’s • Yann LeCun invented Convolutional

    Networks • First NN successfully trained with many layers
  18. Early success at OCR

  19. Natural image classification until 2012 Feature Extractions Classification Data independent

    Supervised Learning dog
  20. Natural image classification until 2012 Feature Extractions Classification Data independent

    Supervised Learning dog cat
  21. Natural image classification until 2012 Feature Extractions Classification Data independent

    Supervised Learning cat
  22. NN Layer Supervised Learning dog Supervised Learning Supervised Learning NN

    Layer NN Layer Image classification today
  23. Image classification today NN Layer Supervised Learning Supervised Learning Supervised

    Learning NN Layer NN Layer dog cat
  24. Image classification today NN Layer Supervised Learning Supervised Learning Supervised

    Learning NN Layer NN Layer dog cat
  25. Image classification today NN Layer Supervised Learning Supervised Learning Supervised

    Learning NN Layer NN Layer dog cat
  26. ImageNet Challenge 2012 • 1.2M images labeled with 1000 object

    categories • AlexNet from the deep learning team of U. of Toronto wins with 15% error rate vs 26% for the second (traditional CV pipeline)
  27. None
  28. ImageNet Challenge 2013 • Clarifai ConvNet model wins at 11%

    error rate • Many other participants used ConvNets
  29. None
  30. ImageNet Challenge 2014 • Monster model: GoogLeNet at 6.7% error

    rate
  31. GoogLeNet vs Andrej • Andrej Karpathy evaluated human performance (himself):

    ~5% error rate • "It is clear that humans will soon only be able to outperform state of the art image classification models by use of significant effort, expertise, and time.” source: What I learned from competing against a ConvNet on ImageNet
  32. ImageNet Challenge 2015 • Microsoft Research Asia wins with networks

    with depths ranging from 34 to 152 layers • New record: 3.6% error rate
  33. Recurrent Neural Networks

  34. source: The Unreasonable Effectiveness of RNNs

  35. Applications of RNNs • Natural Language Processing
 (e.g. Language Modeling,

    Sentiment Analysis) • Machine Translation
 (e.g. English to French) • Speech recognition: audio to text • Speech synthesis: text to audio • Biological sequence modeling (DNA, Proteins)
  36. Language modeling source: The Unreasonable Effectiveness of RNNs

  37. Shakespeare source: The Unreasonable Effectiveness of RNNs

  38. Linux source code

  39. Attentional architectures for Machine Translation

  40. Neural MT source: From language modeling to machine translation

  41. Attentional Neural MT source: From language modeling to machine translation

  42. Attention == Alignment source: Neural MT by Jointly Learning to

    Align and Translate
  43. source: Show, Attend and Tell

  44. Learning to answer questions

  45. Paraphrases from web news

  46. source: Teaching Machines to Read and Comprehend

  47. source: Teaching Machines to Read and Comprehend

  48. Conclusion • ML and DL progress is fast paced •

    Many applications already in production (e.g. speech, image indexing, translation, face recognition) • Very promising results for QA and robot control • Machine Learning is now moving from pattern recognition to higher level reasoning
  49. None
  50. Thank you! http://twitter.com/ogrisel http://speakerdeck.com/ogrisel TIP: download the PDF version of

    the slides to click on the source links