Predictive Modeling and Deep Learning

Entretiens du Nouveau Monde Industriel 2015, Paris.

Transcript

1. Predictive Modeling & Deep Learning Olivier Grisel - ENMI - Paris 2015
2. Outline • Predictive Modeling & Artificial Intelligence • Deep Learning • Computer Vision • Natural Language Understanding and Machine Translation • Learning to Reason and Answer Questions
3. Predictive Modeling
4. type (category) # rooms (int) surface (float m2) public trans (boolean) Apartment 3 50 TRUE House 5 254 FALSE Duplex 4 68 TRUE Apartment 2 32 TRUE sold (float k€) 450 430 712 234 features target samples (train)
5. type (category) # rooms (int) surface (float m2) public trans (boolean) Apartment 3 50 TRUE House 5 254 FALSE Duplex 4 68 TRUE Apartment 2 32 TRUE sold (float k€) 450 430 712 234 features target samples (train) Apartment 2 33 TRUE House 4 210 TRUE samples (test) ? ?
6. Training text docs images sounds transactions Labels Machine Learning Algorithm Model Predictive Modeling Data Flow Feature vectors
7. New text doc image sound transaction Model Expected Label Predictive Modeling Data Flow Feature vector Training text docs images sounds transactions Labels Machine Learning Algorithm Feature vectors
8. Inventory forecasting & trends detection Predictive modeling examples Personalized radios Fraud detection Virality and readers engagement Predictive maintenance Personality matching
9. Artificial Intelligence Predictive Modeling (Data Analytics)
10. Artificial Intelligence Predictive Modeling (Data Analytics) Self-driving cars IBM Watson Movie recommendations Predictive Maintenance
11. Artificial Intelligence Hand-crafted symbolic reasoning systems Predictive Modeling (Data Analytics)
12. Artificial Intelligence Hand-crafted symbolic reasoning systems Machine Learning Predictive Modeling (Data Analytics)
13. Artificial Intelligence Hand-crafted symbolic reasoning systems Machine Learning Deep Learning Predictive Modeling (Data Analytics)
14. Artificial Intelligence Hand-crafted symbolic reasoning systems Machine Learning Deep Learning Predictive Modeling (Data Analytics)
15. Deep Learning • Neural Networks from the 90’s rebranded in 2006+ • « Neuron » is a loose inspiration (not important) • Stacked architecture of modules that compute internal abstract representations from the data • Parameters are tuned from labeled examples
16. Deep Learning for Computer Vision
17. Deep Learning in the 90’s • Yann LeCun invented Convolutional Networks • First NN successfully trained with many layers
18. Early success at OCR
19. Natural image classification until 2012 Feature Extractions Classification Data independent Supervised Learning dog
20. Natural image classification until 2012 Feature Extractions Classification Data independent Supervised Learning dog cat
21. Natural image classification until 2012 Feature Extractions Classification Data independent Supervised Learning cat
22. NN Layer Supervised Learning dog Supervised Learning Supervised Learning NN Layer NN Layer Image classification today
23. Image classification today NN Layer Supervised Learning Supervised Learning Supervised Learning NN Layer NN Layer dog cat
24. Image classification today NN Layer Supervised Learning Supervised Learning Supervised Learning NN Layer NN Layer dog cat
25. Image classification today NN Layer Supervised Learning Supervised Learning Supervised Learning NN Layer NN Layer dog cat
26. ImageNet Challenge 2012 • 1.2M images labeled with 1000 object categories • AlexNet from the deep learning team of U. of Toronto wins with 15% error rate vs 26% for the second (traditional CV pipeline)
28. ImageNet Challenge 2013 • Clarifai ConvNet model wins at 11% error rate • Many other participants used ConvNets
30. ImageNet Challenge 2014 • Monster model: GoogLeNet at 6.7% error rate
31. GoogLeNet vs Andrej • Andrej Karpathy evaluated human performance (himself): ~5% error rate • "It is clear that humans will soon only be able to outperform state of the art image classification models by use of significant effort, expertise, and time.” source: What I learned from competing against a ConvNet on ImageNet
32. ImageNet Challenge 2015 • Microsoft Research Asia wins with networks with depths ranging from 34 to 152 layers • New record: 3.6% error rate
33. Recurrent Neural Networks
34. source: The Unreasonable Effectiveness of RNNs
35. Applications of RNNs • Natural Language Processing
 (e.g. Language Modeling, Sentiment Analysis) • Machine Translation
 (e.g. English to French) • Speech recognition: audio to text • Speech synthesis: text to audio • Biological sequence modeling (DNA, Proteins)
36. Language modeling source: The Unreasonable Effectiveness of RNNs
37. Shakespeare source: The Unreasonable Effectiveness of RNNs
38. Linux source code
39. Attentional architectures for Machine Translation
40. Neural MT source: From language modeling to machine translation
41. Attentional Neural MT source: From language modeling to machine translation
42. Attention == Alignment source: Neural MT by Jointly Learning to Align and Translate
43. source: Show, Attend and Tell
44. Learning to answer questions
45. Paraphrases from web news
46. source: Teaching Machines to Read and Comprehend
47. source: Teaching Machines to Read and Comprehend
48. Conclusion • ML and DL progress is fast paced • Many applications already in production (e.g. speech, image indexing, translation, face recognition) • Very promising results for QA and robot control • Machine Learning is now moving from pattern recognition to higher level reasoning
50. Thank you! http://twitter.com/ogrisel http://speakerdeck.com/ogrisel TIP: download the PDF version of the slides to click on the source links