Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Big Data, Predictive Modeling and tools

Big Data, Predictive Modeling and tools

CCIP 2015

Olivier Grisel

October 08, 2015
Tweet

More Decks by Olivier Grisel

Other Decks in Technology

Transcript

  1. Outline • What Big Data actually is or isn’t •

    Predictive modeling concepts & tools • Big Data architecture & tools
  2. Not Big Data • Data that fits on a spreadsheet

    • Data that can be analyzed in RAM (< 10 GB) • Data operations that can be performed quickly by a traditional database, e.g. single node PostgreSQL server
  3. Canonical Big Data problem: indexing the Web • For each

    word, aggregate the list of page URLs • 2 billion HTML pages: 100TB >10 days just to read sequentially
  4. Other Big Data examples • GSM location event log from

    telco • Transaction log of a big retail network • Activity records for a service with 10s of millions of users
  5. Type # rooms Surface m2 Floor Public Transports Apartment 3

    65 2 Yes House 5 110 NA No Duplex 4 95 4 Yes features samples Sold 300k 1.5M 2.2M target
  6. Type # rooms Surface m2 Floor Public Transports Apartment 3

    65 2 Yes House 5 110 NA No Duplex 4 95 4 Yes features samples Sold 300k 1.5M 2.2M target Apartment 2 35 3 Yes ?
  7. Predictive Modeling • Automated predictions of outcome on new data

    • Alternative to hard-coded rules written by experts • Extract the structure of historical data • Statistical tools to summarize the training data into an executable predictive model
  8. Training text docs images sounds transactions Labels Machine Learning Algorithm

    Model Predictive Modeling Data Flow Feature vectors
  9. New text doc image sound transaction Model Expected Label Predictive

    Modeling Data Flow Feature vector Training text docs images sounds transactions Labels Machine Learning Algorithm Feature vectors
  10. Predictive Models • Ex: Movie recommendations or targeted ads •

    Embedded in a user service to make it more useful / attractive / profitable • A wrong individual decision is not costly • Large number of small automated decisions • Humans would not be fast enough to make the predictions
  11. New text doc image sound transaction Model Expected Label Small

    data Training text docs images sounds transactions Labels Machine Learning Algorithm Feature vectors Feature vector
  12. New text doc image sound transaction Model Expected Label Small

    / Medium data Training text docs images sounds transactions Labels Machine Learning Algorithm Feature vectors Feature vector
  13. New text doc image sound transaction Model Expected Label Small

    / Medium data with Training text docs images sounds transactions Labels Machine Learning Algorithm Feature vectors Feature vector
  14. New text doc image sound transaction Model Expected Label Small

    / Medium data with Training text docs images sounds transactions Labels Machine Learning Algorithm Feature vectors Feature vector
  15. Model Expected Label Big data with Machine Learning Algorithm New

    text doc image sound transaction Training text docs images sounds transactions Labels Feature vectors Feature vector
  16. Model Expected Label Big data with Machine Learning Algorithm New

    text doc image sound transaction Training text docs images sounds transactions Labels Feature vectors Feature vector
  17. Model Expected Label Big data with Machine Learning Algorithm New

    text doc image sound transaction Training text docs images sounds transactions Labels Feature vectors Feature vector
  18. Model Expected Label Big data with Machine Learning Algorithm New

    text doc image sound transaction Training text docs images sounds transactions Labels Feature vectors Feature vector
  19. Distributed Event Log & Queues Distributed Event Stream Processing Distributed

    Storage Distributed Batch Processing On-line Transaction Processing & App Views Analytical Database Predictive Models
  20. Back to the Regionator What if we did not have

    census data on daily mobility?
  21. Back to the Regionator • Use raw daily telco logs

    • Group By (phone, day) to extract daily trips • Join By GPS coordinates to “departement” names • Filter out small trips • Group By (home, work) “departements” • Count
  22. From Big to Small • Feature extraction often shrinks data

    • Filter / Join / Group By / Count • Machine Learning performed on aggregates • Sampling for fast in-memory iterative modeling
  23. Type # rooms Surface (m2) Floor Public Transp. Apart. 3

    65 2 Yes House 5 110 NA No Duplex 4 95 4 Yes features samples
  24. Type # rooms Surface (m2) Floor Public Transp. School (km)

    Flood plain Apart. 3 65 2 Yes 1.0 No House 5 110 NA No 25.0 Yes Duplex 4 95 4 Yes 0.5 No features samples
  25. • Parametric e.g. linear model (traditional stats) vs Non-parametric e.g.

    Random Forests, Neural Networks (Machine Learning) • Understand a model with 10% accuracy vs blindly trust a model with 90% accuracy • Simple models e.g. F = m a, F = - G (m1 + m2) / r^2 will not become false(r) because of big data • New problems can be tackled: computer vision, speech recognition, natural language understanding
  26. • the (experimental) scientific method introduced by Karl Popper is

    based on the falsifiability of formulated hypotheses • theory is correct as long as past predictions hold in new experiments • machine learning train-validation-test splits and cross-validation is similar in spirit • ml model is just a complex theory: correct as long as its predictions still hold