Upgrade to Pro — share decks privately, control downloads, hide ads and more …

PyData SF: Large Scale CTR Prediction - Lessons Learned

HaFl
August 13, 2016

PyData SF: Large Scale CTR Prediction - Lessons Learned

Starting with a basic setup for click-through rate (CTR) prediction, we will step by step improve on it by incorporating the lessons we've learned from operating and scaling such a mission-critical system. The presented lessons will be related to infrastructure, model comprehension, and specifics like how to deal with thresholds. They should be applicable to most ML models used in production.

Link to video: https://youtu.be/zbAuoFUdjAI

HaFl

August 13, 2016
Tweet

More Decks by HaFl

Other Decks in Technology

Transcript

  1. CTR Prediction CTR: Click-Through Rate pCTR: predicted CTR Question How

    likely is the user to click on the ad? Why Proxy for relevance 5.5% 0.8% 9.2% ?
  2. Logistic Regression with thousands of features, trained and tested on

    millions of samples. Current pCTR Model Kuvasz
  3. pCTR Model History (CC) from Flickr: "Wednesday Freedom 11" by

    Parker Knight (CC) from Flickr: "Icelandig sheepdog" by Thomas Quine (CC) from Flickr: by Craige Moore French Brittany Icelandic Sheepdog Jindo Kuvasz
  4. user feedback service logs Log at source of online prediction

    → Prevents downstream modifications of data Logging
  5. data model logs prediction verification fast scalable Make offline training

    iterations fast & scalable Automation is key → end-to-end pipeline → automated visualizations Tools: mrjob, Spark Iterations
  6. Offline Training at Yelp merge logs sampling feature extraction model

    training evaluation mrjob AWS EMR daily scheduled pipeline kicked off manually mrjob AWS EMR Spark mrjob AWS EMR mrjob AWS EMR mrjob AWS EMR new features (CC) from Flickr: "Cloud" by Jason Pratt
  7. Lessons Learned Infrastructure Log at source of online prediction Verify

    predictions Make offline iterations fast & scalable
  8. Focus on a single metric (but don't trust it blindly)

    Evaluation data model prediction verification evaluation fast scalable
  9. Focus on a single metric (but don't trust it blindly)

    Create helpful visualizations Tools: Zeppelin Evaluation data model prediction verification evaluation fast scalable
  10. Visualizations ... feature 1 feature 2 feature 3 ... feature

    contribution Feature contributions sd(feature) * coef Feature value vs. CTR count feature value CTR
  11. logs Beware of biased training data → offline != online

    → pCTR threshold Thresholds user feedback service
  12. pCTR Threshold time training data Model 1 Model 2 Model

    3 Model 4 Idea: Frequent retraining Better: Deliberate sampling of bad ads CTR pCTR
  13. Lessons Learned Infrastructure Log at source of online prediction Verify

    predictions Make offline iterations fast & scalable Model Comprehension Evaluate, evaluate, evaluate Be aware of threshold effects
  14. user feedback service online offline data model logs prediction verification

    evaluation fast scalable well documented fast scalable well documented simplicity
  15. user feedback service online offline data model logs prediction verification

    evaluation fast scalable well documented fast scalable well documented simplicity
  16. Lessons Learned Above all, keep it simple. Infrastructure Log at

    source of online prediction Verify predictions Make offline iterations fast & scalable Model Comprehension Evaluate, evaluate, evaluate Be aware of threshold effects