Pro Yearly is on sale from $80 to $50! »

PyData SF: Large Scale CTR Prediction - Lessons Learned

Efcd304fc29a3883d2aabcf1a07c9390?s=47 HaFl
August 13, 2016

PyData SF: Large Scale CTR Prediction - Lessons Learned

Starting with a basic setup for click-through rate (CTR) prediction, we will step by step improve on it by incorporating the lessons we've learned from operating and scaling such a mission-critical system. The presented lessons will be related to infrastructure, model comprehension, and specifics like how to deal with thresholds. They should be applicable to most ML models used in production.

Link to video: https://youtu.be/zbAuoFUdjAI

Efcd304fc29a3883d2aabcf1a07c9390?s=128

HaFl

August 13, 2016
Tweet

Transcript

  1. Florian Hartl florianh@yelp.com Large Scale CTR Prediction Lessons Learned

  2. Yelp’s Mission Connecting people with great local businesses.

  3. 92M 32 72% 108M Yelp Stats As of Q2 2016

  4. CTR Prediction CTR: Click-Through Rate pCTR: predicted CTR Question How

    likely is the user to click on the ad? Why Proxy for relevance 5.5% 0.8% 9.2% ?
  5. Logistic Regression with thousands of features, trained and tested on

    millions of samples. Current pCTR Model Kuvasz
  6. pCTR Model History (CC) from Flickr: "Wednesday Freedom 11" by

    Parker Knight (CC) from Flickr: "Icelandig sheepdog" by Thomas Quine (CC) from Flickr: by Craige Moore French Brittany Icelandic Sheepdog Jindo Kuvasz
  7. Lessons Learned (CC) from Flickr: "WEL" by luckyno3

  8. user feedback service online offline data model logs

  9. (CC) from Flickr: "The huge crossing" by Miroslav Petrasko Infrastructure

    (CC) from Flickr: "KOGI and WEL" by luckyno3
  10. user feedback service online offline data model logs

  11. user feedback service logs Log at source of online prediction

    → Prevents downstream modifications of data Logging
  12. user feedback service online offline data model logs

  13. data logs prediction verification Assert validity of logged data Verification

    model
  14. user feedback service online offline data model logs prediction verification

  15. data model logs prediction verification fast scalable Make offline training

    iterations fast & scalable Automation is key → end-to-end pipeline → automated visualizations Tools: mrjob, Spark Iterations
  16. Offline Training at Yelp merge logs sampling feature extraction model

    training evaluation mrjob AWS EMR daily scheduled pipeline kicked off manually mrjob AWS EMR Spark mrjob AWS EMR mrjob AWS EMR mrjob AWS EMR new features (CC) from Flickr: "Cloud" by Jason Pratt
  17. Lessons Learned Infrastructure Log at source of online prediction Verify

    predictions Make offline iterations fast & scalable
  18. Model Comprehension (CC) from Flickr: "Bella" by Maureen Lee

  19. user feedback service online offline data model logs prediction verification

    fast scalable
  20. Focus on a single metric (but don't trust it blindly)

    Evaluation data model prediction verification evaluation fast scalable
  21. Our Metric

  22. Focus on a single metric (but don't trust it blindly)

    Create helpful visualizations Tools: Zeppelin Evaluation data model prediction verification evaluation fast scalable
  23. Visualizations ... feature 1 feature 2 feature 3 ... feature

    contribution Feature contributions sd(feature) * coef Feature value vs. CTR count feature value CTR
  24. user feedback service online offline data model logs prediction verification

    evaluation fast scalable
  25. logs Beware of biased training data → offline != online

    → pCTR threshold Thresholds user feedback service
  26. pCTR Threshold CTR pCTR Model 1 Good CTR pCTR Model

    2 Bad CTR pCTR Model 3 Good
  27. pCTR Threshold time training data Model 1 Model 2 Model

    3 Model 4 Idea: Frequent retraining Better: Deliberate sampling of bad ads CTR pCTR
  28. Online Evaluation CTR pCTR Model 1 Good CTR pCTR Model

    2 Bad CTR pCTR Model 3 Good
  29. Online Evaluation CTR pCTR Model 1 Good CTR pCTR Model

    2 Bad CTR pCTR Model 3 Good
  30. Combined Rescoring new model current model online offline

  31. Combined Rescoring new model current model online offline evaluation

  32. Lessons Learned Infrastructure Log at source of online prediction Verify

    predictions Make offline iterations fast & scalable Model Comprehension Evaluate, evaluate, evaluate Be aware of threshold effects
  33. user feedback service online offline data model logs prediction verification

    evaluation fast scalable
  34. user feedback service online offline data model logs prediction verification

    evaluation fast scalable simplicity
  35. simplicity rule-based approach simple models Occam's razor appropriate metric documentation

    "Simple Made Easy"
  36. user feedback service online offline data model logs prediction verification

    evaluation fast scalable well documented fast scalable well documented simplicity
  37. user feedback service online offline data model logs prediction verification

    evaluation fast scalable well documented fast scalable well documented simplicity
  38. Lessons Learned Above all, keep it simple. Infrastructure Log at

    source of online prediction Verify predictions Make offline iterations fast & scalable Model Comprehension Evaluate, evaluate, evaluate Be aware of threshold effects
  39. @YelpEngineering engineeringblog.yelp.com github.com/yelp yelp.com/careers