PyData SF: Large Scale CTR Prediction - Lessons Learned

Starting with a basic setup for click-through rate (CTR) prediction, we will step by step improve on it by incorporating the lessons we've learned from operating and scaling such a mission-critical system. The presented lessons will be related to infrastructure, model comprehension, and specifics like how to deal with thresholds. They should be applicable to most ML models used in production.

Link to video: https://youtu.be/zbAuoFUdjAI

Transcript

1. Florian Hartl florianh@yelp.com Large Scale CTR Prediction Lessons Learned
2. Yelp’s Mission Connecting people with great local businesses.
3. 92M 32 72% 108M Yelp Stats As of Q2 2016
4. CTR Prediction CTR: Click-Through Rate pCTR: predicted CTR Question How likely is the user to click on the ad? Why Proxy for relevance 5.5% 0.8% 9.2% ?
5. Logistic Regression with thousands of features, trained and tested on millions of samples. Current pCTR Model Kuvasz
6. pCTR Model History (CC) from Flickr: "Wednesday Freedom 11" by Parker Knight (CC) from Flickr: "Icelandig sheepdog" by Thomas Quine (CC) from Flickr: by Craige Moore French Brittany Icelandic Sheepdog Jindo Kuvasz
7. Lessons Learned (CC) from Flickr: "WEL" by luckyno3
8. user feedback service online offline data model logs
9. (CC) from Flickr: "The huge crossing" by Miroslav Petrasko Infrastructure (CC) from Flickr: "KOGI and WEL" by luckyno3
10. user feedback service online offline data model logs
11. user feedback service logs Log at source of online prediction → Prevents downstream modifications of data Logging
12. user feedback service online offline data model logs
13. data logs prediction verification Assert validity of logged data Verification model
14. user feedback service online offline data model logs prediction verification
15. data model logs prediction verification fast scalable Make offline training iterations fast & scalable Automation is key → end-to-end pipeline → automated visualizations Tools: mrjob, Spark Iterations
16. Offline Training at Yelp merge logs sampling feature extraction model training evaluation mrjob AWS EMR daily scheduled pipeline kicked off manually mrjob AWS EMR Spark mrjob AWS EMR mrjob AWS EMR mrjob AWS EMR new features (CC) from Flickr: "Cloud" by Jason Pratt
17. Lessons Learned Infrastructure Log at source of online prediction Verify predictions Make offline iterations fast & scalable
18. Model Comprehension (CC) from Flickr: "Bella" by Maureen Lee
19. user feedback service online offline data model logs prediction verification fast scalable
20. Focus on a single metric (but don't trust it blindly) Evaluation data model prediction verification evaluation fast scalable
21. Our Metric
22. Focus on a single metric (but don't trust it blindly) Create helpful visualizations Tools: Zeppelin Evaluation data model prediction verification evaluation fast scalable
23. Visualizations ... feature 1 feature 2 feature 3 ... feature contribution Feature contributions sd(feature) * coef Feature value vs. CTR count feature value CTR
24. user feedback service online offline data model logs prediction verification evaluation fast scalable
25. logs Beware of biased training data → offline != online → pCTR threshold Thresholds user feedback service
26. pCTR Threshold CTR pCTR Model 1 Good CTR pCTR Model 2 Bad CTR pCTR Model 3 Good
27. pCTR Threshold time training data Model 1 Model 2 Model 3 Model 4 Idea: Frequent retraining Better: Deliberate sampling of bad ads CTR pCTR
28. Online Evaluation CTR pCTR Model 1 Good CTR pCTR Model 2 Bad CTR pCTR Model 3 Good
29. Online Evaluation CTR pCTR Model 1 Good CTR pCTR Model 2 Bad CTR pCTR Model 3 Good
30. Combined Rescoring new model current model online offline
31. Combined Rescoring new model current model online offline evaluation
32. Lessons Learned Infrastructure Log at source of online prediction Verify predictions Make offline iterations fast & scalable Model Comprehension Evaluate, evaluate, evaluate Be aware of threshold effects
33. user feedback service online offline data model logs prediction verification evaluation fast scalable
34. user feedback service online offline data model logs prediction verification evaluation fast scalable simplicity
35. simplicity rule-based approach simple models Occam's razor appropriate metric documentation "Simple Made Easy"
36. user feedback service online offline data model logs prediction verification evaluation fast scalable well documented fast scalable well documented simplicity
37. user feedback service online offline data model logs prediction verification evaluation fast scalable well documented fast scalable well documented simplicity
38. Lessons Learned Above all, keep it simple. Infrastructure Log at source of online prediction Verify predictions Make offline iterations fast & scalable Model Comprehension Evaluate, evaluate, evaluate Be aware of threshold effects
39. @YelpEngineering engineeringblog.yelp.com github.com/yelp yelp.com/careers