PyData SF: Large Scale CTR Prediction - Lessons Learned

Slide 1

Slide 1 text

Florian Hartl [email protected] Large Scale CTR Prediction Lessons Learned

Slide 2

Slide 2 text

Yelp’s Mission Connecting people with great local businesses.

Slide 3

Slide 3 text

92M 32 72% 108M Yelp Stats As of Q2 2016

Slide 4

Slide 4 text

CTR Prediction CTR: Click-Through Rate pCTR: predicted CTR Question How likely is the user to click on the ad? Why Proxy for relevance 5.5% 0.8% 9.2% ?

Slide 5

Slide 5 text

Logistic Regression with thousands of features, trained and tested on millions of samples. Current pCTR Model Kuvasz

Slide 6

Slide 6 text

pCTR Model History (CC) from Flickr: "Wednesday Freedom 11" by Parker Knight (CC) from Flickr: "Icelandig sheepdog" by Thomas Quine (CC) from Flickr: by Craige Moore French Brittany Icelandic Sheepdog Jindo Kuvasz

Slide 7

Slide 7 text

Lessons Learned (CC) from Flickr: "WEL" by luckyno3

Slide 8

Slide 8 text

user feedback service online offline data model logs

Slide 9

Slide 9 text

(CC) from Flickr: "The huge crossing" by Miroslav Petrasko Infrastructure (CC) from Flickr: "KOGI and WEL" by luckyno3

Slide 10

Slide 10 text

user feedback service online offline data model logs

Slide 11

Slide 11 text

user feedback service logs Log at source of online prediction → Prevents downstream modifications of data Logging

Slide 12

Slide 12 text

user feedback service online offline data model logs

Slide 13

Slide 13 text

data logs prediction verification Assert validity of logged data Verification model

Slide 14

Slide 14 text

user feedback service online offline data model logs prediction verification

Slide 15

Slide 15 text

data model logs prediction verification fast scalable Make offline training iterations fast & scalable Automation is key → end-to-end pipeline → automated visualizations Tools: mrjob, Spark Iterations

Slide 16

Slide 16 text

Offline Training at Yelp merge logs sampling feature extraction model training evaluation mrjob AWS EMR daily scheduled pipeline kicked off manually mrjob AWS EMR Spark mrjob AWS EMR mrjob AWS EMR mrjob AWS EMR new features (CC) from Flickr: "Cloud" by Jason Pratt

Slide 17

Slide 17 text

Lessons Learned Infrastructure Log at source of online prediction Verify predictions Make offline iterations fast & scalable

Slide 18

Slide 18 text

Model Comprehension (CC) from Flickr: "Bella" by Maureen Lee

Slide 19

Slide 19 text

user feedback service online offline data model logs prediction verification fast scalable

Slide 20

Slide 20 text

Focus on a single metric (but don't trust it blindly) Evaluation data model prediction verification evaluation fast scalable

Slide 21

Slide 21 text

Our Metric

Slide 22

Slide 22 text

Focus on a single metric (but don't trust it blindly) Create helpful visualizations Tools: Zeppelin Evaluation data model prediction verification evaluation fast scalable

Slide 23

Slide 23 text

Visualizations ... feature 1 feature 2 feature 3 ... feature contribution Feature contributions sd(feature) * coef Feature value vs. CTR count feature value CTR

Slide 24

Slide 24 text

user feedback service online offline data model logs prediction verification evaluation fast scalable

Slide 25

Slide 25 text

logs Beware of biased training data → offline != online → pCTR threshold Thresholds user feedback service

Slide 26

Slide 26 text

pCTR Threshold CTR pCTR Model 1 Good CTR pCTR Model 2 Bad CTR pCTR Model 3 Good

Slide 27

Slide 27 text

pCTR Threshold time training data Model 1 Model 2 Model 3 Model 4 Idea: Frequent retraining Better: Deliberate sampling of bad ads CTR pCTR

Slide 28

Slide 28 text

Online Evaluation CTR pCTR Model 1 Good CTR pCTR Model 2 Bad CTR pCTR Model 3 Good

Slide 29

Slide 29 text

Online Evaluation CTR pCTR Model 1 Good CTR pCTR Model 2 Bad CTR pCTR Model 3 Good

Slide 30

Slide 30 text

Combined Rescoring new model current model online offline

Slide 31

Slide 31 text

Combined Rescoring new model current model online offline evaluation

Slide 32

Slide 32 text

Lessons Learned Infrastructure Log at source of online prediction Verify predictions Make offline iterations fast & scalable Model Comprehension Evaluate, evaluate, evaluate Be aware of threshold effects

Slide 33

Slide 33 text

user feedback service online offline data model logs prediction verification evaluation fast scalable

Slide 34

Slide 34 text

user feedback service online offline data model logs prediction verification evaluation fast scalable simplicity

Slide 35

Slide 35 text

simplicity rule-based approach simple models Occam's razor appropriate metric documentation "Simple Made Easy"

Slide 36

Slide 36 text

user feedback service online offline data model logs prediction verification evaluation fast scalable well documented fast scalable well documented simplicity

Slide 37

Slide 37 text

user feedback service online offline data model logs prediction verification evaluation fast scalable well documented fast scalable well documented simplicity

Slide 38

Slide 38 text

Lessons Learned Above all, keep it simple. Infrastructure Log at source of online prediction Verify predictions Make offline iterations fast & scalable Model Comprehension Evaluate, evaluate, evaluate Be aware of threshold effects

Slide 39

Slide 39 text

@YelpEngineering engineeringblog.yelp.com github.com/yelp yelp.com/careers