$30 off During Our Annual Pro Sale. View Details »

Strataconf 2014 - Chicago Bars, Prisoner's Dile...

Strataconf 2014 - Chicago Bars, Prisoner's Dilemma, and Practical Models in Search

My talk from Strataconf 2014...it has a Shia Lebeouf GIF

Chris Harland

April 08, 2014
Tweet

More Decks by Chris Harland

Other Decks in Research

Transcript

  1. Chicago Bars, Prisoner’s Dilemma, and Practical Models in Search Chris

    Harland Data Scientist @ Microsoft Strataconf 2014 Santa Clara, Ca @cdubhland [email protected]
  2. Chicago Bars, Prisoner’s Dilemma, and Practical Models in Search Chris

    Harland Data Scientist @ Microsoft Strataconf 2014 Santa Clara, Ca @cdubhland [email protected] Who I am…
  3. Chicago Bars, Prisoner’s Dilemma, and Practical Models in Search Chris

    Harland Data Scientist @ Microsoft Strataconf 2014 Santa Clara, Ca @cdubhland [email protected] Who I am… What I do…
  4. Chicago Bars, Prisoner’s Dilemma, and Practical Models in Search Chris

    Harland Data Scientist @ Microsoft Strataconf 2014 Santa Clara, Ca @cdubhland [email protected] Who I am… What I do… Where I work…
  5. Chicago Bars, Prisoner’s Dilemma, and Practical Models in Search Chris

    Harland Data Scientist @ Microsoft Strataconf 2014 Santa Clara, Ca @cdubhland [email protected] Who I am… What I do… Where I work… Where you can find me…
  6. Goal: Best score on leaderboard The “best” model is not

    always the one that achieves the desired results… Goal: Make money
  7. Life is tough for a Chicago Bar… “No new tavern

    licenses can be issued to any location that is within 400 feet of existing businesses already licensed for the sale of alcoholic liquor in certain zoning districts.” – City of Chicago Ordinance
  8. Bars need to make more money Get new customers Or…

    Get current customers to spend more
  9. Bars need to make more money Get new customers Or…

    Get current customers to spend more
  10. Build a recommendation (or at least rank) Collaborative Filtering From

    my bag of 200k possible “users”…who do I send a mailer to first?
  11. User Group 02 0.986 User Group 15 0.963 User Group

    13 0.942 User Group 20 0.921 User Group 16 0.900 User Group 05 0.898 My Users Start Sending Mail
  12. 13.5% lift in response rate Response rate -> $$ No

    change in business model / tactics
  13. What users features are important? IsCoronaDrinker: -> strong predictive power

    -> captures a lot variation How does a bar digest this information?
  14. What users features are important? IsCoronaDrinker: -> strong predictive power

    -> captures a lot variation How does a bar digest this information? They don’t need a model… They need an action
  15. Bars need to make more money Get new customers Or…

    Get current customers to spend more
  16. How do we interpret this problem given the data at

    hand? Bucket users into spending types
  17. How do we interpret this problem given the data at

    hand? Bucket users into spending types Find the good buckets
  18. Old friend Corona… IsCoronaDrinker is of high importance / predictive

    power But it doesn’t help my bar… They can’t make you drink Corona…
  19. Remove features from model…leave ones with actionable segmentation Pay cost

    in accuracy for the benefit of action… (few percent)
  20. Remove features from model…leave ones with actionable segmentation Pay cost

    in accuracy for the benefit of action… (few percent) visitsHappyHour bubbles up the variable importance…
  21. Amount of perceived special time users spend at your bar

    Cheap food for full price drinks…
  22. Ulterior motive: start at happy hour… stay past… $$ Happy

    Hour Time Window Just put people on the edge of the window
  23. Okay…but how do you do all of this for one

    bar? You don’t…you do it for a lot of bars
  24. Image by Chris Jensen and Greg Riestenberg Players: Diamond Circle

    Scenario: Crime Decision: Defect Penalty: Prison
  25. Image by Chris Jensen and Greg Riestenberg Players: Diamond Circle

    Scenario: Crime Decision: Defect Penalty: Prison Turn on each other
  26. Image by Chris Jensen and Greg Riestenberg Players: Diamond Circle

    Scenario: Crime Decision: Defect Penalty: Prison Luck out
  27. Image by Chris Jensen and Greg Riestenberg Players: Diamond Circle

    Scenario: Crime Decision: Defect Penalty: Prison Keep quiet
  28. Image by Chris Jensen and Greg Riestenberg Players: Diamond Circle

    Scenario: Crime Decision: Defect Penalty: Prison Players: Bar A Bar B Scenario: Data Decision: Share Penalty: Loss of potential $$
  29. We hold all the data…but don’t expose to participants This

    is the most crucial piece of the whole system… Central Data Bar A Bar B Bar C
  30. But what about search users? Search is whatever users want

    it to be… Value can come from exploring search behavior and surfacing scenarios…
  31. What is a user to a data scientist? Collection of

    log lines… What does a user mean when they type “Tom Cruise”?
  32. We have a graph… We have an adjacent graph (think

    Wikipedia)… Find the “best” path between nodes in adjacent graph…
  33. Legend Tom Cruise Mia Sara Tim Curry David Bennent Nodes

    are from session graph… Links are from adjacent graph Defining the transition from one graph to the other is tough
  34. Models like this are great for back end understanding… They

    allow for long tail behavior bucketing…
  35. Models like this are great for back end understanding… They

    allow for long tail behavior bucketing… But…they are bad for naïve application…almost no one saw “Legend”
  36. Models like this are great for back end understanding… They

    allow for long tail behavior bucketing… But…they are bad for naïve application…almost no one saw “Legend” And can sometimes transition to production…
  37. When making a model… Create with a purpose… Abstract your

    business question…but not too far… Understand when good is good enough…
  38. Chicago Bars, Prisoner’s Dilemma, and Practical Models in Search Chris

    Harland Data Scientist @ Microsoft Strataconf 2014 Santa Clara, Ca @cdubhland [email protected]