10 Simulations

1. Anne Schuth (Blendle / University of Amsterdam, The Netherlands) Krisztian Balog (University of Stavanger, Norway) Tutorial at ECIR 2016 in Padua, Italy Simulations
2. Why Simulate Users? • If we believe in online evaluation?
3. Why Simulate Users? • If we believe in online evaluation? • Because
4. Why Simulate Users? • If we believe in online evaluation? • Because • We want to understand online evaluation
5. Why Simulate Users? • If we believe in online evaluation? • Because • We want to understand online evaluation • We want to be able to control variables
6. Why Simulate Users? • If we believe in online evaluation? • Because • We want to understand online evaluation • We want to be able to control variables • We want to test ideas before we expose them to users
7. Why Simulate Users? • If we believe in online evaluation? • Because • We want to understand online evaluation • We want to be able to control variables • We want to test ideas before we expose them to users • We make mistakes
8. Simulations • Random Buckets
9. Simulations • Random Buckets • Historical interactions
10. Simulations • Random Buckets • Historical interactions • Collected from real users (that saw presumably bad rankings)
11. Simulations • Random Buckets • Historical interactions • Collected from real users (that saw presumably bad rankings) • Click Models
12. Simulations • Random Buckets • Historical interactions • Collected from real users (that saw presumably bad rankings) • Click Models • Use TREC-Style evaluation set
13. Simulations • Random Buckets • Historical interactions • Collected from real users (that saw presumably bad rankings) • Click Models • Use TREC-Style evaluation set • Sample queries (with replacement)
14. Simulations • Random Buckets • Historical interactions • Collected from real users (that saw presumably bad rankings) • Click Models • Use TREC-Style evaluation set • Sample queries (with replacement) • Condition clicks on relevance assessments
15. Click Models • Random Click Model (RCM) • Rank-based CTR Model (RCTR) • Document-based CTR Model (DCTR) • Position-based Model (PBM) • Dynamic Bayesian Network Model (DBN) • User Browsing Model (UBM) • Cascade Model (CM)
16. Cascade Click Model • Assume users go through a SERP from top to bottom
17. Cascade Click Model • Assume users go through a SERP from top to bottom • Examine every snippet
18. Cascade Click Model • Assume users go through a SERP from top to bottom • Examine every snippet • With probability P(Click=1|R) a user clicks
19. Cascade Click Model • Assume users go through a SERP from top to bottom • Examine every snippet • With probability P(Click=1|R) a user clicks • R is the relevance label
20. Cascade Click Model • Assume users go through a SERP from top to bottom • Examine every snippet • With probability P(Click=1|R) a user clicks • R is the relevance label • After clicking, a user is satisfied and stops with probability P(Stop=1|Click=1,R)
21. Cascade Click Model • Assume users go through a SERP from top to bottom • Examine every snippet • With probability P(Click=1|R) a user clicks • R is the relevance label • After clicking, a user is satisfied and stops with probability P(Stop=1|Click=1,R) • These probabilities can be inferred from logs
22. Simulations doc 1 doc 2 doc 3 doc 4 doc 5 doc 2 doc 4 doc 7 doc 1 doc 3 Site Participant
23. Simulations doc 1 doc 2 doc 3 doc 4 doc 5 doc 2 doc 4 doc 7 doc 1 doc 3 Site Participant
24. Simulations doc 1 doc 3 doc 2 doc 4 doc 7 Site Participant
25. Simulations doc 1 doc 3 doc 2 doc 4 doc 7 Site Participant
26. Simulations doc 1 doc 3 doc 2 doc 4 doc 7 Site Participant
27. Simulations doc 1 doc 3 doc 2 doc 4 doc 7 Site Participant P(Click=1|R=1) = 0.9
28. Simulations doc 1 doc 3 doc 2 doc 4 doc 7 Site Participant P(Click=1|R=1) = 0.9 P(Click=1|R=0) = 0.05
29. Simulations doc 1 doc 3 doc 2 doc 4 doc 7 Site Participant P(Click=1|R=1) = 0.9 P(Click=1|R=0) = 0.05 P(Click=1|R=1) = 0.9
30. Simulations doc 1 doc 3 doc 2 doc 4 doc 7 Site Participant P(Click=1|R=1) = 0.9 P(Click=1|R=0) = 0.05 P(Click=1|R=1) = 0.9 P(Stop=1|Click=1,R=1) = 0.9
31. Simulations • Used in Living Labs • Sign up for sites marked with