Tweet
Tweet

Slide 1

Slide 1 text

Anne Schuth (Blendle / University of Amsterdam, The Netherlands) Krisztian Balog (University of Stavanger, Norway) Tutorial at ECIR 2016 in Padua, Italy Simulations

Slide 2

Slide 2 text

Why Simulate Users? • If we believe in online evaluation?

Slide 3

Slide 3 text

Why Simulate Users? • If we believe in online evaluation? • Because

Slide 4

Slide 4 text

Why Simulate Users? • If we believe in online evaluation? • Because • We want to understand online evaluation

Slide 5

Slide 5 text

Why Simulate Users? • If we believe in online evaluation? • Because • We want to understand online evaluation • We want to be able to control variables

Slide 6

Slide 6 text

Why Simulate Users? • If we believe in online evaluation? • Because • We want to understand online evaluation • We want to be able to control variables • We want to test ideas before we expose them to users

Slide 7

Slide 7 text

Why Simulate Users? • If we believe in online evaluation? • Because • We want to understand online evaluation • We want to be able to control variables • We want to test ideas before we expose them to users • We make mistakes

Slide 8

Slide 8 text

Simulations • Random Buckets

Slide 9

Slide 9 text

Simulations • Random Buckets • Historical interactions

Slide 10

Slide 10 text

Simulations • Random Buckets • Historical interactions • Collected from real users (that saw presumably bad rankings)

Slide 11

Slide 11 text

Simulations • Random Buckets • Historical interactions • Collected from real users (that saw presumably bad rankings) • Click Models

Slide 12

Slide 12 text

Simulations • Random Buckets • Historical interactions • Collected from real users (that saw presumably bad rankings) • Click Models • Use TREC-Style evaluation set

Slide 13

Slide 13 text

Simulations • Random Buckets • Historical interactions • Collected from real users (that saw presumably bad rankings) • Click Models • Use TREC-Style evaluation set • Sample queries (with replacement)

Slide 14

Slide 14 text

Simulations • Random Buckets • Historical interactions • Collected from real users (that saw presumably bad rankings) • Click Models • Use TREC-Style evaluation set • Sample queries (with replacement) • Condition clicks on relevance assessments

Slide 15

Slide 15 text

Click Models • Random Click Model (RCM) • Rank-based CTR Model (RCTR) • Document-based CTR Model (DCTR) • Position-based Model (PBM) • Dynamic Bayesian Network Model (DBN) • User Browsing Model (UBM) • Cascade Model (CM)

Slide 16

Slide 16 text

Cascade Click Model • Assume users go through a SERP from top to bottom

Slide 17

Slide 17 text

Cascade Click Model • Assume users go through a SERP from top to bottom • Examine every snippet

Slide 18

Slide 18 text

Cascade Click Model • Assume users go through a SERP from top to bottom • Examine every snippet • With probability P(Click=1|R) a user clicks

Slide 19

Slide 19 text

Cascade Click Model • Assume users go through a SERP from top to bottom • Examine every snippet • With probability P(Click=1|R) a user clicks • R is the relevance label

Slide 20

Slide 20 text

Cascade Click Model • Assume users go through a SERP from top to bottom • Examine every snippet • With probability P(Click=1|R) a user clicks • R is the relevance label • After clicking, a user is satisfied and stops with probability P(Stop=1|Click=1,R)

Slide 21

Slide 21 text

Cascade Click Model • Assume users go through a SERP from top to bottom • Examine every snippet • With probability P(Click=1|R) a user clicks • R is the relevance label • After clicking, a user is satisfied and stops with probability P(Stop=1|Click=1,R) • These probabilities can be inferred from logs

Slide 22

Slide 22 text

Simulations doc 1 doc 2 doc 3 doc 4 doc 5 doc 2 doc 4 doc 7 doc 1 doc 3 Site Participant

Slide 23

Slide 23 text

Simulations doc 1 doc 2 doc 3 doc 4 doc 5 doc 2 doc 4 doc 7 doc 1 doc 3 Site Participant

Slide 24

Slide 24 text

Simulations doc 1 doc 3 doc 2 doc 4 doc 7 Site Participant

Slide 25

Slide 25 text

Simulations doc 1 doc 3 doc 2 doc 4 doc 7 Site Participant

Slide 26

Slide 26 text

Simulations doc 1 doc 3 doc 2 doc 4 doc 7 Site Participant

Slide 27

Slide 27 text

Simulations doc 1 doc 3 doc 2 doc 4 doc 7 Site Participant P(Click=1|R=1) = 0.9

Slide 28

Slide 28 text

Simulations doc 1 doc 3 doc 2 doc 4 doc 7 Site Participant P(Click=1|R=1) = 0.9 P(Click=1|R=0) = 0.05

Slide 29

Slide 29 text

Simulations doc 1 doc 3 doc 2 doc 4 doc 7 Site Participant P(Click=1|R=1) = 0.9 P(Click=1|R=0) = 0.05 P(Click=1|R=1) = 0.9

Slide 30

Slide 30 text

Simulations doc 1 doc 3 doc 2 doc 4 doc 7 Site Participant P(Click=1|R=1) = 0.9 P(Click=1|R=0) = 0.05 P(Click=1|R=1) = 0.9 P(Stop=1|Click=1,R=1) = 0.9

Slide 31

Slide 31 text

Simulations • Used in Living Labs • Sign up for sites marked with