Upgrade to Pro — share decks privately, control downloads, hide ads and more …

10 Simulations

Avatar for LiLa'16 LiLa'16
March 20, 2016

10 Simulations

Avatar for LiLa'16

LiLa'16

March 20, 2016
Tweet

More Decks by LiLa'16

Other Decks in Research

Transcript

  1. Anne Schuth (Blendle / University of Amsterdam, The Netherlands) Krisztian

    Balog (University of Stavanger, Norway) Tutorial at ECIR 2016 in Padua, Italy Simulations
  2. Why Simulate Users? • If we believe in online evaluation?

    • Because • We want to understand online evaluation
  3. Why Simulate Users? • If we believe in online evaluation?

    • Because • We want to understand online evaluation • We want to be able to control variables
  4. Why Simulate Users? • If we believe in online evaluation?

    • Because • We want to understand online evaluation • We want to be able to control variables • We want to test ideas before we expose them to users
  5. Why Simulate Users? • If we believe in online evaluation?

    • Because • We want to understand online evaluation • We want to be able to control variables • We want to test ideas before we expose them to users • We make mistakes
  6. Simulations • Random Buckets • Historical interactions • Collected from

    real users (that saw presumably bad rankings) • Click Models
  7. Simulations • Random Buckets • Historical interactions • Collected from

    real users (that saw presumably bad rankings) • Click Models • Use TREC-Style evaluation set
  8. Simulations • Random Buckets • Historical interactions • Collected from

    real users (that saw presumably bad rankings) • Click Models • Use TREC-Style evaluation set • Sample queries (with replacement)
  9. Simulations • Random Buckets • Historical interactions • Collected from

    real users (that saw presumably bad rankings) • Click Models • Use TREC-Style evaluation set • Sample queries (with replacement) • Condition clicks on relevance assessments
  10. Click Models • Random Click Model (RCM) • Rank-based CTR

    Model (RCTR) • Document-based CTR Model (DCTR) • Position-based Model (PBM) • Dynamic Bayesian Network Model (DBN) • User Browsing Model (UBM) • Cascade Model (CM)
  11. Cascade Click Model • Assume users go through a SERP

    from top to bottom • Examine every snippet
  12. Cascade Click Model • Assume users go through a SERP

    from top to bottom • Examine every snippet • With probability P(Click=1|R) a user clicks
  13. Cascade Click Model • Assume users go through a SERP

    from top to bottom • Examine every snippet • With probability P(Click=1|R) a user clicks • R is the relevance label
  14. Cascade Click Model • Assume users go through a SERP

    from top to bottom • Examine every snippet • With probability P(Click=1|R) a user clicks • R is the relevance label • After clicking, a user is satisfied and stops with probability P(Stop=1|Click=1,R)
  15. Cascade Click Model • Assume users go through a SERP

    from top to bottom • Examine every snippet • With probability P(Click=1|R) a user clicks • R is the relevance label • After clicking, a user is satisfied and stops with probability P(Stop=1|Click=1,R) • These probabilities can be inferred from logs
  16. Simulations doc 1 doc 2 doc 3 doc 4 doc

    5 doc 2 doc 4 doc 7 doc 1 doc 3 Site Participant
  17. Simulations doc 1 doc 2 doc 3 doc 4 doc

    5 doc 2 doc 4 doc 7 doc 1 doc 3 Site Participant
  18. Simulations doc 1 doc 3 doc 2 doc 4 doc

    7 Site Participant P(Click=1|R=1) = 0.9
  19. Simulations doc 1 doc 3 doc 2 doc 4 doc

    7 Site Participant P(Click=1|R=1) = 0.9 P(Click=1|R=0) = 0.05
  20. Simulations doc 1 doc 3 doc 2 doc 4 doc

    7 Site Participant P(Click=1|R=1) = 0.9 P(Click=1|R=0) = 0.05 P(Click=1|R=1) = 0.9
  21. Simulations doc 1 doc 3 doc 2 doc 4 doc

    7 Site Participant P(Click=1|R=1) = 0.9 P(Click=1|R=0) = 0.05 P(Click=1|R=1) = 0.9 P(Stop=1|Click=1,R=1) = 0.9