Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Intrinsic Elicitation: A Model and Design Approach for Games Collecting Human Subject Data

9f5f8d89c1c298beb2e5d3b1e41941e2?s=47 Sebastian Deterding
August 09, 2018
65

Intrinsic Elicitation: A Model and Design Approach for Games Collecting Human Subject Data

Presentation at FDG'18 on how to design games that collect human subject data where you have no ground truth.

9f5f8d89c1c298beb2e5d3b1e41941e2?s=128

Sebastian Deterding

August 09, 2018
Tweet

Transcript

  1. Intrinsic Elicitation A Model and Design Approach for Games Collecting

    Human Subject Data David Gundry University of York Sebastian Deterding University of York
  2. None
  3. Human Subject Data Solution consensus formalisable do your best preferences,

    beliefs
  4. Existing work • Motivating or ensuring data quality • Treat

    motivation and data quality as separate concerns • Gamification + Validation
  5. None
  6. None
  7. The Challenge of Human Subject Data • Cannot validate –

    no ground truth • Games are motivation and threat to validity
  8. The Challenge of Human Subject Data • Dishonest responding •

    Demand characteristics • Priming • Confounds • Gaming the system • Careless responding
  9. Looking at Surveys • Why do people provide (dis)honest data?

  10. Looking at Surveys • Why do people provide (dis)honest data?

    • Why do people do things within a game? – Rational Player Model (Jonas Heide Smith, 2006)
  11. None
  12. None
  13. None
  14. None
  15. None
  16. Strategic Choice Data Provision Mechanic Use Valid Target Data

  17. Strategic Choice Data Provision Mechanic Use Valid Target Data Centrality

    Necessity Veracity
  18. NECESSITY Embody data provision tasks as the game’s interesting mechanics

  19. None
  20. None
  21. CENTRALITY Make data-providing mechanics as strategically central and effortless as

    possible
  22. None
  23. VERACITY Where honest responses are needed, ensure they have the

    highest strategic utility and lowest effort
  24. None
  25. None
  26. Summary • Gamification+Validation doesn't work for human subject data •

    Rational Game User Model & Intrinsic Elicitation
  27. Ongoing work • Evaluating Intrinsic Elicitation • Linguistic data •

    Pastry Chefs (4 player card game) • “Word-order game” (single player digital game) playpastrychefs.com
  28. Thanks for listening! @davegundry deg500@york.ac.uk David Gundry

  29. Smith, J.H., 2006. Plans and Purposes. PhD Thesis. IT University

    of Copenhagen
  30. None