Intrinsic Elicitation: A Model and Design Approach for Games Collecting Human Subject Data

Presentation at FDG'18 on how to design games that collect human subject data where you have no ground truth.

Transcript

1. Intrinsic Elicitation A Model and Design Approach for Games Collecting Human Subject Data David Gundry University of York Sebastian Deterding University of York
3. Human Subject Data Solution consensus formalisable do your best preferences, beliefs
4. Existing work • Motivating or ensuring data quality • Treat motivation and data quality as separate concerns • Gamification + Validation
7. The Challenge of Human Subject Data • Cannot validate – no ground truth • Games are motivation and threat to validity
8. The Challenge of Human Subject Data • Dishonest responding • Demand characteristics • Priming • Confounds • Gaming the system • Careless responding
9. Looking at Surveys • Why do people provide (dis)honest data?
10. Looking at Surveys • Why do people provide (dis)honest data? • Why do people do things within a game? – Rational Player Model (Jonas Heide Smith, 2006)
16. Strategic Choice Data Provision Mechanic Use Valid Target Data
17. Strategic Choice Data Provision Mechanic Use Valid Target Data Centrality Necessity Veracity
18. NECESSITY Embody data provision tasks as the game’s interesting mechanics
21. CENTRALITY Make data-providing mechanics as strategically central and effortless as possible
23. VERACITY Where honest responses are needed, ensure they have the highest strategic utility and lowest effort
26. Summary • Gamification+Validation doesn't work for human subject data • Rational Game User Model & Intrinsic Elicitation
27. Ongoing work • Evaluating Intrinsic Elicitation • Linguistic data • Pastry Chefs (4 player card game) • “Word-order game” (single player digital game) playpastrychefs.com
28. Thanks for listening! @davegundry deg500@york.ac.uk David Gundry
29. Smith, J.H., 2006. Plans and Purposes. PhD Thesis. IT University of Copenhagen