Intrinsic Elicitation
A Model and Design Approach for Games
Collecting Human Subject Data
David Gundry
University of York
Sebastian Deterding
University of York
Slide 2
Slide 2 text
No content
Slide 3
Slide 3 text
Human Subject Data
Solution
consensus
formalisable
do your best
preferences,
beliefs
Slide 4
Slide 4 text
Existing work
• Motivating or ensuring data quality
• Treat motivation and data quality as separate
concerns
• Gamification + Validation
Slide 5
Slide 5 text
No content
Slide 6
Slide 6 text
No content
Slide 7
Slide 7 text
The Challenge of Human Subject Data
• Cannot validate – no ground truth
• Games are motivation and threat to validity
Slide 8
Slide 8 text
The Challenge of Human Subject Data
• Dishonest responding
• Demand characteristics
• Priming
• Confounds
• Gaming the system
• Careless responding
Slide 9
Slide 9 text
Looking at Surveys
• Why do people provide (dis)honest data?
Slide 10
Slide 10 text
Looking at Surveys
• Why do people provide (dis)honest data?
• Why do people do things within a game?
– Rational Player Model (Jonas Heide Smith, 2006)
Slide 11
Slide 11 text
No content
Slide 12
Slide 12 text
No content
Slide 13
Slide 13 text
No content
Slide 14
Slide 14 text
No content
Slide 15
Slide 15 text
No content
Slide 16
Slide 16 text
Strategic
Choice
Data
Provision
Mechanic
Use
Valid Target
Data
Slide 17
Slide 17 text
Strategic
Choice
Data
Provision
Mechanic
Use
Valid Target
Data
Centrality Necessity Veracity
Slide 18
Slide 18 text
NECESSITY
Embody data provision tasks as the game’s interesting mechanics
Slide 19
Slide 19 text
No content
Slide 20
Slide 20 text
No content
Slide 21
Slide 21 text
CENTRALITY
Make data-providing mechanics as strategically central and effortless
as possible
Slide 22
Slide 22 text
No content
Slide 23
Slide 23 text
VERACITY
Where honest responses are needed, ensure they have the highest
strategic utility and lowest effort
Slide 24
Slide 24 text
No content
Slide 25
Slide 25 text
No content
Slide 26
Slide 26 text
Summary
• Gamification+Validation doesn't work for human
subject data
• Rational Game User Model & Intrinsic Elicitation
Slide 27
Slide 27 text
Ongoing work
• Evaluating Intrinsic Elicitation
• Linguistic data
• Pastry Chefs (4 player card game)
• “Word-order game” (single player digital game)
playpastrychefs.com