hajimizu
August 21, 2019

Second talk in ISAGA summer school 2019

Capturing, analyzing and interpreting data from games

August 21, 2019

Transcript

1. ISAGA Summer School 2019 Capturing, analyzing and interpreting data from

games Aug. 21st, 2019 Hajime Mizuyama mizuyama@ise.aoyama.ac.jp, hajimizu@kth.se 1
2. ISAGA Summer School 2019 Agenda • Answering research questions and

data analysis • Some fictional examples of game play experiments – Case 1: Repetition – Case 2: Randomization – Case 3: Local control (Blocking) – Case 4: Designing experiments efficiently • Discussion on experimental study using games • Summary 2
3. ISAGA Summer School 2019 => Based on data Answering research

questions • Research questions can be answered: – Deductively Theoretical research, ex. in mathematics, etc. – Inductively Confirmation, quantification, etc. of hypotheses – Abductively Generation of hypotheses Constructive approach is also categorized to this class. 3
4. ISAGA Summer School 2019 Confirmatory & exploratory study Confirmatory study

(for induction) • Statistical Test • Estimated parameter values in supervised learning (regression, classification, etc.) Exploratory study (for abduction) • Summarization and visualization • Unsupervised learning (clustering, dimensionality reduction, etc.) • Emergent model structure in supervised learning 4
5. ISAGA Summer School 2019 Experimental & observational study Experimental study

• The values of explanatory variables are set or changed to specified ones through intervention and the resultant response is measured. • Suitable for testing or quantifying causal relationship. Observational study • Both explanatory and response variables are merely measured, sometimes with no prior distinction between them. • Cohort or case-control approach is used for causal inference. 5

Exploratory
7. ISAGA Summer School 2019 Two-dimensional framework Experimental Observational • Sometimes,

it is not easy to control the values of explanatory variables. • Cohort or case-control approach may be used in such cases. • It makes analysis easier to obtain data in a balanced way. • Stratification can be combined with exploratory analysis. 7 Confirmatory Exploratory
8. ISAGA Summer School 2019 Agenda • Answering research questions and

data analysis • Some fictional examples of game play experiments – Case 1: Repetition – Case 2: Randomization – Case 3: Local control (Blocking) – Case 4: Designing experiments efficiently • Discussion on experimental study using games • Summary 8
9. ISAGA Summer School 2019 Case 1 A y Factor Response

RQ: Does factor A have an effect on response y? 9 A1 , A2 , … Levels Suppose that we let a participant play the game once with every level of factor A, and measure the response. Is it OK to answer YES, when the response values are different?
10. ISAGA Summer School 2019 Realized values & distributions A1 A2

y 10 Response needs to be treated as a random variable*. A difference between realized values may be attributed to chance. What is of concern is the difference between the distributions behind them. (*According to central limit theorem, normal distribution is often assumed.) Effect?
11. ISAGA Summer School 2019 Fisher’s principle 1: repetition A1 A2

y (\$, ' () ((, ' () 11 Response needs to be treated as a random variable*. A difference between realized values may be attributed to chance. What is of concern is the difference between the distributions behind them. (*According to central limit theorem, normal distribution is often assumed.)
12. ISAGA Summer School 2019 One-way design Data A1 y11 y12

y1N A2 y21 y22 y2N AI yI1 yI2 yIN 12
13. ISAGA Summer School 2019 Model equation +, = + +

+ +, Assumptions – Sum of main effects is 0: \$ + ( + ⋯ + 2 = 0. – Error terms independently follow a Normal distribution with mean 0. Factorial effect model for one-way design General mean Main effect of A Error ~(, ' ( 7 ) 13
14. ISAGA Summer School 2019 Mean values & low of large

numbers Data Mean A1 y11 y12 y1N A2 y21 y22 y2N AI yI1 yI2 yIN 9 \$ = + \$ + 1 ; ,<\$ = \$, 9 ( = + ( + 1 ; ,<\$ = (, 9 2 = + 2 + 1 ; ,<\$ = 2, 9 = + 1 ; +<\$ 2 ; ,<\$ = +, +, = + + + +, 14
15. ISAGA Summer School 2019 Total sum of squares of data

Decomposition of sum of squares Residual sum of sq. Se Sum of sq. b/w A SA = 0 @ = ; +<\$ 2 ; ,<\$ = +, − 9 ( = ; +<\$ 2 ; ,<\$ = +, − 9 + + 9 + − 9 ( = ; +<\$ 2 ; ,<\$ = +, − 9 + ( + 2 ; +<\$ 2 ; ,<\$ = (+, − 9 +)(9 + − 9 ) + ; +<\$ 2 9 + − 9 ( Expected value of Se ' = − 1 ' ( Expected value of SA D = ; +<\$ 2 + ( + − 1 ' ( 15
16. ISAGA Summer School 2019 Expected value of Se [ ]

( ) 2 2 2 1 1 1 1 2 1 2 1 1 1 2 ) 1 ( 1 1 e e e I i T i i I i T i i T I i T i i T I i T i i T I i T i i I i T i i I i N n i in e N I N IN tr E E E E y y E S E s s s × - = × ÷ ø ö ç è æ - = × ÷ ø ö ç è æ - = ú û ù ê ë é ÷ ø ö ç è æ - = ú ú û ù ê ê ë é ÷ ø ö ç è æ - ÷ ø ö ç è æ - = ú ú û ù ê ê ë é ÷ ø ö ç è æ - = ú ú û ù ê ê ë é ÷ ø ö ç è æ - = ú û ù ê ë é - = å å å å å å åå = = = = = = = = v v I e v v I e e v v I v v I e e v v I y v v I 16
17. ISAGA Summer School 2019 Expected value of SA [ ]

( ) 2 1 2 2 1 2 2 1 1 2 1 1 1 2 2 1 1 2 2 1 1 2 ) 1 ( 1 1 e I i i e I i i e T I i T i i I i i T I i T i i T T I i T i i T I i i T I i T i i I i i T I i T i i I i i A I a N IN N IN a N tr a N E a N E a N E y y N E S E s s s × - + = × ÷ ø ö ç è æ - + = × ÷ ø ö ç è æ - + = ú ú û ù ê ê ë é ÷ ø ö ç è æ - ÷ ø ö ç è æ - + = ú ú û ù ê ê ë é ÷ ø ö ç è æ - + = ú ú û ù ê ê ë é ÷ ø ö ç è æ - = ú û ù ê ë é - = å å å å å å å å å å å = = = = = = = = = = = vv v v e vv v v vv v v e e vv v v y vv v v 17
18. ISAGA Summer School 2019 Analysis of variance (ANOVA) Effect SS

DF Mean sq. F value A SA fA = I-1 VA = SA /(I-1) FA = VA /Ve Residual Se fe = I(N-1) Ve = Se /I(N-1) ― Total ST IN-1 ― ― Null hypothesis: Factor A has no effect on y (\$ = ( = ⋯ = 2 = 0). 0 1 2 3 4 5 Distribution of FA when null hypothesis is true A likely value under the hypothesis => hold it An unlikely value under the hypothesis => reject it 18
19. ISAGA Summer School 2019 Essence of statistical test Null hypothesis

• Typical hypothesis is that the factor has no effect on the response. Test statistic and its distribution under the null hypothesis • In ANOVA, statistic is F-value, known to follow F distribution. Realized value of the test statistic and its p-value • The smaller the p-value, more unlikely under the null hypothesis. Hold or reject the null hypothesis • Determined according to the p-value, but no universal threshold. 19
20. ISAGA Summer School 2019 Avoiding HARKing HARKing • Hypothesizing after

the results are known. • This is likely to lead to an overconfident claim, and should be avoided. Dos and don’ts • Do not turn exploratory study into confirmatory one in the middle, especially after data are obtained. • In confirmatory study, clearly specify the hypotheses and analysis methods for them before playing games. 20
21. ISAGA Summer School 2019 Agenda • Answering research questions and

data analysis • Some fictional examples of game play experiments – Case 1: Repetition – Case 2: Randomization – Case 3: Local control (Blocking) – Case 4: Designing experiments efficiently • Discussion on experimental study using games • Summary 21
22. ISAGA Summer School 2019 Case 2 A y Factor Response

RQ: Does factor A have an effect on response y? 22 A1 , A2 , … Confounder Suppose that the game is played several times in each level of A by a same player (or a team of players), but different players play in different levels. Is it an adequate experimental design?
23. ISAGA Summer School 2019 (Full) confounding A1 A2 y 23

Potential effect of factor A cannot be separately tested or evaluated from that of player. This phenomenon is called full confounding. One tool to resolve this is randomization, which translates the confounder’s effect from systematic to random error. Player 1 Player 2
24. ISAGA Summer School 2019 Fisher’s principle 2: randomization A1 A2

y 24 Potential effect of factor A cannot be separately tested or evaluated from that of player. This phenomenon is called full confounding. One tool to resolve this is randomization, which translates the confounder’s effect from systematic to random error. Player 1 Player 2 All experimental runs are conducted in a random order.
25. ISAGA Summer School 2019 Agenda • Answering research questions and

data analysis • Some fictional examples of game play experiments – Case 1: Repetition – Case 2: Randomization – Case 3: Local control (Blocking) – Case 4: Designing experiments efficiently • Discussion on experimental study using games • Summary 25
26. ISAGA Summer School 2019 Case 3 A y Factor Response

RQ: Does factor A have an effect on response y? 26 A1 , A2 , … Suppose that the game is played several times in each level of A by a same player (or a team of players), but different players play in different levels. Is it an adequate experimental design? Confounder
27. ISAGA Summer School 2019 Fisher’s principle 3: local control (blocking)

A1 A2 y 27 Another tool to deal with a confounding variable is local control or blocking, which treats the confounding variable as a factor and assigns it to the experimental design. This makes it possible to separate the effect of interested factor from that of the confounder. Player 1 Player 2 Player 3 Experimental runs of each player are conducted in a random order.
28. ISAGA Summer School 2019 Randomized block design R1 R2 RN

A1 y11 y12 y1N A2 y21 y22 y2N AI yI1 yI2 yIN 28
29. ISAGA Summer School 2019 Model equation +, = + +

+ , + +, Assumptions – Main effects satisfy: ， – Interaction between block and other factors can be ignored. – Error terms follow: Model for randomized block design General mean Factor A’s Block R’s Error main effect main effect ~(, ' ( 7 ) ~(, G ( 7 ) ; +<\$ 2 + = 0 29
30. ISAGA Summer School 2019 Total sum of squares of data

Decomposition of sum of squares SS b/w A: SA SS b/w R: SR Residual SS: Se @ = DG = ; +<\$ 2 9 +7 − 9 ( + ; ,<\$ = 9 7, − 9 ( + ; +<\$ 2 ; ,<\$ = +, − 9 +7 − 9 7, + 9 ( D = ; +<\$ 2 + ( + − 1 ' ( G = − 1 G ( + − 1 ' ( ' = ( − 1) − 1 ' ( 30
31. ISAGA Summer School 2019 Analysis of variance (ANOVA) Effect SS

DF Mean sq. F value A SA fA = I-1 VA = SA /fA FA = VA /Ve R SR fR = N-1 VR = SR /fR FR = VR /Ve Residual Se fe = (I-1)(N-1) Ve = Se /fe ― Total ST IN-1 ― ― 31
32. ISAGA Summer School 2019 Randomization vs. local control (blocking) •

Randomization is simple and easy, but enlarges error variance and thus degrades the power of statistical test. • Blocking gives you more detailed result, but tends to require more experimental runs. • Blocking is only applicable to known confounders, and they need to be controllable or at least observable. • Blocking is restricted by the number of units available to each block. • Even when blocking is used, randomization within each block is recommended. 32 Day 1 Day 2 Day 3 Full randomization A2 , A3 , A3 A1 , A1 , A2 A1 , A3 , A2 Randomized block A3 , A1 , A2 A1 , A3 , A2 A1 , A2 , A3
33. ISAGA Summer School 2019 Agenda • Answering research questions and

data analysis • Some fictional examples of game play experiments – Case 1: Repetition – Case 2: Randomization – Case 3: Local control (Blocking) – Case 4: Designing experiments efficiently • Discussion on experimental study using games • Summary 33
34. ISAGA Summer School 2019 Case 4 A y Factors Response

RQ: Do factors A, B, … have effects on response y? 34 A1 , A2 , … Confounders B C B1 , B2 , … C1 , C2 , …
35. ISAGA Summer School 2019 Factorial design & combinatorial explosion Factorial

design • The most basic experimental design, when more than two factors are taken up. • Multiple experimental runs are conducted in every possible combination of the levels of factors. • Total number of runs: × S 'T'UV WXYZ[U => Combinatorial explosion 35
36. ISAGA Summer School 2019 Number of experiments & degree of

freedom 2 4 6 8 10 0 500 1000 1500 2000 因子数 実験回数 主効果の 自由度 交互作用の 自由度 残差の 自由度 2 level factors 3level factors 2 4 6 8 10 0 500 1000 1500 2000 因子数 実験回数 主効果の 自由度 交互作用の 自由度 残差の 自由度 36
37. ISAGA Summer School 2019 Main effects & interactions #1 A1

A2 Main effect of factor A A1 A2 Main effect of factor B B2 B1 37 y
38. ISAGA Summer School 2019 Main effects & interactions #2 y

A1 A2 B2 Only main effects of A and B B1 A1 A2 B2 Main effects and interaction A×B B1 38
39. ISAGA Summer School 2019 Fractional factorial design • The number

of necessary experimental runs is drastically reduced, by ignoring higher order interactions. • Only a fraction of factorial design needs to be carried out. • How to construct such a design is a bit mathematical. • Orthogonal arrays are useful tool for constructing the design. • Computer support (ex. R packages) is also available. Fractional factorial design & orthogonal arrays 39
40. ISAGA Summer School 2019 Agenda • Answering research questions and

data analysis • Some fictional examples of game play experiments – Case 1: Repetition – Case 2: Randomization – Case 3: Local control (Blocking) – Case 4: Designing experiments efficiently • Discussion on experimental study using games • Summary 40
41. ISAGA Summer School 2019 Common discussion • Are findings obtained

by using games applicable to the reality? • What to measure and how? • Are statistical methods suitable/effective for analyzing game data? – Games often seem to involve chaotic (irreproducible) behaviors. – Game data tend to have large variances. – There may be too many outliers. – Distribution of the data does not seem to be stable. 41 Reluctantly yes. Since no perfect alternative exists, we cannot help but need to rely on it. We should use it with care.
42. ISAGA Summer School 2019 Agenda • Answering research questions and

data analysis • Some fictional examples of game play experiments – Case 1: Repetition – Case 2: Randomization – Case 3: Local control (Blocking) – Case 4: Designing experiments efficiently • Discussion on experimental study using games • Summary 42
43. ISAGA Summer School 2019 Summary • It is important to

specify the characteristic of study and design experiments appropriately before playing games. • We should design experiments according to Fisher’s principles; 1. Repetition makes it possible to evaluate the magnitude of error variance. 2. Randomization will translate the potential effects of confounders from systematic error to random error. 3. Local control (blocking) makes it possible to separate the effects of interested factors from those of confounders. • Design of experiments (DOE) techniques will help enhance the efficiency of experiments. 43
44. ISAGA Summer School 2019 What next? • (Revisit) statistical testing

and estimation • Fractional factorial designs, and orthogonal arrays • Linear regression, and response surface methodology • Nonparametric statistical methods • Multivariate data analysis • Data visualization, and exploratory analysis • Machine learning (supervised and unsupervised learning), etc. 44

welcome