Sergio Estupiñán Ph.D. & Key Results (June 2020)

TECFA's team seminar series.
15.06.2020

Transcript

1. Fine-grained evaluation of the Interactive Narrative Experience: A Continuation Desire perspective Photo credits: ”Impressionism" by ハング (Digital, 2015). Sergio Estupiñán University of Geneva Switzerland Sergio.Estupinan@unige.ch Report Sergio’s Ph.D & Key Results June 2020
2. Agenda Photo credits: ”Impressionism" by ハング (Digital, 2015). Fine-grained Evaluation of the Interactive Narrative Experience | Sergio Estupinan Sergio’s Ph.D. & Key Results Report June 2020 1/14 Context Problem Research Questions Reconsidering the INE Instruments & Studies Key Results
3. Interactive Digital Narrative Research projects @ TECFA Fine-grained Evaluation of the Interactive Narrative Experience | Sergio Estupinan Sergio’s Ph.D. & Key Results Report June 2020 2/14 FELINE 2016-2019 http://tecfa.unige.ch/tecfa/research/feline/ TBI-SIM 2010-2013 http://nothingfordinner.org Brought to you by Nicolas Szilas & SNSF Context
4. The Interactive Narrative Experience Fine-grained Evaluation of the Interactive Narrative Experience | Sergio Estupinan Sergio’s Ph.D. & Key Results Report June 2020 3/14 Aylett, R., & Louchart, S. (2007) Questionnaires (IRIS, Media Studies) Interaction Start of playtesting Interactive Digital Narrative System t 0 t f Interactive Narrative Experience “It is relevant to know which part of a story elicits certain experiences” Roth (2015) Problem
5. Sampling and Evaluating Engagement & Narrative Perception Fine-grained Evaluation of the Interactive Narrative Experience | Sergio Estupinan Sergio’s Ph.D. & Key Results Report June 2020 4/14 Research Questions [RQ1] Can engagement levels and narrative perception be sampled during runtime without spoiling the Interactive Narrative Experience? [RQ2] Can a relation be established between patterns of user-initiated narrative acts and engagement levels sampled during runtime? [RQ3] Can runtime- collected affective data be used to explain engagement states in an Interactive Narrative? Continuation Desire (Schoenau-Fog, 2011)
6. Advancing the understanding of the Interactive Narrative Experience Fine-grained Evaluation of the Interactive Narrative Experience | Sergio Estupinan Sergio’s Ph.D. & Key Results Report June 2020 5/14 Reconsidering the INE INE as a process Continuation Desire Consistency Intention – Action Engagement Trajectories Consolidating Affect Re-playability
7. User Testing Study Design Fine-grained Evaluation of the Interactive Narrative Experience | Sergio Estupinan Sergio’s Ph.D. & Key Results Report June 2020 6/14 Instruments & Studies Study 1: Crowdsourced (online); N=130 Control Group N=40 • No interruptions. Interruption Group N=90 • Interruptions (x3). Pre: Gaming frequency, Initial CD, SAM guidance Post: Re-playability, Effectance, SUS, Remarks. Post: Did interruptions spoil? (2Qs: Likert, SEQ) Study 2: Biometrics (onsite); N=30 Control Group N=16 • Facial expressions via webcam. • E4 Bracelet. • No interruptions. Interruption Group N=14 • Facial expressions via webcam. • E4 bracelet. • Interruptions (x3). Pre: Gaming frequency, Initial CD, SAM guidance Post: Re-playability, Effectance, SUS, Remarks. Post: Did interruptions spoil? (2Qs: Likert, SEQ) Between-subjects design
8. Interruption triggering system Sampling without spoiling Fine-grained Evaluation of the Interactive Narrative Experience | Sergio Estupinan Sergio’s Ph.D. & Key Results Report June 2020 7/14 Key Results Disruptiveness [Interruption group] Neutral opinion (4.11 AVG, SD=1.61) (1 = Strongly disagree, 7 = Strongly agree) Ease of use (SEQ) [Interruption group] Moderately easy (5.79 AVG) (1= Very difficult, 7 = Very easy) Replayability desire Control group: Disagree a little (3.07 AVG) Interruption group Disagree a little (2.82 AVG) No significant difference. An independent two-tailed t-test was performed, finding no significant difference (p = 0.536). Calculated Cohen’s d is 0.1362, effect size is very small. [RQ1] Can engagement levels and narrative perception be sampled during runtime without spoiling the Interactive Narrative Experience?
9. Process Mining for looking into Engagemement Trajectories Fine-grained Evaluation of the Interactive Narrative Experience | Sergio Estupinan Sergio’s Ph.D. & Key Results Report June 2020 8/14 Key Results Process Mining seems to be highly valuable in discovering the diverse paths in which users interact with the system. Moreover, it was useful for spotting the elements of narrative paths linked to elevated engagement (hooked trajectory), which in the case of the IDN tested seemed to be mostly linked to seeking the involvement of other characters in the achievement of certain activities [RQ2] Can a relation be established between patterns of user-initiated narrative acts and engagement levels sampled during runtime?
10. Affect matters!? SAM &Continuation Desire Fine-grained Evaluation of the Interactive Narrative Experience | Sergio Estupinan Sergio’s Ph.D. & Key Results Report June 2020 9/14 Key Results [RQ3] Can runtime-collected affective data be used to explain engagement states in an Interactive Narrative? Study 1, Interruption group (N=90) Valence Arousal Dominance > s1_samCD_multipleRegression <- lm(s1_cd123 ~ s1_p123 + s1_a123 + s1_d123) > summary(s1_samCD_multipleRegression) Call: lm(formula = s1_cd123 ~ s1_p123 + s1_a123 + s1_d123) Residuals: Min 1Q Median 3Q Max -3.7190 -1.0762 -0.0872 1.1436 4.0681 Coefficients: Estimate Std. Error t value Pr(>|t|) (Intercept) 6.42077 0.52682 12.188 < 2e-16 *** s1_p123 -0.84857 0.09563 -8.873 < 2e-16 *** s1_a123 -0.16481 0.09097 -1.812 0.0711 . s1_d123 0.36121 0.06588 5.483 9.71e-08 *** --- Signif. codes: 0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1 Residual standard error: 1.555 on 266 degrees of freedom Multiple R-squared: 0.3847, Adjusted R-squared: 0.3777 F-statistic: 55.43 on 3 and 266 DF, p-value: < 2.2e-16 Study 2, Interruption group (N=14) > s2_samCD_multipleRegression <- lm(s2_cd_all ~ s2_sam_p + s2_sam_a + s2_sam_d) > summary(s2_samCD_multipleRegression) Call: lm(formula = s2_cd_all ~ s2_sam_p + s2_sam_a + s2_sam_d) Residuals: Min 1Q Median 3Q Max -1.99366 -0.48258 0.06761 0.53920 1.54648 Coefficients: Estimate Std. Error t value Pr(>|t|) (Intercept) 8.0639 0.7179 11.233 4.34e-14 *** s2_sam_p -0.4059 0.1429 -2.841 0.00698 ** s2_sam_a -0.1343 0.1254 -1.071 0.29052 s2_sam_d -0.1052 0.1496 -0.703 0.48594 --- Signif. codes: 0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1 Residual standard error: 0.8058 on 41 degrees of freedom Multiple R-squared: 0.2088, Adjusted R-squared: 0.1509 F-statistic: 3.606 on 3 and 41 DF, p-value: 0.02115
11. Affect matters!? SAM &Continuation Desire Fine-grained Evaluation of the Interactive Narrative Experience | Sergio Estupinan Sergio’s Ph.D. & Key Results Report June 2020 10/14 Key Results [RQ3] Can runtime-collected affective data be used to explain engagement states in an Interactive Narrative? Study 1, Interruption group (N=90) Valence Arousal Dominance > s1_samCD_multipleRegression <- lm(s1_cd123 ~ s1_p123 + s1_a123 + s1_d123) > summary(s1_samCD_multipleRegression) Call: lm(formula = s1_cd123 ~ s1_p123 + s1_a123 + s1_d123) Residuals: Min 1Q Median 3Q Max -3.7190 -1.0762 -0.0872 1.1436 4.0681 Coefficients: Estimate Std. Error t value Pr(>|t|) (Intercept) 6.42077 0.52682 12.188 < 2e-16 *** s1_p123 -0.84857 0.09563 -8.873 < 2e-16 *** s1_a123 -0.16481 0.09097 -1.812 0.0711 . s1_d123 0.36121 0.06588 5.483 9.71e-08 *** --- Signif. codes: 0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1 Residual standard error: 1.555 on 266 degrees of freedom Multiple R-squared: 0.3847, Adjusted R-squared: 0.3777 F-statistic: 55.43 on 3 and 266 DF, p-value: < 2.2e-16 Study 2, Interruption group (N=14) > s2_samCD_multipleRegression <- lm(s2_cd_all ~ s2_sam_p + s2_sam_a + s2_sam_d) > summary(s2_samCD_multipleRegression) Call: lm(formula = s2_cd_all ~ s2_sam_p + s2_sam_a + s2_sam_d) Residuals: Min 1Q Median 3Q Max -1.99366 -0.48258 0.06761 0.53920 1.54648 Coefficients: Estimate Std. Error t value Pr(>|t|) (Intercept) 8.0639 0.7179 11.233 4.34e-14 *** s2_sam_p -0.4059 0.1429 -2.841 0.00698 ** s2_sam_a -0.1343 0.1254 -1.071 0.29052 s2_sam_d -0.1052 0.1496 -0.703 0.48594 --- Signif. codes: 0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1 Residual standard error: 0.8058 on 41 degrees of freedom Multiple R-squared: 0.2088, Adjusted R-squared: 0.1509 F-statistic: 3.606 on 3 and 41 DF, p-value: 0.02115 How to interpret this correctly?
12. Affect matters!? Facial expressions & CD (set 1) Fine-grained Evaluation of the Interactive Narrative Experience | Sergio Estupinan Sergio’s Ph.D. & Key Results Report June 2020 11/14 Key Results [RQ3] Can runtime-collected affective data be used to explain engagement states in an Interactive Narrative? How do I interpret this correctly? > s2_bioCD_multipleRegression <- lm(allPartic_s2_bio_CD123 ~ allPartic_s2_bio_int123_joy + allPartic_s2_bio_int123_valence + allPartic_s2_bio_int123_engagement) > summary(s2_bioCD_multipleRegression) Call: lm(formula = allPartic_s2_bio_CD123 ~ allPartic_s2_bio_int123_joy + allPartic_s2_bio_int123_valence + allPartic_s2_bio_int123_engagement) Residuals: Min 1Q Median 3Q Max -2.2872 -0.2303 0.1471 0.6805 1.0784 Coefficients: Estimate Std. Error t value Pr(>|t|) (Intercept) 6.16050 0.19853 31.030 <2e-16 *** allPartic_s2_bio_int123_joy 0.11990 0.07343 1.633 0.111 allPartic_s2_bio_int123_valence -0.04818 0.04112 -1.172 0.249 allPartic_s2_bio_int123_engagement -0.02841 0.02366 -1.201 0.237 --- Signif. codes: 0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1 Residual standard error: 0.8845 on 38 degrees of freedom Multiple R-squared: 0.08721, Adjusted R-squared: 0.01515 F-statistic: 1.21 on 3 and 38 DF, p-value: 0.3192 Joy, Sadness, Anger, Valence, Engagement Joy correlates weakly with CD, but regression analysis indicates elevated p-value and very low R- squared. What could I do with this?
13. Affect matters!? Facial expressions & CD (set 2) Fine-grained Evaluation of the Interactive Narrative Experience | Sergio Estupinan Sergio’s Ph.D. & Key Results Report June 2020 12/14 Key Results [RQ3] Can runtime-collected affective data be used to explain engagement states in an Interactive Narrative? Do I interpret this correctly? > s2_bioCD_multipleRegression_Frustr <- lm(allPartic_s2_bio_int123_CD ~ allPartic_s2_bio_int123_Disgust + allPartic_s2_bio_int123_Smirk + allPartic_s2_bio_int123_LipPress) > summary(s2_bioCD_multipleRegression_Frustr) Call: lm(formula = allPartic_s2_bio_int123_CD ~ allPartic_s2_bio_int123_Disgust + allPartic_s2_bio_int123_Smirk + allPartic_s2_bio_int123_LipPress) Residuals: Min 1Q Median 3Q Max -2.4392 -0.3994 0.1259 0.5852 1.1126 Coefficients: Estimate Std. Error t value Pr(>|t|) (Intercept) 6.79291 0.24149 28.129 <2e-16 *** allPartic_s2_bio_int123_Disgust -0.76216 0.38330 -1.988 0.0540 . allPartic_s2_bio_int123_Smirk -0.04667 0.02679 -1.742 0.0896 . allPartic_s2_bio_int123_LipPress 0.07157 0.03355 2.133 0.0394 * --- Signif. codes: 0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1 Residual standard error: 0.7969 on 38 degrees of freedom Multiple R-squared: 0.259, Adjusted R-squared: 0.2005 F-statistic: 4.428 on 3 and 38 DF, p-value: 0.00915 this is the Standard Deviation of the residuals Brow Furrow, Lip Press, Smirk, Disgust Together, facial expressions Disgust, Smirk, Lip Press could explain about 20% of the variability in Continuation Desire.
14. Affect matters!? EDA & CD Fine-grained Evaluation of the Interactive Narrative Experience | Sergio Estupinan Sergio’s Ph.D. & Key Results Report June 2020 13/14 Key Results [RQ3] Can runtime-collected affective data be used to explain engagement states in an Interactive Narrative? How do I interpret this correctly? • Tonic EDA tendency to gradually naturally increase – probably not good to analyze this dimension? • Phasic EDA basically shows no correlation.
15. Fine-grained evaluation of the Interactive Narrative Experience: A Continuation Desire perspective Photo credits: ”Impressionism" by ハング (Digital, 2015). Sergio Estupiñán University of Geneva Switzerland Sergio.Estupinan@unige.ch Report Sergio’s Ph.D & Key Results June 2020 Thanks for your attention! à Q&A time