Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Software Analytics for Digital Games

Thomas Zimmermann
May 18, 2015
4.1k

Software Analytics for Digital Games

Keynote presented at the Fourth International Workshop on Games and Software Engineering (GAS 2015).

http://2015.gasworkshop.org/

Thomas Zimmermann

May 18, 2015
Tweet

Transcript

  1. © Microsoft Corporation Software Analytics for Digital Games Thomas Zimmermann,

    Microsoft Research, USA Joint work with Nachi Nagappan and many others. http://aka.ms/esegroup
  2. © Microsoft Corporation history of software analytics Tim Menzies, Thomas

    Zimmermann: Software Analytics: So What? IEEE Software 30(4): 31-37 (2013)
  3. © Microsoft Corporation trinity of software analytics Dongmei Zhang, Shi

    Han, Yingnong Dang, Jian-Guang Lou, Haidong Zhang, Tao Xie: Software Analytics in Practice. IEEE Software 30(5): 30-37, September/October 2013. MSR Asia Software Analytics group: http://research.microsoft.com/en-us/groups/sa/
  4. © Microsoft Corporation http://aka.ms/145Questions Andrew Begel, Thomas Zimmermann. Analyze This!

    145 Questions for Data Scientists in Software Engineering. ICSE 2014
  5. © Microsoft Corporation Microsoft’s Top 10 Questions Essential Essential +

    Worthwhile How do users typically use my application? 80.0% 99.2% What parts of a software product are most used and/or loved by customers? 72.0% 98.5% How effective are the quality gates we run at checkin? 62.4% 96.6% How can we improve collaboration and sharing between teams? 54.5% 96.4% What are the best key performance indicators (KPIs) for monitoring services? 53.2% 93.6% What is the impact of a code change or requirements change to the project and its tests? 52.1% 94.0% What is the impact of tools on productivity? 50.5% 97.2% How do I avoid reinventing the wheel by sharing and/or searching for code? 50.0% 90.9% What are the common patterns of execution in my application? 48.7% 96.6% How well does test coverage correspond to actual code usage by our customers? 48.7% 92.0%
  6. © Microsoft Corporation Xbox Live Influence of games and achievements

    on (paid) Xbox live memberships Influence of friends on titles played Characterizing players with Xbox Live data Gameplay Game play dynamics (Project Gotham Racing 4) => ESEM 2012 Impact of social behavior on retention (Beta of a AAA title) Influence of gameplay on skill (Halo Reach) => CHI 2013 Assists in a car racing game (Forza 4) => FDG 2014 How to create a successful initial session in games => CHI Play 2014 Engineering Differences between game and traditional software development => ICSE 2014 Lessons learned from game development (ongoing) Mining software repositories from games (ongoing) Exploratory Personalization with Avatars in Xbox Geographic influence, temporal influence, and structural influence Thanks to our collaborators in Xbox and Microsoft Games Studios. Thanks to interns Ken Hullett, Sauvik Das, Jeff Huang, Gifford Cheung, Thomas Debeauvais, Erik Harpstead and visiting researchers Tim Menzies and Emerson Murphy-Hill.
  7. © Microsoft Corporation Skill in Halo Reach Jeff Huang, Thomas

    Zimmermann, Nachiappan Nagappan, Charles Harrison, Bruce C. Phillips: Mastering the art of war: how patterns of gameplay influence skill in Halo. CHI 2013: 695-704
  8. How do patterns of play affect players’ skill in Halo

    Reach? 5 Skill and Other Titles 6 Skill Changes and Retention 7 Mastery and Demographics 8 Predicting Skill 2 Play Intensity 3 Skill after Breaks 4 Skill before Breaks 1 General Statistics
  9. The Cohort of Players The mean skill value µ for

    each player after each Team Slayer match µ ranges between 0 and 10, although 50% fall between 2.5 and 3.5 Initially µ = 3 for each player, stabilizing after a couple dozen matches TrueSkill in Team Slayer We looked at the cohort of players who started in the release week with complete set of gameplay for those players up to 7 months later (over 3 million players) 70 Person Survey about Player Experience
  10. 2.1 2.3 2.5 2.7 2.9 3.1 0 10 20 30

    40 50 60 70 80 90 100 mu Games Played So Far 2 Play Intensity Median skill typically increases slowly over time Skill
  11. 2 Play Intensity (Games per Week) 2.1 2.3 2.5 2.7

    2.9 3.1 0 10 20 30 40 50 60 70 80 90 100 mu Games Played So Far 0 - 2 games / week [N=59164] 2 - 4 games / week [N=101448] 4 - 8 games / week [N=226161] 8 - 16 games / week [N=363832] 16 - 32 games / week [N=319579] 32 - 64 games / week [N=420258] 64 - 128 games / week [N=415793] 128 - 256 games / week [N=245725] Median skill typically increases slowly over time Skill
  12. 2 Play Intensity (Games per Week) 2.1 2.3 2.5 2.7

    2.9 3.1 0 10 20 30 40 50 60 70 80 90 100 mu Games Played So Far 0 - 2 games / week [N=59164] 2 - 4 games / week [N=101448] 4 - 8 games / week [N=226161] 8 - 16 games / week [N=363832] 16 - 32 games / week [N=319579] 32 - 64 games / week [N=420258] 64 - 128 games / week [N=415793] 128 - 256 games / week [N=245725] Median skill typically increases slowly over time Skill
  13. 2 Play Intensity (Games per Week) 2.1 2.3 2.5 2.7

    2.9 3.1 0 10 20 30 40 50 60 70 80 90 100 mu Games Played So Far 0 - 2 games / week [N=59164] 2 - 4 games / week [N=101448] 4 - 8 games / week [N=226161] 8 - 16 games / week [N=363832] 16 - 32 games / week [N=319579] 32 - 64 games / week [N=420258] 64 - 128 games / week [N=415793] 128 - 256 games / week [N=245725] But players who play more overall eventually surpass those who play 4–8 games per week (not shown in chart) Players who play 4–8 games per week do best Median skill typically increases slowly over time Skill
  14. 3 Change in Skill Following a Break “In the most

    drastic scenario, you can lose up to 80 percent of your fitness level in as few as two weeks [of taking a break]…”
  15. -0.03 -0.02 -0.01 0 0.01 0.02 0.03 0 5 10

    15 20 25 30 35 40 45 50 Δmu Days of Break Next Game 2 Games Later 3 Games Later 4 Games Later 5 games later 10 games later 3 Change in Skill Following a Break Median skill slightly increases after each game played without breaks Longer breaks correlate with larger skill drops, but not linearly On average, it takes 8–10 games to regain skill lost after 30 day breaks Breaks of 1–2 days correlate in tiny drops in skill Change in Skill
  16. © Microsoft Corporation Driving skill in Forza Motorsports 4 Thomas

    Debeauvais, Thomas Zimmermann, Nachiappan Nagappan, Kevin Carter, Ryan Cooper, Dan Greenawalt, Tyson Solberg: An Empirical Study of Driving Skill in Forza Motorsports 4. FDG 2014
  17. © Microsoft Corporation Driving skill in Forza Motorsports 4 5%

    of player base, sampled randomly 200k players who played 25M races Assist usage Assist transitions
  18. © Microsoft Corporation The assist bundles in Forza 4 Easy

    Medium Hard Advanced Expert Stability prevents the car from spinning when cornering too fast ON OFF Traction prevents the car from spinning when accelerating ON OFF Braking supports the player when he/she brakes or should brake Assisted w/ ABS ABS OFF Shifting helps the player in passing gears Automatic w/o clutch Manual w/o clutch Manual w/ clutch Line overlays the optimal trajectory to follow on the track Full Brake OFF Damage determines how much the performance of the car can change during the race Cosmetic Limited Simulation
  19. © Microsoft Corporation number of races Assist usage over number

    of races career mode online multiplayer number of races
  20. © Microsoft Corporation 0% 10% 20% 30% 40% 50% 60%

    70% 80% 90% 100% success failure yoyo never disabled Assist transitions
  21. © Microsoft Corporation Factors that contribute to the success of

    disabling an assist Factor More likely to keep the assist disabled … Significant for … Number of races Players who disable an assist early All assists Races per day Players who race fewer games a day All assists Rear-wheel drive (race before) Players who drove a car with rear-wheel drive All assists Car Performance Index (race before) Players who drove a car with lower PI All assists Position (race before) Players who finished first All assists but Traction and Clutch Career mode (race before) Players who did not play career mode Autobrake, ABS, Autoshift, Full line, Brake line
  22. © Microsoft Corporation The First Hour Experience Gifford Cheung, Thomas

    Zimmermann, Nachiappan Nagappan: The First Hour Experience: How the Initial Play can Engage (or Lose) New Players. CHI Play 2014
  23. © Microsoft Corporation the “initial experience” the “first time user

    experience” “tutorial time” the “5-5-5” (five seconds, minutes, hours) “the first 10 seconds, the first minute, the first ten minutes, the first hour” “setting the hook” also known as
  24. © Microsoft Corporation “If your game is abandoned in the

    first hour that’s a pretty low moment. It’s a complete rejection of the game play in all ways, you know, game play, narrative, style. … It’s a pretty catastrophic failure.” Designer C a Critical Hour
  25. © Microsoft Corporation The Data 212 Amazon reviews of Xbox

    360 games Searching for keywords such as “first”, “hour”, “minute”, “start”
  26. © Microsoft Corporation The Data 212 Amazon reviews of Xbox

    360 games Searching for keywords such as “first”, “hour”, “minute”, “start” 35 First hour reviews
  27. © Microsoft Corporation The Data 212 Amazon reviews of Xbox

    360 games Searching for keywords such as “first”, “hour”, “minute”, “start” 35 First hour reviews 6 Interviews with Game Industry Professionals (designer, tester, user researcher)
  28. © Microsoft Corporation Methodology Grounded Theory Open, Axial, Selective Analysis

    (Amazon + First hour reviews) Invited an auditor to review the process and emergent categories for comprehensibility and closeness to data. Used the interviews in part to check the findings (Selective).
  29. © Microsoft Corporation Findings: The First Experience Arc Expectations Specific

    Anticipations Experiences Momentary & Engaging Annoying, Frustrating, or Tedious Outcome Abandonment Continuing Exceptions Deal-breakers Holdouts
  30. © Microsoft Corporation Deal-breakers: Frustrations that supersede all positives. “An

    UNSKIPPABLE cut scene. Annoying. So, we continue playing and now there are UNSKIPPABLE cut scenes every few plays of the players taunting each other before the ball is hiked! Really? I needed to see his defenseman saying something foul to my running back RIGHT NOW! NO, I don't! We made it through the rest of the game some how and decided that it was one of the worst playing experiences of any game we've ever played. Reselling this ASAP! BTW, the football game play was actually pretty good. Running, throwing and fielding all seemed to work well. In fact, I would say that it worked better than Madden. Its such a shame that they made the rest of the game so unplayable.” (Blitz: The League, Amazon Review)
  31. © Microsoft Corporation Holdouts: Playing a game you hate Anticipated

    Elements “I can't stop before meeting some aliens, right?” (Crysis 2) A good game mechanic “Would I keep playing? I guess so. I really do like the combat system, which captures Captain America's acrobatic grace and power quite well, but the limited enemy variety is allowing that repetitious feeling to sink in. … I'll soldier on for a bit, but I don't expect to finish this fight.” (First hour review of Captain America: Super Soldier) Narrative Completionists (external factor) Social reasons (external factor)
  32. © Microsoft Corporation Game development Emerson R. Murphy-Hill, Thomas Zimmermann,

    Nachiappan Nagappan: Cowboys, ankle sprains, and keepers of quality: how is video game development different from software development? ICSE 2014
  33. © Microsoft Corporation Methodology: Interviews • Drew 14 participants from

    LinkedIn 2+ years dev experience in games AND 2+ years dev experience outside of games • Opened with a general question about “what’s the difference between games and other types of software?” • Then focused on SWEBOK and Work topics
  34. © Microsoft Corporation Methodology: Surveys • Created 84 candidate statements

    based on interviews, narrowed down to 28 • Randomly sampled three populations, within Microsoft: – 300 engineers who work on games – 300 engineers who work on Microsoft Office – 300 engineers who work elsewhere
  35. © Microsoft Corporation Significant Differences Game developers have less clear

    requirements than non-game developers. Game developers tend to use what they perceive as an Agile process more than non-game developers. Creativity is valued more in game development teams. The ability to communicate with non-engineers is valued more on game development teams. Game development requires a more diverse team. People are more impressed by game developers’ work.
  36. © Microsoft Corporation Software Analytics for Digital Games Thomas Zimmermann,

    Microsoft Research, USA Joint work with Nachi Nagappan and many others. http://aka.ms/esegroup