Software Analytics for Digital Games

Slide 1

Slide 1 text

Slide 2

Slide 2 text

Slide 3

Slide 3 text

Slide 4

Slide 4 text

Slide 5

Slide 5 text

Slide 6

Slide 6 text

© Microsoft Corporation trinity of software analytics Dongmei Zhang, Shi Han, Yingnong Dang, Jian-Guang Lou, Haidong Zhang, Tao Xie: Software Analytics in Practice. IEEE Software 30(5): 30-37, September/October 2013. MSR Asia Software Analytics group: http://research.microsoft.com/en-us/groups/sa/

Slide 7

Slide 7 text

Slide 8

Slide 8 text

Slide 9

Slide 9 text

Slide 10

Slide 10 text

Slide 11

Slide 11 text

© Microsoft Corporation Microsoft’s Top 10 Questions Essential Essential + Worthwhile How do users typically use my application? 80.0% 99.2% What parts of a software product are most used and/or loved by customers? 72.0% 98.5% How effective are the quality gates we run at checkin? 62.4% 96.6% How can we improve collaboration and sharing between teams? 54.5% 96.4% What are the best key performance indicators (KPIs) for monitoring services? 53.2% 93.6% What is the impact of a code change or requirements change to the project and its tests? 52.1% 94.0% What is the impact of tools on productivity? 50.5% 97.2% How do I avoid reinventing the wheel by sharing and/or searching for code? 50.0% 90.9% What are the common patterns of execution in my application? 48.7% 96.6% How well does test coverage correspond to actual code usage by our customers? 48.7% 92.0%

Slide 12

Slide 12 text

Slide 13

Slide 13 text

Slide 14

Slide 14 text

Slide 15

Slide 15 text

Slide 16

Slide 16 text

Slide 17

Slide 17 text

© Microsoft Corporation Xbox Live Influence of games and achievements on (paid) Xbox live memberships Influence of friends on titles played Characterizing players with Xbox Live data Gameplay Game play dynamics (Project Gotham Racing 4) => ESEM 2012 Impact of social behavior on retention (Beta of a AAA title) Influence of gameplay on skill (Halo Reach) => CHI 2013 Assists in a car racing game (Forza 4) => FDG 2014 How to create a successful initial session in games => CHI Play 2014 Engineering Differences between game and traditional software development => ICSE 2014 Lessons learned from game development (ongoing) Mining software repositories from games (ongoing) Exploratory Personalization with Avatars in Xbox Geographic influence, temporal influence, and structural influence Thanks to our collaborators in Xbox and Microsoft Games Studios. Thanks to interns Ken Hullett, Sauvik Das, Jeff Huang, Gifford Cheung, Thomas Debeauvais, Erik Harpstead and visiting researchers Tim Menzies and Emerson Murphy-Hill.

Slide 18

Slide 18 text

© Microsoft Corporation Skill in Halo Reach Jeff Huang, Thomas Zimmermann, Nachiappan Nagappan, Charles Harrison, Bruce C. Phillips: Mastering the art of war: how patterns of gameplay influence skill in Halo. CHI 2013: 695-704

Slide 19

Slide 19 text

Slide 20

Slide 20 text

How do patterns of play affect players’ skill in Halo Reach? 5 Skill and Other Titles 6 Skill Changes and Retention 7 Mastery and Demographics 8 Predicting Skill 2 Play Intensity 3 Skill after Breaks 4 Skill before Breaks 1 General Statistics

Slide 21

Slide 21 text

The Cohort of Players The mean skill value µ for each player after each Team Slayer match µ ranges between 0 and 10, although 50% fall between 2.5 and 3.5 Initially µ = 3 for each player, stabilizing after a couple dozen matches TrueSkill in Team Slayer We looked at the cohort of players who started in the release week with complete set of gameplay for those players up to 7 months later (over 3 million players) 70 Person Survey about Player Experience

Slide 22

Slide 22 text

2 Play Intensity Telegraph operators gradually increase typing speed over time

Slide 23

Slide 23 text

2.1 2.3 2.5 2.7 2.9 3.1 0 10 20 30 40 50 60 70 80 90 100 mu Games Played So Far 2 Play Intensity Median skill typically increases slowly over time Skill

Slide 24

Slide 24 text

2 Play Intensity (Games per Week) 2.1 2.3 2.5 2.7 2.9 3.1 0 10 20 30 40 50 60 70 80 90 100 mu Games Played So Far 0 - 2 games / week [N=59164] 2 - 4 games / week [N=101448] 4 - 8 games / week [N=226161] 8 - 16 games / week [N=363832] 16 - 32 games / week [N=319579] 32 - 64 games / week [N=420258] 64 - 128 games / week [N=415793] 128 - 256 games / week [N=245725] Median skill typically increases slowly over time Skill

Slide 25

Slide 25 text

Slide 26

Slide 26 text

2 Play Intensity (Games per Week) 2.1 2.3 2.5 2.7 2.9 3.1 0 10 20 30 40 50 60 70 80 90 100 mu Games Played So Far 0 - 2 games / week [N=59164] 2 - 4 games / week [N=101448] 4 - 8 games / week [N=226161] 8 - 16 games / week [N=363832] 16 - 32 games / week [N=319579] 32 - 64 games / week [N=420258] 64 - 128 games / week [N=415793] 128 - 256 games / week [N=245725] But players who play more overall eventually surpass those who play 4–8 games per week (not shown in chart) Players who play 4–8 games per week do best Median skill typically increases slowly over time Skill

Slide 27

Slide 27 text

3 Change in Skill Following a Break “In the most drastic scenario, you can lose up to 80 percent of your fitness level in as few as two weeks [of taking a break]…”

Slide 28

Slide 28 text

-0.03 -0.02 -0.01 0 0.01 0.02 0.03 0 5 10 15 20 25 30 35 40 45 50 Δmu Days of Break Next Game 2 Games Later 3 Games Later 4 Games Later 5 games later 10 games later 3 Change in Skill Following a Break Median skill slightly increases after each game played without breaks Longer breaks correlate with larger skill drops, but not linearly On average, it takes 8–10 games to regain skill lost after 30 day breaks Breaks of 1–2 days correlate in tiny drops in skill Change in Skill

Slide 29

Slide 29 text

© Microsoft Corporation Driving skill in Forza Motorsports 4 Thomas Debeauvais, Thomas Zimmermann, Nachiappan Nagappan, Kevin Carter, Ryan Cooper, Dan Greenawalt, Tyson Solberg: An Empirical Study of Driving Skill in Forza Motorsports 4. FDG 2014

Slide 30

Slide 30 text

Slide 31

Slide 31 text

Slide 32

Slide 32 text

Slide 33

Slide 33 text

Slide 34

Slide 34 text

© Microsoft Corporation The assist bundles in Forza 4 Easy Medium Hard Advanced Expert Stability prevents the car from spinning when cornering too fast ON OFF Traction prevents the car from spinning when accelerating ON OFF Braking supports the player when he/she brakes or should brake Assisted w/ ABS ABS OFF Shifting helps the player in passing gears Automatic w/o clutch Manual w/o clutch Manual w/ clutch Line overlays the optimal trajectory to follow on the track Full Brake OFF Damage determines how much the performance of the car can change during the race Cosmetic Limited Simulation

Slide 35

Slide 35 text

Slide 36

Slide 36 text

Slide 37

Slide 37 text

Slide 38

Slide 38 text

Slide 39

Slide 39 text

© Microsoft Corporation Factors that contribute to the success of disabling an assist Factor More likely to keep the assist disabled … Significant for … Number of races Players who disable an assist early All assists Races per day Players who race fewer games a day All assists Rear-wheel drive (race before) Players who drove a car with rear-wheel drive All assists Car Performance Index (race before) Players who drove a car with lower PI All assists Position (race before) Players who finished first All assists but Traction and Clutch Career mode (race before) Players who did not play career mode Autobrake, ABS, Autoshift, Full line, Brake line

Slide 40

Slide 40 text

© Microsoft Corporation The First Hour Experience Gifford Cheung, Thomas Zimmermann, Nachiappan Nagappan: The First Hour Experience: How the Initial Play can Engage (or Lose) New Players. CHI Play 2014

Slide 41

Slide 41 text

© Microsoft Corporation the “initial experience” the “first time user experience” “tutorial time” the “5-5-5” (five seconds, minutes, hours) “the first 10 seconds, the first minute, the first ten minutes, the first hour” “setting the hook” also known as

Slide 42

Slide 42 text

© Microsoft Corporation “If your game is abandoned in the first hour that’s a pretty low moment. It’s a complete rejection of the game play in all ways, you know, game play, narrative, style. … It’s a pretty catastrophic failure.” Designer C a Critical Hour

Slide 43

Slide 43 text

Slide 44

Slide 44 text

Slide 45

Slide 45 text

© Microsoft Corporation The Data 212 Amazon reviews of Xbox 360 games Searching for keywords such as “first”, “hour”, “minute”, “start” 35 First hour reviews 6 Interviews with Game Industry Professionals (designer, tester, user researcher)

Slide 46

Slide 46 text

© Microsoft Corporation Methodology Grounded Theory Open, Axial, Selective Analysis (Amazon + First hour reviews) Invited an auditor to review the process and emergent categories for comprehensibility and closeness to data. Used the interviews in part to check the findings (Selective).

Slide 47

Slide 47 text

© Microsoft Corporation Findings: The First Experience Arc Expectations Specific Anticipations Experiences Momentary & Engaging Annoying, Frustrating, or Tedious Outcome Abandonment Continuing Exceptions Deal-breakers Holdouts

Slide 48

Slide 48 text

© Microsoft Corporation Deal-breakers: Frustrations that supersede all positives. “An UNSKIPPABLE cut scene. Annoying. So, we continue playing and now there are UNSKIPPABLE cut scenes every few plays of the players taunting each other before the ball is hiked! Really? I needed to see his defenseman saying something foul to my running back RIGHT NOW! NO, I don't! We made it through the rest of the game some how and decided that it was one of the worst playing experiences of any game we've ever played. Reselling this ASAP! BTW, the football game play was actually pretty good. Running, throwing and fielding all seemed to work well. In fact, I would say that it worked better than Madden. Its such a shame that they made the rest of the game so unplayable.” (Blitz: The League, Amazon Review)

Slide 49

Slide 49 text

© Microsoft Corporation Holdouts: Playing a game you hate Anticipated Elements “I can't stop before meeting some aliens, right?” (Crysis 2) A good game mechanic “Would I keep playing? I guess so. I really do like the combat system, which captures Captain America's acrobatic grace and power quite well, but the limited enemy variety is allowing that repetitious feeling to sink in. … I'll soldier on for a bit, but I don't expect to finish this fight.” (First hour review of Captain America: Super Soldier) Narrative Completionists (external factor) Social reasons (external factor)

Slide 50

Slide 50 text

© Microsoft Corporation Game development Emerson R. Murphy-Hill, Thomas Zimmermann, Nachiappan Nagappan: Cowboys, ankle sprains, and keepers of quality: how is video game development different from software development? ICSE 2014

Slide 51

Slide 51 text

Slide 52

Slide 52 text

© Microsoft Corporation Methodology: Interviews • Drew 14 participants from LinkedIn 2+ years dev experience in games AND 2+ years dev experience outside of games • Opened with a general question about “what’s the difference between games and other types of software?” • Then focused on SWEBOK and Work topics

Slide 53

Slide 53 text

© Microsoft Corporation Methodology: Surveys • Created 84 candidate statements based on interviews, narrowed down to 28 • Randomly sampled three populations, within Microsoft: – 300 engineers who work on games – 300 engineers who work on Microsoft Office – 300 engineers who work elsewhere

Slide 54

Slide 54 text

Slide 55

Slide 55 text

Slide 56

Slide 56 text

© Microsoft Corporation Significant Differences Game developers have less clear requirements than non-game developers. Game developers tend to use what they perceive as an Agile process more than non-game developers. Creativity is valued more in game development teams. The ability to communicate with non-engineers is valued more on game development teams. Game development requires a more diverse team. People are more impressed by game developers’ work.

Slide 57

Slide 57 text

Slide 58

Slide 58 text