Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Software Analytics for Digital Games

Thomas Zimmermann
May 18, 2015
4.1k

Software Analytics for Digital Games

Keynote presented at the Fourth International Workshop on Games and Software Engineering (GAS 2015).

http://2015.gasworkshop.org/

Thomas Zimmermann

May 18, 2015
Tweet

More Decks by Thomas Zimmermann

Transcript

  1. © Microsoft Corporation
    Software Analytics for Digital Games
    Thomas Zimmermann, Microsoft Research, USA
    Joint work with Nachi Nagappan and many others.
    http://aka.ms/esegroup

    View full-size slide

  2. © Microsoft Corporation

    View full-size slide

  3. © Microsoft Corporation
    Use of data, analysis, and
    systematic reasoning to
    [inform and] make
    decisions
    3

    View full-size slide

  4. © Microsoft Corporation
    history of software analytics
    Tim Menzies, Thomas Zimmermann: Software Analytics: So What?
    IEEE Software 30(4): 31-37 (2013)

    View full-size slide

  5. © Microsoft Corporation

    View full-size slide

  6. © Microsoft Corporation
    trinity of software analytics
    Dongmei Zhang, Shi Han, Yingnong Dang, Jian-Guang Lou, Haidong Zhang, Tao Xie:
    Software Analytics in Practice. IEEE Software 30(5): 30-37, September/October 2013.
    MSR Asia Software Analytics group: http://research.microsoft.com/en-us/groups/sa/

    View full-size slide

  7. © Microsoft Corporation
    Tom’s three Cupcakes of
    Software Analytics
    diversity people sharing

    View full-size slide

  8. © Microsoft Corporation
    diversity

    View full-size slide

  9. © Microsoft Corporation
    The Stakeholders
    The Tools The Questions

    View full-size slide

  10. © Microsoft Corporation
    http://aka.ms/145Questions
    Andrew Begel, Thomas Zimmermann. Analyze This! 145 Questions for Data Scientists
    in Software Engineering. ICSE 2014

    View full-size slide

  11. © Microsoft Corporation
    Microsoft’s Top 10 Questions Essential
    Essential +
    Worthwhile
    How do users typically use my application? 80.0% 99.2%
    What parts of a software product are most used and/or loved by
    customers?
    72.0% 98.5%
    How effective are the quality gates we run at checkin? 62.4% 96.6%
    How can we improve collaboration and sharing between teams? 54.5% 96.4%
    What are the best key performance indicators (KPIs) for
    monitoring services?
    53.2% 93.6%
    What is the impact of a code change or requirements change to
    the project and its tests?
    52.1% 94.0%
    What is the impact of tools on productivity? 50.5% 97.2%
    How do I avoid reinventing the wheel by sharing and/or searching
    for code?
    50.0% 90.9%
    What are the common patterns of execution in my application? 48.7% 96.6%
    How well does test coverage correspond to actual code usage by
    our customers?
    48.7% 92.0%

    View full-size slide

  12. © Microsoft Corporation
    people

    View full-size slide

  13. © Microsoft Corporation
    The Decider The Brain The Innovator
    Photo of MSA 2010 by Daniel M German ([email protected])
    The Researcher

    View full-size slide

  14. © Microsoft Corporation
    sharing

    View full-size slide

  15. © Microsoft Corporation
    Sharing Insights
    Sharing Methods
    Sharing Models
    Sharing Data

    View full-size slide

  16. © Microsoft Corporation
    Games Research
    in the ESE Group

    View full-size slide

  17. © Microsoft Corporation
    Xbox Live Influence of games and achievements on (paid) Xbox live memberships
    Influence of friends on titles played
    Characterizing players with Xbox Live data
    Gameplay Game play dynamics (Project Gotham Racing 4) => ESEM 2012
    Impact of social behavior on retention (Beta of a AAA title)
    Influence of gameplay on skill (Halo Reach) => CHI 2013
    Assists in a car racing game (Forza 4) => FDG 2014
    How to create a successful initial session in games => CHI Play 2014
    Engineering Differences between game and traditional software development => ICSE 2014
    Lessons learned from game development (ongoing)
    Mining software repositories from games (ongoing)
    Exploratory Personalization with Avatars in Xbox
    Geographic influence, temporal influence, and structural influence
    Thanks to our collaborators in Xbox and Microsoft Games Studios.
    Thanks to interns Ken Hullett, Sauvik Das, Jeff Huang, Gifford Cheung, Thomas Debeauvais,
    Erik Harpstead and visiting researchers Tim Menzies and Emerson Murphy-Hill.

    View full-size slide

  18. © Microsoft Corporation
    Skill in Halo Reach
    Jeff Huang, Thomas Zimmermann, Nachiappan Nagappan, Charles
    Harrison, Bruce C. Phillips: Mastering the art of war: how patterns of
    gameplay influence skill in Halo. CHI 2013: 695-704

    View full-size slide

  19. © Microsoft Corporation

    View full-size slide

  20. How do patterns of play affect
    players’ skill in Halo Reach?
    5 Skill and Other Titles
    6 Skill Changes and Retention
    7 Mastery and Demographics
    8 Predicting Skill
    2 Play Intensity
    3 Skill after Breaks
    4 Skill before Breaks
    1 General Statistics

    View full-size slide

  21. The Cohort of Players
    The mean skill value µ for each player after each Team Slayer match
    µ ranges between 0 and 10, although 50% fall between 2.5 and 3.5
    Initially µ = 3 for each player, stabilizing after a couple dozen matches
    TrueSkill in Team Slayer
    We looked at the cohort of players who started in the release week
    with complete set of gameplay for those players up to 7 months later
    (over 3 million players)
    70 Person Survey about Player Experience

    View full-size slide

  22. 2 Play Intensity
    Telegraph operators gradually increase typing speed over time

    View full-size slide

  23. 2.1
    2.3
    2.5
    2.7
    2.9
    3.1
    0 10 20 30 40 50 60 70 80 90 100
    mu
    Games Played So Far
    2 Play Intensity
    Median skill typically
    increases slowly over time
    Skill

    View full-size slide

  24. 2 Play Intensity (Games per Week)
    2.1
    2.3
    2.5
    2.7
    2.9
    3.1
    0 10 20 30 40 50 60 70 80 90 100
    mu
    Games Played So Far
    0 - 2 games / week [N=59164]
    2 - 4 games / week [N=101448]
    4 - 8 games / week [N=226161]
    8 - 16 games / week [N=363832]
    16 - 32 games / week [N=319579]
    32 - 64 games / week [N=420258]
    64 - 128 games / week [N=415793]
    128 - 256 games / week [N=245725]
    Median skill typically
    increases slowly over time
    Skill

    View full-size slide

  25. 2 Play Intensity (Games per Week)
    2.1
    2.3
    2.5
    2.7
    2.9
    3.1
    0 10 20 30 40 50 60 70 80 90 100
    mu
    Games Played So Far
    0 - 2 games / week [N=59164]
    2 - 4 games / week [N=101448]
    4 - 8 games / week [N=226161]
    8 - 16 games / week [N=363832]
    16 - 32 games / week [N=319579]
    32 - 64 games / week [N=420258]
    64 - 128 games / week [N=415793]
    128 - 256 games / week [N=245725]
    Median skill typically
    increases slowly over time
    Skill

    View full-size slide

  26. 2 Play Intensity (Games per Week)
    2.1
    2.3
    2.5
    2.7
    2.9
    3.1
    0 10 20 30 40 50 60 70 80 90 100
    mu
    Games Played So Far
    0 - 2 games / week [N=59164]
    2 - 4 games / week [N=101448]
    4 - 8 games / week [N=226161]
    8 - 16 games / week [N=363832]
    16 - 32 games / week [N=319579]
    32 - 64 games / week [N=420258]
    64 - 128 games / week [N=415793]
    128 - 256 games / week [N=245725]
    But players who play
    more overall eventually
    surpass those who play
    4–8 games per week
    (not shown in chart)
    Players who play 4–8
    games per week do best
    Median skill typically
    increases slowly over time
    Skill

    View full-size slide

  27. 3 Change in Skill Following a Break
    “In the most drastic scenario, you can lose
    up to 80 percent of your fitness level in as
    few as two weeks [of taking a break]…”

    View full-size slide

  28. -0.03
    -0.02
    -0.01
    0
    0.01
    0.02
    0.03
    0 5 10 15 20 25 30 35 40 45 50
    Δmu
    Days of Break
    Next Game
    2 Games Later
    3 Games Later
    4 Games Later
    5 games later
    10 games later
    3 Change in Skill Following a Break
    Median skill slightly
    increases after each game
    played without breaks
    Longer breaks correlate
    with larger skill drops, but
    not linearly
    On average, it takes 8–10
    games to regain skill lost
    after 30 day breaks
    Breaks of 1–2 days
    correlate in tiny
    drops in skill
    Change in Skill

    View full-size slide

  29. © Microsoft Corporation
    Driving skill in
    Forza Motorsports 4
    Thomas Debeauvais, Thomas Zimmermann, Nachiappan Nagappan,
    Kevin Carter, Ryan Cooper, Dan Greenawalt, Tyson Solberg: An Empirical
    Study of Driving Skill in Forza Motorsports 4. FDG 2014

    View full-size slide

  30. © Microsoft Corporation
    Driving skill in Forza Motorsports 4
    5% of player base,
    sampled randomly
    200k players who
    played 25M races
    Assist usage
    Assist transitions

    View full-size slide

  31. © Microsoft Corporation

    View full-size slide

  32. © Microsoft Corporation
    Approaching a turn in Forza 4
    – in EASY mode –

    View full-size slide

  33. © Microsoft Corporation
    Approaching a turn in Forza 4
    – in EXPERT mode –

    View full-size slide

  34. © Microsoft Corporation
    The assist bundles in Forza 4
    Easy Medium Hard Advanced Expert
    Stability
    prevents the car from spinning
    when cornering too fast
    ON OFF
    Traction
    prevents the car from spinning
    when accelerating
    ON OFF
    Braking
    supports the player when he/she
    brakes or should brake
    Assisted
    w/ ABS
    ABS OFF
    Shifting helps the player in passing gears
    Automatic
    w/o clutch
    Manual
    w/o clutch
    Manual
    w/ clutch
    Line
    overlays the optimal trajectory to
    follow on the track
    Full Brake OFF
    Damage
    determines how much the
    performance of the car can
    change during the race
    Cosmetic Limited Simulation

    View full-size slide

  35. © Microsoft Corporation
    number of races
    Assist usage over number of races
    career mode
    online multiplayer
    number of races

    View full-size slide

  36. © Microsoft Corporation
    Assist transitions
    time
    The player disables
    the assist

    View full-size slide

  37. © Microsoft Corporation
    Assist transitions
    time
    The player disables
    the assist

    View full-size slide

  38. © Microsoft Corporation
    0%
    10%
    20%
    30%
    40%
    50%
    60%
    70%
    80%
    90%
    100%
    success failure yoyo never disabled
    Assist transitions

    View full-size slide

  39. © Microsoft Corporation
    Factors that contribute to the success
    of disabling an assist
    Factor More likely to keep the assist disabled … Significant for …
    Number of races Players who disable an assist early All assists
    Races per day Players who race fewer games a day All assists
    Rear-wheel drive
    (race before)
    Players who drove a car with rear-wheel
    drive
    All assists
    Car Performance Index
    (race before)
    Players who drove a car with lower PI All assists
    Position
    (race before)
    Players who finished first All assists but
    Traction and Clutch
    Career mode
    (race before)
    Players who did not play career mode Autobrake, ABS,
    Autoshift, Full line,
    Brake line

    View full-size slide

  40. © Microsoft Corporation
    The First Hour
    Experience
    Gifford Cheung, Thomas Zimmermann, Nachiappan Nagappan: The First
    Hour Experience: How the Initial Play can Engage (or Lose) New Players.
    CHI Play 2014

    View full-size slide

  41. © Microsoft Corporation
    the “initial experience”
    the “first time user
    experience”
    “tutorial time”
    the “5-5-5” (five seconds,
    minutes, hours)
    “the first 10 seconds, the
    first minute, the first ten
    minutes, the first hour”
    “setting the hook”
    also known as

    View full-size slide

  42. © Microsoft Corporation
    “If your game is abandoned
    in the first hour that’s a
    pretty low moment. It’s a
    complete rejection of the
    game play in all ways, you
    know, game play, narrative,
    style. … It’s a pretty
    catastrophic failure.”
    Designer C
    a Critical Hour

    View full-size slide

  43. © Microsoft Corporation
    The Data
    212 Amazon reviews of Xbox 360
    games
    Searching for keywords such as
    “first”, “hour”, “minute”, “start”

    View full-size slide

  44. © Microsoft Corporation
    The Data
    212 Amazon reviews of Xbox 360
    games
    Searching for keywords such as
    “first”, “hour”, “minute”, “start”
    35 First hour reviews

    View full-size slide

  45. © Microsoft Corporation
    The Data
    212 Amazon reviews of Xbox 360
    games
    Searching for keywords such as
    “first”, “hour”, “minute”, “start”
    35 First hour reviews
    6 Interviews with Game
    Industry Professionals
    (designer, tester, user
    researcher)

    View full-size slide

  46. © Microsoft Corporation
    Methodology
    Grounded Theory
    Open, Axial, Selective Analysis
    (Amazon + First hour reviews)
    Invited an auditor to review the
    process and emergent categories
    for comprehensibility and
    closeness to data.
    Used the interviews in part to
    check the findings (Selective).

    View full-size slide

  47. © Microsoft Corporation
    Findings:
    The First Experience Arc
    Expectations
    Specific Anticipations
    Experiences
    Momentary & Engaging
    Annoying, Frustrating, or Tedious
    Outcome
    Abandonment
    Continuing
    Exceptions
    Deal-breakers
    Holdouts

    View full-size slide

  48. © Microsoft Corporation
    Deal-breakers: Frustrations that
    supersede all positives.
    “An UNSKIPPABLE cut scene. Annoying. So, we continue playing and now there
    are UNSKIPPABLE cut scenes every few plays of the players taunting each other
    before the ball is hiked! Really? I needed to see his defenseman saying something
    foul to my running back RIGHT NOW! NO, I don't! We made it through the rest of
    the game some how and decided that it was one of the worst playing experiences of
    any game we've ever played. Reselling this ASAP!
    BTW, the football game play was actually pretty good. Running, throwing and
    fielding all seemed to work well. In fact, I would
    say that it worked better than Madden. Its such
    a shame that they made the rest of the game
    so unplayable.”
    (Blitz: The League, Amazon Review)

    View full-size slide

  49. © Microsoft Corporation
    Holdouts: Playing a game you hate
    Anticipated Elements
    “I can't stop before meeting some aliens, right?” (Crysis 2)
    A good game mechanic
    “Would I keep playing? I guess so. I really do like the combat system, which captures
    Captain America's acrobatic grace and power quite well, but the limited enemy variety is
    allowing that repetitious feeling to sink in. … I'll soldier on for a bit, but I don't expect to
    finish this fight.” (First hour review of Captain America: Super Soldier)
    Narrative
    Completionists (external factor)
    Social reasons (external factor)

    View full-size slide

  50. © Microsoft Corporation
    Game development
    Emerson R. Murphy-Hill, Thomas Zimmermann, Nachiappan Nagappan:
    Cowboys, ankle sprains, and keepers of quality: how is video game
    development different from software development? ICSE 2014

    View full-size slide

  51. © Microsoft Corporation

    View full-size slide

  52. © Microsoft Corporation
    Methodology: Interviews
    • Drew 14 participants from LinkedIn
    2+ years dev experience in games AND
    2+ years dev experience outside of games
    • Opened with a general question about
    “what’s the difference between games
    and other types of software?”
    • Then focused on SWEBOK and
    Work topics

    View full-size slide

  53. © Microsoft Corporation
    Methodology: Surveys
    • Created 84 candidate statements based
    on interviews, narrowed down to 28
    • Randomly sampled three populations,
    within Microsoft:
    – 300 engineers who work on games
    – 300 engineers who work on Microsoft Office
    – 300 engineers who work elsewhere

    View full-size slide

  54. © Microsoft Corporation

    View full-size slide

  55. © Microsoft Corporation

    View full-size slide

  56. © Microsoft Corporation
    Significant Differences
    Game developers have less clear requirements than
    non-game developers.
    Game developers tend to use what they perceive as an
    Agile process more than non-game developers.
    Creativity is valued more in game development teams.
    The ability to communicate with non-engineers is valued
    more on game development teams.
    Game development requires a more diverse team.
    People are more impressed by game developers’ work.

    View full-size slide

  57. © Microsoft Corporation
    Software Analytics for Digital Games
    Thomas Zimmermann, Microsoft Research, USA
    Joint work with Nachi Nagappan and many others.
    http://aka.ms/esegroup

    View full-size slide

  58. © Microsoft Corporation
    Thank you!

    View full-size slide