Invited talk presented at the FSE 2014 conference.
© Microsoft Corporation
View Slide
© Microsoft Corporation“On the fountain, there should be 2 jugs, do you seethem? A 5 gallon and a 3 gallon. Fill one of the jugwith exactly 4 gallons of water and place it on thescale and the timer will stop. It must be precise, oneounce of more or less will result in detonation.If you're still alive in 5 minutes, we'll speak.”
© Microsoft CorporationAction movies Software developmentHeroes save the world Engineers build softwareTight deadlinesYes YesWrong information can be disastrousExploding bombsWorld dominationCancelled/delayed projectsLow quality softwareLost dataThe endingUsually happy end. Sometimes happy end.
© Microsoft CorporationThe personal health assistant Baymaxfrom the Disney picture“Big Hero 6”
© Microsoft CorporationEmpower people involved with softwareto make sound data-driven decisionsabout software.Bug trackingSoftware analyticsGames analyticsSoftware qualityProcess improvement(branches, build)Productivity
© Microsoft CorporationESE Group inSummer 2014ESE Group inSummer 2013
© Microsoft CorporationCodeMine
© Microsoft CorporationSix years ago…
© Microsoft CorporationWindowsBrendan Murphy Nachi Nagappan
© Microsoft CorporationBuildOrganizationSource CodeWork ItemCode ReviewTestcallsresolvesopensresolvessubmitsbelongs toimplementsrequestscomments onsubmitsasbelongs totestsuseseditssubmittedintomovesdefinesdefinesworkswithProcess Informationshipsfromshipscreated onbuilt onScheduleProductTest JobExecutableIntegrationBranchChangeSourceFileProcedure /MethodClass / TypeReviewFeature /DefectPersonJacek Czerwonka, Nachiappan Nagappan, Wolfram Schulte, Brendan Murphy:CODEMINE: Building a Software Development Data Analytics Platform at Microsoft.IEEE Software 30(4): 64-71 (2013)
© Microsoft CorporationRisk Prediction
© Microsoft CorporationmainnetworkingmultimediaBranches at MicrosoftChanges are isolated=> Less build and test breaksProcess overheadTime delay (velocity)integrationintegration
© Microsoft CorporationBlue nodes areedits to the fileOrange nodes aremove operations
© Microsoft CorporationVisualizing code velocityMostly editsMostly integrationsAvg. time for CL to reach next branch:<= 1 week>= 3 weeks
© Microsoft CorporationAssessing branchesDelay(Cost)Provided Isolation(Benefit)Green dotsare brancheswith high benefitand low costRed dotsare brancheswith high costbut low benefitEach dotis a branchChristian Bird, Thomas Zimmermann:Assessing the value of branches with what-if analysis. SIGSOFT FSE 2012
© Microsoft CorporationAssessing branchesDelay(Cost)Provided Isolation(Benefit)Green dotsare brancheswith high benefitand low costRed dotsare brancheswith high costbut low benefitEach dotis a branchChristian Bird, Thomas Zimmermann:Assessing the value of branches with what-if analysis. SIGSOFT FSE 2012If high-cost-low-benefit had been removed,changes would each have saved 8.9 days of delayand only introduced 0.04 additional conflicts.
© Microsoft CorporationSimplified branch treesBranching Taxonomy. B. Murphy, J. Czerwonka, and L. Williams. Microsoft ResearchTechnical Report. MSR-TR-2014-23. http://research.microsoft.com/apps/pubs/?id=209683
© Microsoft CorporationField Studies
© Microsoft CorporationCowboys, ankle sprains, and keepers of quality: How isvideo game development different fromsoftware development? Emerson R. Murphy-Hill,Thomas Zimmermann, Nachiappan Nagappan. ICSE 2014Understanding and improving softwarebuild teams.Shaun Phillips, Thomas Zimmermann, Christian Bird.ICSE 2014A field study ofrefactoring challenges and benefits.Miryung Kim, Thomas Zimmermann, Nachiappan Nagappan.SIGSOFT FSE 2012.Refactoring
© Microsoft Corporationhttp://aka.ms/145QuestionsAndrew Begel, Thomas Zimmermann.Analyze This! 145 Questions for Data Scientists in Software Engineering. ICSE 2014
© Microsoft CorporationMicrosoft’s Top 10 Questions EssentialEssential +WorthwhileHow do users typically use my application? 80.0% 99.2%What parts of a software product are most used and/or loved bycustomers?72.0% 98.5%How effective are the quality gates we run at checkin? 62.4% 96.6%How can we improve collaboration and sharing between teams? 54.5% 96.4%What are the best key performance indicators (KPIs) formonitoring services?53.2% 93.6%What is the impact of a code change or requirements change tothe project and its tests?52.1% 94.0%What is the impact of tools on productivity? 50.5% 97.2%How do I avoid reinventing the wheel by sharing and/or searchingfor code?50.0% 90.9%What are the common patterns of execution in my application? 48.7% 96.6%How well does test coverage correspond to actual code usage byour customers?48.7% 92.0%More at http://aka.ms/145Questions
© Microsoft CorporationGames
© Microsoft CorporationThanks to our collaborators in Xbox, Microsoft Games Studios, and Turn 10.Thanks to interns Ken Hullett, Sauvik Das, Jeff Huang, Gifford Cheung, Thomas Debeauvais,Erik Harpstead and visiting researchers Tim Menzies and Emerson Murphy-Hill.Xbox Live Influence of games and achievements on (paid) Xbox live membershipsInfluence of friends on titles playedCharacterizing players with Xbox Live dataGameplay Impact of social behavior on retention (Beta of a AAA title)Influence of gameplay on skill (Halo Reach) => CHI 2013Assists in a car racing game (Forza 4) => FDG 2014How to create a successful initial session in games => CHI Play 2014Engineering Differences between game and traditional software development => ICSE 2014Lessons learned from game development (ongoing)Mining software repositories from games (ongoing)Exploratory Personalization with Avatars in XboxGeographic influence, temporal influence, and structural influence
© Microsoft CorporationDriving skill in Forza Motorsports 45% of player base,sampled randomly200k players whoplayed 25M racesAssist usageAssist transitionsThomas Debeauvais, Thomas Zimmermann, Nachiappan Nagappan, Kevin Carter, Ryan Cooper,Dan Greenawalt, Tyson Solberg: An Empirical Study of Driving Skill in Forza Motorsports 4.FDG 2014
© Microsoft CorporationApproaching a turn in Forza 4– in EASY mode –
© Microsoft CorporationApproaching a turn in Forza 4– in HARD mode –
© Microsoft CorporationThe assist bundles in Forza 4Easy Medium Hard Advanced ExpertStabilityprevents the car from spinningwhen cornering too fastON OFFTractionprevents the car from spinningwhen acceleratingON OFFBrakingsupports the player when he/shebrakes or should brakeAssistedw/ ABSABS OFFShifting helps the player in passing gearsAutomaticw/o clutchManualw/o clutchManualw/ clutchLineoverlays the optimal trajectory tofollow on the trackFull Brake OFFDamagedetermines how much theperformance of the car canchange during the raceCosmetic Limited Simulation
© Microsoft Corporationnumber of racesAssist usage over number of racescareer modeonline multiplayernumber of races
© Microsoft CorporationAssist transitionsenableddisabledrace beforerace afteryoyofailuresuccesstimetimeThe player disablesthe assist
© Microsoft Corporation0%10%20%30%40%50%60%70%80%90%100%success failure yoyo never disabledAssist transitions
© Microsoft CorporationFactors that contribute to the successof disabling an assistFactor More likely to keep the assist disabled … Significant for …Number of races Players who disable an assist early All assistsRaces per day Players who race fewer games a day All assistsRear-wheel drive(race before)Players who drove a car with rear-wheeldriveAll assistsCar Performance Index(race before)Players who drove a car with lower PI All assistsPosition(race before)Players who finished first All assists butTraction and ClutchCareer mode(race before)Players who did not play career mode Autobrake, ABS,Autoshift, Full line,Brake line
© Microsoft Corporation“Your work has been incredibly helpful to myteam. Just this week, we’ve had 8 hours ofmeeting to design our core gameplay loopsbased directly on your data. Quite literally,we project your data and the player profileson the wall while we design. Its awesome.”About our work on a different game title.
© Microsoft CorporationHow to measure insight?Amount of discussion theinsight generates?Number of times the usersinvite you back?Number of issues visited andretired in a meeting?Number ofhypotheses rejected?Tim Menzies
© Microsoft CorporationQuestions youwant to askQuestionsdatasupportsQuestionsuser caresaboutInductive EngineeringTim Menzies, Christian Bird, Thomas Zimmermann, Wolfram Schulte, Ekrem Kocaganeli.The Inductive Software Engineering Manifesto: Principles for Industrial Data Mining.MALETS 2011
© Microsoft CorporationInductive Engineering1. Users before algorithms2. Plan for scale3. Early feedback4. Be open-minded5. Do smart learning6. Live with the data you have7. Broad skill set, big toolkit