Using Data to Inform Decisions in Agile Software Development

Presentation of the paper "Towards using Data to Inform Decisions in Agile Software Development: Views of Available Data" held on July 28th 2019 at the 14th International Conference on Software Technologies (ICSOFT'19) in Prague.

Paper authors: Christoph Matthies, Guenter Hesse. Hasso Plattner Institute, University of Potsdam, Germany

Transcript

1. Hasso Plattner Institute
   University of Potsdam, Germany
   christoph.matthies@hpi.de
   @chrisma0
   Using Data to Inform Decisions in Agile
   Software Development
   Christoph Matthies, Guenter Hesse
   Prague, July ’19
   

   View Slide

2. Motivation
   2
   for using data in software engineering
   Software engineering must shed the folkloric advice and
   anecdotal evidence [...] and replace them with
   convincing results, established with [...] empirical methods
   – Bertrand Meyer [Meyer, 2013]
   “
   ”
   [Meyer, 2013] B. Meyer, H. Gall, M. Harman, and G. Succi, “Empirical Answers to Fundamental Software
   Engineering Problems (Panel),” in Proceedings of the 2013 9th Joint Meeting on Foundations of Software
   Engineering, ser. ESEC/FSE 2013. New York, USA: ACM, 2013, pp. 14–18.
   Picture: https://commons.wikimedia.org/wiki/File:Bertrand_Meyer_recent.jpg
   

   View Slide

3. Importance of Decision-Making
   3
   in software development projects
   ■ Software project success hinges on many factors
   □ Budget, scope, team organization, technology, architecture
   □ Customer engagement, requirements analysis
   [Drury, 2012] Drury, M., Conboy, K., and Power, K. (2012). Obstacles to decision making
   in Agile software development teams. Journal of Systems and Software, 85(6):1239–1254.
   [Taherdoost, 2018] Taherdoost, H. and Keshavarzsaleh, A. (2018). A Theoretical Review on IT Project Success/Failure
   Factors and Evaluating the Associated Risks. In Mathematical and Computational Methods in Electrical Engineering
   

   View Slide

4. Importance of Decision-Making
   4
   in software development projects
   ■ Software project success hinges on many factors
   □ Budget, scope, team organization, technology, architecture
   □ Customer engagement, requirements analysis
   ■ Correct project decisions critical for success [Drury, 2012]
   □ Overruns of budget and schedule
   □ Project cancellation [Taherdoost, 2018]
   [Drury, 2012] Drury, M., Conboy, K., and Power, K. (2012). Obstacles to decision making
   in Agile software development teams. Journal of Systems and Software, 85(6):1239–1254.
   [Taherdoost, 2018] Taherdoost, H. and Keshavarzsaleh, A. (2018). A Theoretical Review on IT Project Success/Failure
   Factors and Evaluating the Associated Risks. In Mathematical and Computational Methods in Electrical Engineering
   

   View Slide

5. Motivation
   5
   for using data in Agile software development processes
   Scrum is founded on empirical process control theory [...]
   knowledge comes from experience and making decisions
   based on what is known. [...]
   – The Scrum Guide [Schwaber, 2017]
   [Schwaber, 2017] K. Schwaber, J. Sutherland, “The Scrum Guide - The Definitive Guide to Scrum,” 2017,
   [online] http://scrumguides.org/docs/scrumguide/v2017/2017-Scrum-Guide-US.pdf
   Picture: https://www.scrum.org/resources/2017-scrum-guide-update-ken-schwaber-and-jeff-sutherland
   “
   ”
   

   View Slide

6. Decision-Making in Agile
   6
   Evidence-based decision-making in teams
   ■ Agile teams should make use of available data to enable
   “evidence-based decision making” [Fitzgerald, 2014]
   ■ However, few details available on concrete implementations
   [Fitzgerald, 2014] Fitzgerald, B., Musiał, M., and Stol, K.-J. (2014). Evidence-based decision making in lean software project management. In
   Companion Proceedings of the 36th International Conference on Software Engineering - ICSE Companion 2014, pages 93–102, New York,
   New York, USA. ACM Press.
   

   View Slide

7. Decision-Making in Agile
   7
   Evidence-based decision-making in teams
   ■ Agile teams should make use of available data to enable
   “evidence-based decision making” [Fitzgerald, 2014]
   ■ However, few details available on concrete implementations
   Main Research Questions
   ■ What data is available and relevant to Agile teams?
   ■ What methods are transferable from other domains?
   [Fitzgerald, 2014] Fitzgerald, B., Musiał, M., and Stol, K.-J. (2014). Evidence-based decision making in lean software project management. In
   Companion Proceedings of the 36th International Conference on Software Engineering - ICSE Companion 2014, pages 93–102, New York,
   New York, USA. ACM Press.
   

   View Slide

8. Related Work [Svensson, 2019]
   8
   Unfulfilled Potential of DDDM in Agile
   [Svensson, 2019] Svensson, R.B., Feldt, R., & Torkar, R. “The Unfulfilled Potential of Data-Driven Decision Making in Agile Software Development”,
   20th International Conference on Agile Software Development (XP), 2019 (preprint), https://arxiv.org/abs/1904.03948
   

   View Slide

9. Potential of DDDM
   9
   Lots of potential, little usage [Svensson, 2019]
   ■ Survey of software practitioners
   ■ How is data used in the company for making decisions?
   [Svensson, 2019] Svensson, R.B., Feldt, R., & Torkar, R. “The Unfulfilled Potential of Data-Driven Decision Making in Agile Software Development”,
   20th International Conference on Agile Software Development (XP), 2019 (preprint), https://arxiv.org/abs/1904.03948
   

   View Slide

10. Data-Driven Decision-Making
    10
    In business contexts
    ■ DDDM contrasted with “gut feeling” [Provost, 2013]
    □ Standard in business in the absence of analysis technologies
    [Provost, 2013] Provost, F. and Fawcett, T. (2013). Data Science and its Relationship to Big Data
    and Data-Driven Decision Making. Big Data, 1(1):51–59.
    [Brynjolfsson, 2011] Brynjolfsson, E., Hitt, L. M., and Kim, H. H. (2011). Strength in Numbers:
    How Does Data-Driven Decisionmaking Affect Firm Performance? SSRN Electronic Journal.
    [Schrage, 2016] Schrage, M. (2016). How the big data explosion has changed decision making.
    Harvard Business Review Digital Articles, pages 2–5
    

    View Slide

11. Data-Driven Decision-Making
    11
    In business contexts
    ■ DDDM contrasted with “gut feeling” [Provost, 2013]
    □ Standard in business in the absence of analysis technologies
    ■ Technology use in business has fueled DDDM adoption
    □ Increased company productivity [Brynjolfsson, 2011]
    □ Transparency & real-time information intensifies tensions,
    management must adapt [Schrage, 2016]
    [Provost, 2013] Provost, F. and Fawcett, T. (2013). Data Science and its Relationship to Big Data
    and Data-Driven Decision Making. Big Data, 1(1):51–59.
    [Brynjolfsson, 2011] Brynjolfsson, E., Hitt, L. M., and Kim, H. H. (2011). Strength in Numbers:
    How Does Data-Driven Decisionmaking Affect Firm Performance? SSRN Electronic Journal.
    [Schrage, 2016] Schrage, M. (2016). How the big data explosion has changed decision making.
    Harvard Business Review Digital Articles, pages 2–5
    

    View Slide

12. Agile Software Development
    12
    Development Processes and their connection to data
    ■ Plan-driven, rational decision-making [Moe, 2012]
    □ Sets of rules that govern behavior
    □ Assumption: outcome probabilities are known [Zannier, 2007]
    [Moe, 2012] Moe, N. B., Aurum, A., and Dybå, T. (2012). Challenges of shared decision-making:
    A multiple case study of agile software development. Information and Software Technology, 54(8):853–865.
    [Zannier, 2007] Zannier, C. and Maurer, F. (2006). Foundations of Agile Decision Making from Agile Mentors
    and Developers. In Abrahamsson, P., Marchesi, M., and Succi, G., editors, Extreme Programming and
    Agile Processes in Software Engineering, pages 11–20, Berlin, Heidelberg. Springer Berlin Heidelberg.
    

    View Slide

13. Agile Software Development
    13
    Development Processes and their connection to data
    ■ Plan-driven, rational decision-making [Moe, 2012]
    □ Sets of rules that govern behavior
    □ Assumption: outcome probabilities are known [Zannier, 2007]
    ■ Data enables teams to implement Agile feedback cycles
    ■ Agile, naturalistic decision-making
    □ Situational behavior, context-dependent rules
    [Moe, 2012] Moe, N. B., Aurum, A., and Dybå, T. (2012). Challenges of shared decision-making:
    A multiple case study of agile software development. Information and Software Technology, 54(8):853–865.
    [Zannier, 2007] Zannier, C. and Maurer, F. (2006). Foundations of Agile Decision Making from Agile Mentors
    and Developers. In Abrahamsson, P., Marchesi, M., and Succi, G., editors, Extreme Programming and
    Agile Processes in Software Engineering, pages 11–20, Berlin, Heidelberg. Springer Berlin Heidelberg.
    

    View Slide

14. Levels of Decisions
    14
    The impact of Agile “self-organizing” teams
    ■ Strategic plans define long-term goals
    ■ Tactical plans involve managing projects to achieve goals
    ■ Operational plans define milestones, and the actual work
    

    View Slide

15. Data Use in Business
    15
    A wealth of data to use for decisions
    ■ Businesses gather detailed statistics
    □ Users, suppliers, partners & competitors
    □ ERP systems for collection, BI for analysis [Chen, 2012]
    [Chen, 2012] Chen, H., Chiang, R. H. L., and Storey, V. C. (2012). Business intelligence and analytics:
    From big data to big impact. MIS quarterly, 36(4).
    [Svensson, 2019] Svensson, R. B., Feldt, R., and Torkar, R. (2019). The Unfulfilled Potential of Data-Driven
    Decision Making in Agile Software Development. In Agile Processes in Software Engineering and Extreme
    Programming, pages 69–85. Springer International Publishing.
    

    View Slide

16. Data Use in Business
    16
    A wealth of data to use for decisions
    ■ Businesses gather detailed statistics
    □ Users, suppliers, partners & competitors
    □ ERP systems for collection, BI for analysis [Chen, 2012]
    ■ Continuing digitization allows additional data collection
    □ “Reality mining”: cars, smart devices, etc. [Svensson, 2019]
    □ Clickstream data, UI interactions, high time-resolution
    [Chen, 2012] Chen, H., Chiang, R. H. L., and Storey, V. C. (2012). Business intelligence and analytics:
    From big data to big impact. MIS quarterly, 36(4).
    [Svensson, 2019] Svensson, R. B., Feldt, R., and Torkar, R. (2019). The Unfulfilled Potential of Data-Driven
    Decision Making in Agile Software Development. In Agile Processes in Software Engineering and Extreme
    Programming, pages 69–85. Springer International Publishing.
    

    View Slide

17. Data in Software Development
    17
    Much more than code
    ■ Software development process itself: source of valuable data
    □ Intrinsically relevant to making decisions
    □ Not only code, but a multitude of supporting artifacts
    [de Souza, 2005] de Souza, C., Froehlich, J., and Dourish, P. (2005). Seeking the Source: Software Source Code as a Social and Technical Artifact. In
    Proceedings of the 2005 international ACM SIGGROUP conference on Supporting group work - GROUP ’05, page 197, New York, New York, USA.
    ACM, ACM Press.
    [Ziftci & Reardon] Ziftci, C. and Reardon, J. (2017). Who broke the build? Automatically identifying changes that induce test failures in continuous
    integration at google scale. In Proceedings - 2017 IEEE/ACM 39th International Conference on Software Engineering: Software Engineering in
    Practice Track, ICSE-SEIP 2017, pages 113–122.
    

    View Slide

18. Data in Software Development
    18
    Much more than code
    ■ Software development process itself: source of valuable data
    □ Intrinsically relevant to making decisions
    □ Not only code, but a multitude of supporting artifacts
    ■ Processes are “inscribed” into software artifacts [de Souza, 2005]
    □ E.g. VCS: who, when, what, why, status
    □ Also includes evidence of failures [Ziftci & Reardon, 2017]
    [de Souza, 2005] de Souza, C., Froehlich, J., and Dourish, P. (2005). Seeking the Source: Software Source Code as a Social and Technical Artifact. In
    Proceedings of the 2005 international ACM SIGGROUP conference on Supporting group work - GROUP ’05, page 197, New York, New York, USA.
    ACM, ACM Press.
    [Ziftci & Reardon] Ziftci, C. and Reardon, J. (2017). Who broke the build? Automatically identifying changes that induce test failures in continuous
    integration at google scale. In Proceedings - 2017 IEEE/ACM 39th International Conference on Software Engineering: Software Engineering in
    Practice Track, ICSE-SEIP 2017, pages 113–122.
    

    View Slide

19. Software Project Data
    19
    [Guo et al., 2016] Guo, J., Rahimi, M., Cleland-Huang, J., Rasin, A., Hayes, J. H., & Vierhauser, M, “Cold-start software analytics”, In Proceedings of the
    13th International Workshop on Mining Software Repositories - MSR ’16, pp. 142–153, New York, New York, USA, ACM Press, 2016.
    “a gold-mine of actionable information” [Guo et al., 2016]
    

    View Slide

20. Software Project Data
    20
    “a gold-mine of actionable information” [Guo et al., 2016]
    Primary purpose: communication
    [Guo et al., 2016] Guo, J., Rahimi, M., Cleland-Huang, J., Rasin, A., Hayes, J. H., & Vierhauser, M, “Cold-start software analytics”, In Proceedings of the
    13th International Workshop on Mining Software Repositories - MSR ’16, pp. 142–153, New York, New York, USA, ACM Press, 2016.
    

    View Slide

21. Software Project Data
    21
    Use this data for reflection
    [Guo et al., 2016] Guo, J., Rahimi, M., Cleland-Huang, J., Rasin, A., Hayes, J. H., & Vierhauser, M, “Cold-start software analytics”, In Proceedings of the
    13th International Workshop on Mining Software Repositories - MSR ’16, pp. 142–153, New York, New York, USA, ACM Press, 2016.
    “a gold-mine of actionable information” [Guo et al., 2016]
    

    View Slide

22. Scrum Process
    22
    Where Agile decision-making happens [Schwaber & Sutherland, 2017]
    1
    2
    3
    4
    5
    6
    7
    Sprint
    Planning
    Sprint
    Backlog
    Backlog
    Sprint
    Daily
    Stakeholder
    Input
    Product
    4 iterations
    Sprint
    Retrospective Finished
    Increment
    Sprint Review
    [Schwaber & Sutherland, 2017] Schwaber, K., & Sutherland, J, “The Scrum Guide - The Definitive Guide to Scrum: The Rules of the Game”, 2017,
    [online] Available: http://scrumguides.org/docs/scrumguide/v2017/2017-Scrum-Guide-US.pdf
    

    View Slide

23. Sprint Planning
    23
    1
    2
    3
    4
    5
    6
    7
    Sprint
    Planning
    Sprint
    Backlog
    Backlog
    Sprint
    Daily
    Stakeholder
    Input
    Product
    4 iterations
    Sprint
    Retrospective Finished
    Increment
    Sprint Review
    Building a backlog of work for the next Sprint
    

    View Slide

24. Sprint Planning
    24
    Building a backlog of work for the next Sprint
    ■ Selection and prioritization of features for next iteration
    □ Based on stakeholders’ experiences & opinions [Svensson, 2019]
    □ Lack explanation and links to evidence
    

    View Slide

25. Sprint Planning
    25
    Building a backlog of work for the next Sprint
    ■ Selection and prioritization of features for next iteration
    □ Based on stakeholders’ experiences & opinions [Svensson, 2019]
    □ Lack explanation and links to evidence
    ■ Enable short feedback cycles between devs and real customers
    □ Metrics from deployed software, real use cases
    □ Observe actual (mis)usage, engineering relevant metrics
    □ Base changes on results
    

    View Slide

26. Daily Scrum
    26
    Quick status update
    1
    2
    3
    4
    5
    6
    7
    Sprint
    Planning
    Sprint
    Backlog
    Backlog
    Sprint
    Daily
    Stakeholder
    Input
    Product
    4 iterations
    Sprint
    Retrospective Finished
    Increment
    Sprint Review
    

    View Slide

27. Daily Scrum
    27
    Quick status update
    ■ Short daily status meeting
    □ “What did I do?”, “What will I do?”, “Blockers?”
    □ Strictly time-boxed
    

    View Slide

28. Daily Scrum
    28
    Quick status update
    ■ Short daily status meeting
    □ “What did I do?”, “What will I do?”, “Blockers?”
    □ Strictly time-boxed
    ■ Development artifacts hold information on what was done
    □ Concrete items that were completed
    □ Rely less on personal recollection
    □ Optimize for speed
    

    View Slide

29. Sprint Retrospective
    29
    Reflecting on past work and improving
    1
    2
    3
    4
    5
    6
    7
    Sprint
    Planning
    Sprint
    Backlog
    Backlog
    Sprint
    Daily
    Stakeholder
    Input
    Product
    4 iterations
    Sprint
    Retrospective Finished
    Increment
    Sprint Review
    

    View Slide

30. Retrospectives
    30
    Reflecting on past work and improving
    What should be
    improved?
    Retrospective
    Meeting
    

    View Slide

31. 31
    Did we improve
    what we planned?
    Retrospective
    Meeting
    Retrospectives
    Reflecting on past work and improving
    

    View Slide

32. 32
    Did we improve
    what we planned?
    Retrospective
    Meeting
    Project Data
    Evidence of last
    iteration’s work
    code
    docs
    stats
    Retrospectives
    Reflecting on past work and improving
    

    View Slide

33. 33
    Did we improve
    what we planned?
    Retrospective
    Meeting
    Project Data
    Evidence of last
    iteration’s work
    code
    docs
    stats
    Retrospectives
    Reflecting on past work and improving
    

    View Slide

34. Related Research Fields
    34
    Mining Software Repositories
    ■ Draw from MSR techniques [Prana, 2018]
    ■ MSR: Focus on large amounts of repositories
    □ “most widely used open source license?” [Dyer, 2013]
    ■ Little research: Few repositories,
    intricate knowledge of creators / users
    [Prana, 2018] Prana, G. A. A., Treude, C., Thung, F., Atapattu, T., & Lo, D. “Categorizing the Content of GitHub
    README Files”. Empirical Software Engineering. 2018. https://doi.org/10.1007/s10664-018-9660-3
    [Dyer, 2013] Dyer, R., Nguyen, H. A., Rajan, H., & Nguyen, T. N. “Boa: A language and infrastructure for
    analyzing ultra-large-scale software repositories”. In Proceedings - International Conference on Software
    Engineering. pp. 422–431. 2013. IEEE.
    

    View Slide

35. Conclusion
    35
    ■ Agile requires teams to make decisions based on
    best information available [Schuh, 2004]
    ■ Definition of “best information” ongoing research challenge
    [Dybå and Dingsøyr, 2008] Dybå, T. and Dingsøyr, T. (2008). Empirical studies of agile software development:
    A systematic review. Information and Software Technology, 50(9-10):833–859.
    [Schuh, 2004] Schuh, P. (2004). Integrating Agile Development In The Real World. Charles River Media, Inc.,
    Rockland, MA, USA, 1st edition.
    

    View Slide

36. Conclusion
    36
    ■ Agile requires teams to make decisions based on
    best information available [Schuh, 2004]
    ■ Definition of “best information” ongoing research challenge
    ■ Individuals can ignore analyses or own preferences due to rules,
    traditions or influence of others [Dybå and Dingsøyr, 2008]
    [Dybå and Dingsøyr, 2008] Dybå, T. and Dingsøyr, T. (2008). Empirical studies of agile software development:
    A systematic review. Information and Software Technology, 50(9-10):833–859.
    [Schuh, 2004] Schuh, P. (2004). Integrating Agile Development In The Real World. Charles River Media, Inc.,
    Rockland, MA, USA, 1st edition.
    

    View Slide

37. Conclusion
    37
    ■ Agile requires teams to make decisions based on
    best information available [Schuh, 2004]
    ■ Definition of “best information” ongoing research challenge
    ■ Individuals can ignore analyses or own preferences due to rules,
    traditions or influence of others [Dybå and Dingsøyr, 2008]
    Data analysis and interpretation by Agile teams (supported by
    context changes) can enable better informed decision
    [Dybå and Dingsøyr, 2008] Dybå, T. and Dingsøyr, T. (2008). Empirical studies of agile software development:
    A systematic review. Information and Software Technology, 50(9-10):833–859.
    [Schuh, 2004] Schuh, P. (2004). Integrating Agile Development In The Real World. Charles River Media, Inc.,
    Rockland, MA, USA, 1st edition.
    

    View Slide

38. Summary
    38
    

    View Slide

39. Image Credits
    39
    In order of appearance
    ■ Data directed Decision Making by H Alberto Gongora from the Noun Project (CC BY 3.0 US)
    ■ Project by useiconic.com from the Noun Project (CC BY 3.0 US)
    ■ Question by unlimicon from the Noun Project (CC BY 3.0 US)
    ■ Business by David Glöckler from the Noun Project (CC BY 3.0 US)
    ■ Agile Development by Bê Sabino from the Noun Project (CC BY 3.0 US)
    ■ Digitization by priyanka from the Noun Project (CC BY 3.0 US)
    ■ Developer by becris from the Noun Project (CC BY 3.0 US)
    ■ Plan by Gregor Cresnar from the Noun Project (CC BY 3.0 US)
    ■ Daily Scrum by Swen-Peter Ekkebus from the Noun Project (CC BY 3.0 US)
    ■ Documents by LUFTFI GANI AL ACHMAD from the Noun Project (CC BY 3.0 US)
    ■ Idea Generation by priyanka from the Noun Project (CC BY 3.0 US)
    

    View Slide