Introducing Continuous Delivery of Mobile Apps in a Corporate Environment: A Case Study

Presentation at RCoSE workshop 2015 in Florence, Italy

Transcript

1. 1
   Sebastian Klepper, Stephan Krusche, Sebastian Peters, Bernd Bruegge and Lukas Alperowitz
   Technische Universität München, Chair for Applied Software Engineering
   Introducing Continuous Delivery of Mobile Apps

   in a Corporate Environment: A Case Study
   23 May 2015
   RCoSE 2015
   Stephan Krusche
   Twitter: @skrusche
   Email: s.krusche@tum.de
   Web: www.skrusche.de
   

   View Slide

2. RCoSE, ICSE’15 - Sebastian Klepper, Stephan Krusche et al. - Introducing Continuous Delivery of Mobile Apps in a Corporate Environment: A Case Study 2
   Digital technologies become key factors for
   companies, in particular mobile computing
   Mobile-Computing is a strong
   driver for digital transformation
   Agile Development is the key to
   an effective realization
   

   View Slide

3. RCoSE, ICSE’15 - Sebastian Klepper, Stephan Krusche et al. - Introducing Continuous Delivery of Mobile Apps in a Corporate Environment: A Case Study
   Rugby’s Release Management Workflow
   3
   Version
   Control
   Server
   Developer
   1
   notify
   upload
   build 5
   download
   6
   Issue
   Tracker
   notify
   store feedback
   10
   release
   4
   Release

   Manager
   checkout, compile
   and test build
   2
   upload feedback
   give
   feedback
   9
   Continuous
   Integration
   Server
   Continuous
   Delivery
   Server
   7
   Device
   8
   inform about
   build status
   commit
   3
   11
   User
   RCoSE 2014
   Evaluation in university projects —> leads to more releases & feedback
   

   View Slide

4. RCoSE, ICSE’15 - Sebastian Klepper, Stephan Krusche et al. - Introducing Continuous Delivery of Mobile Apps in a Corporate Environment: A Case Study
   Situation at Capgemini revealed with interviews
   Continuous integration and delivery of mobile apps in a (heterogeneous) project-
   based organization is complex and expensive and therefore neglected at Capgemini
   4
   Process
   No standardized development process for mobile applications,
   often isolated custom solutions
   Continuity Continuity is missing in the development
   Infrastructure Existing project environments are not sufficient
   Prioritization
   Automatic delivery and testing is neglected in favor of
   development speed
   

   View Slide

5. RCoSE, ICSE’15 - Sebastian Klepper, Stephan Krusche et al. - Introducing Continuous Delivery of Mobile Apps in a Corporate Environment: A Case Study
   Additional Requirements at Capgemini
   • Support of multiple mobile platforms (including cross-platform)
   • High security for business critical applications
   • Consideration of access control and data privacy
   • Support of different project environments
   • Consideration of dependency management
   • Ability to distinguish between user-generated and automatic feedback
   • Ability to manually deliver an application
   ➡Project managers want the ability to tailor the workflow to their own needs
   5
   

   View Slide

6. RCoSE, ICSE’15 - Sebastian Klepper, Stephan Krusche et al. - Introducing Continuous Delivery of Mobile Apps in a Corporate Environment: A Case Study 6
   User
   Developer
   Delivery
   Integration
   Configuration
   Management
   Feedback
   Components of the release management workflow
   

   View Slide

7. RCoSE, ICSE’15 - Sebastian Klepper, Stephan Krusche et al. - Introducing Continuous Delivery of Mobile Apps in a Corporate Environment: A Case Study
   Rugby’s extended and tailorable release management workflow
   7
   Developer
   Test
   2.2
   Delivery Service
   Integration System
   Repository
   Push
   
   changes
   1.1
   Observe/notify
   
   to trigger build
   1.2
   Fetch changes
   1.3
   Upload app
   3.1
   Package
   2.3
   Collect
   
   metrics
   2.4
   Issue Tracker
   Observe changes
   1.5
   Observe/notify
   
   about build
   2.5
   Resolve
   
   dependencies
   1.4
   Update
   
   status
   4.4
   Build
   2.1
   Release Manager
   Send email
   3.2
   Update status
   4.2
   Integration
   Configuration
   Management
   Delivery Feedback
   Send automatic
   
   feedback
   4.1
   User
   Forward email
   3.3
   Send user-generated feedback
   4.3
   Download app
   3.4
   

   View Slide

8. RCoSE, ICSE’15 - Sebastian Klepper, Stephan Krusche et al. - Introducing Continuous Delivery of Mobile Apps in a Corporate Environment: A Case Study
   1) Example of tailored workflow for a larger project
   8
   Developer
   Build
   2.1
   Upload app
   3.1
   Test
   2.2
   Delivery Service User
   Download app
   3.4
   Send email
   3.2
   Integration System
   Repository
   Push
   
   changes
   1.1
   Package
   2.3
   Collect
   
   metrics
   2.4
   Issue Tracker
   Observe changes
   1.5
   Resolve
   
   dependencies
   1.4 Send automatic
   
   feedback
   4.1
   Forward email
   3.3
   Update status
   4.2
   Release
   Manager
   Trigger build
   1.2
   Email Server
   Fetch changes
   1.3
   Notify
   
   about build
   2.5
   Integration
   Configuration
   Management
   Delivery Feedback
   manual build only automatic
   feedback
   automatic
   distribution
   

   View Slide

9. RCoSE, ICSE’15 - Sebastian Klepper, Stephan Krusche et al. - Introducing Continuous Delivery of Mobile Apps in a Corporate Environment: A Case Study
   2) Example of tailored workflow for a smaller project
   9
   Developer
   Build 2.1
   Test 2.2
   Delivery Service
   Integration System
   Repository
   Push
   
   changes
   1.1
   Trigger build
   1.2
   Fetch changes
   1.3
   Upload app 3.1
   Package 2.3
   Send user-generated feedback
   4.3
   Issue Tracker
   Forward email
   3.3
   Update
   
   status
   4.4
   Release Manager
   Send email
   3.2
   User
   Download app
   3.4
   Download app
   3.0
   Integration
   Configuration
   Management
   Delivery Feedback
   no dependency
   management
   manual
   distribution
   only user-
   generated
   feedback
   no metrics
   

   View Slide

10. RCoSE, ICSE’15 - Sebastian Klepper, Stephan Krusche et al. - Introducing Continuous Delivery of Mobile Apps in a Corporate Environment: A Case Study
    Evaluation
    10
    

    View Slide

11. RCoSE, ICSE’15 - Sebastian Klepper, Stephan Krusche et al. - Introducing Continuous Delivery of Mobile Apps in a Corporate Environment: A Case Study
    Research Questions
    1) Integration: How are projects building and testing changes and how often and timely
    are they doing it?
    2) Testing: Which types of tests are projects employing and how are these tests run
    respectively?
    3) Metrics: Which metrics are collected, how are they collected and how is the
    information used?
    4) Delivery: How are projects delivering new builds to users, how much time and effort
    does it cost and how many users can be reached?
    5) Feedback: Which feedback channels are used, how frequently is feedback collected
    and what is the quality of feedback?
    11
    

    View Slide

12. RCoSE, ICSE’15 - Sebastian Klepper, Stephan Krusche et al. - Introducing Continuous Delivery of Mobile Apps in a Corporate Environment: A Case Study
    Study Design
    1) Investigation of 8 projects developing mobile applications (mostly iOS and Android) at
    Capgemini with customers from different industrial sectors
    2) Introduction of Rugby’s extended and tailorable release management workflow
    3) Conduction of personal interviews about the use of the workflow
    • Interviewees with multiple years of experience in the mobile domain
    • Familiar with agile methodologies and continuous integration
    • Little experience with automated delivery
    12
    

    View Slide

13. RCoSE, ICSE’15 - Sebastian Klepper, Stephan Krusche et al. - Introducing Continuous Delivery of Mobile Apps in a Corporate Environment: A Case Study
    Findings: Integration
    • More projects use integration systems
    (88% instead of 25% before)
    • Frequency of integration increased
    • More projects use dependency
    management (75% instead of 13% before)
    ➡Reduced time for integration and
    immediate feedback about the build state
    13
    38%$
    13%$
    38%$
    88%$
    25%$
    0%$
    10%$
    20%$
    30%$
    40%$
    50%$
    60%$
    70%$
    80%$
    90%$
    100%$
    Before$ After$
    Immediately$ Regularly$ Sporadically$
    

    View Slide

14. RCoSE, ICSE’15 - Sebastian Klepper, Stephan Krusche et al. - Introducing Continuous Delivery of Mobile Apps in a Corporate Environment: A Case Study
    Findings: Testing
    • Effort and implementation of tests
    remain unchanged
    • Automation of test cases greatly
    increased
    ➡High time savings and immediate
    visibility of regression failures
    • Acceptance, usability and exploratory
    test execution remain manually
    14
    13%$
    38%$ 38%$
    13%$
    13%$
    38%$
    13%$
    38%$
    13%$
    38%$
    50%$
    63%$
    13%$
    75%$
    38%$
    0%$
    10%$
    20%$
    30%$
    40%$
    50%$
    60%$
    70%$
    80%$
    90%$
    100%$
    Before$After$ Before$After$ Before$After$
    Unit$ Integration$ System$
    Automated$ Semi@automated$ Manual$
    

    View Slide

15. RCoSE, ICSE’15 - Sebastian Klepper, Stephan Krusche et al. - Introducing Continuous Delivery of Mobile Apps in a Corporate Environment: A Case Study
    Findings: Metrics
    Majority of projects makes use of automatic metrics:
    • Static analysis tools provide codebase metrics
    like code quality and test coverage
    ➡ Continuous quality assurance
    • The integration system reports build success
    rate and duration of build and test steps
    ➡ Status monitoring
    • Project metrics (e.g. velocity, cycle time) are
    obtained from the issue tracker
    ➡Decision support
    15
    8%#
    65%#
    75%#
    38%#
    13%#
    5%#
    6%# 8%#
    5%#
    0%#
    10%#
    20%#
    30%#
    40%#
    50%#
    60%#
    70%#
    80%#
    90%#
    100%#
    Before#After# Before#After# Before#After#
    Codebase# Integration# Project#
    Automated# SemiCautomated# Manual#
    

    View Slide

16. RCoSE, ICSE’15 - Sebastian Klepper, Stephan Krusche et al. - Introducing Continuous Delivery of Mobile Apps in a Corporate Environment: A Case Study
    Findings: Delivery
    • About 60% use automated delivery, about
    40% semi-automated delivery (before 0%)
    • Involvement of team members reduced by
    25% (median)
    • Delivery time reduced to 5 min (median)
    ➡High time savings and immediate
    availability of changes in releases
    16
    43% 38%
    57%
    50%
    100% 100%
    13%
    0%
    10%
    20%
    30%
    40%
    50%
    60%
    70%
    80%
    90%
    100%
    Before After Before After
    Internal External
    Automated Semi>automated Manual
    

    View Slide

17. RCoSE, ICSE’15 - Sebastian Klepper, Stephan Krusche et al. - Introducing Continuous Delivery of Mobile Apps in a Corporate Environment: A Case Study
    Findings: Feedback
    • The feedback collection did not significantly change
    ➡Unstructured feedback channels are preferred
    • The quality of the feedback did not significantly change
    ➡Synchronous communication of feedback has the highest quality
    17
    88% 88%
    13%
    50%
    25%
    13%
    38%
    13%
    13%
    25%
    25%
    0%
    10%
    20%
    30%
    40%
    50%
    60%
    70%
    80%
    90%
    100%
    Phone Email Virtual:
    meetings
    Real:meetings Custom:tools Dedicated:
    tools
    At:least:weekly At:least:monthly Less:than:monthly
    25%
    38%
    25%
    50%
    13%
    38%
    75%
    13%
    13%
    38%
    13%
    25%
    25%
    0%
    10%
    20%
    30%
    40%
    50%
    60%
    70%
    80%
    90%
    100%
    Phone Email Virtual:
    meetings
    Real:meetings Custom:tools Dedicated:
    tools
    Excellent Good Acceptable Insufficient Useless
    

    View Slide

18. RCoSE, ICSE’15 - Sebastian Klepper, Stephan Krusche et al. - Introducing Continuous Delivery of Mobile Apps in a Corporate Environment: A Case Study
    Threats to Validity
    • Reliability: Duration of the evaluation period, number of metrics, or level of detail may
    have influence on the reliability of our results
    • Generalizability: Number of projects and variation of project characteristics may be
    too low in order to achieve generalizable results
    • Selection bias: Projects participating in our survey may have already worked in an
    agile fashion and had interest in automated integration and delivery
    • Researcher bias: An appreciation for agile in general or our solution in particular and
    positive results of our previous study may have influenced our evaluation
    18
    

    View Slide

19. RCoSE, ICSE’15 - Sebastian Klepper, Stephan Krusche et al. - Introducing Continuous Delivery of Mobile Apps in a Corporate Environment: A Case Study
    Conclusion
    • We showed how to introduce continuous delivery in a corporate environment where no
    standardized workflow for mobile projects existed before
    • Duration of integration and delivery decreased while their frequency increased
    • Automated testing and metrics collection improved
    • Due to the close evaluation after the introduction of the workflows we could not find
    significant changes in the feedback collection
    ➡The tailorable workflow provides a good compromise between technical promises
    and actual business needs
    ➡The value of the introduced workflow is appreciated, especially low time and effort
    required for setup that allows projects to get started quickly
    19
    

    View Slide

20. RCoSE, ICSE’15 - Sebastian Klepper, Stephan Krusche et al. - Introducing Continuous Delivery of Mobile Apps in a Corporate Environment: A Case Study
    Outlook
    • Rugby’s components have the potential for highly complex and individual setups, e.g.
    • Finer control over the build process with build matrices and branch filters
    • Automated user interface tests
    • Integration tests with a backend built in parallel
    • Longer observation period in the evaluation
    • Improve the incorporation of user needs
    • Evaluate whether development behavior and feedback change over time
    20
    

    View Slide

21. 21
    Introducing Continuous Delivery of Mobile Apps

    in a Corporate Environment: A Case Study
    Thank you!
    Sebastian Klepper, Stephan Krusche, Sebastian Peters, Bernd Bruegge and Lukas Alperowitz
    Technische Universität München, Chair for Applied Software Engineering
    RCoSE 2015
    Stephan Krusche
    Twitter: @skrusche
    Email: s.krusche@tum.de
    Web: www.skrusche.de
    

    View Slide

22. RCoSE, ICSE’15 - Sebastian Klepper, Stephan Krusche et al. - Introducing Continuous Delivery of Mobile Apps in a Corporate Environment: A Case Study
    References (1)
    (1) M. Fitzgerald, N. Kruschwitz, D. Bonnet, and M. Welch, “Embracing digital technology: A new strategic imperative,” MIT Sloan Management Review
    in collaboration with Capgemini Consulting, 2013.
    (2) S. Khalaf, “Flurry analytics insights,” 2015, retrieved February 26, 2015 from http://www.flurry.com/blog/flurry-insights.
    (3) S. Krusche and B. Bruegge, “User feedback in mobile development,” in Proceedings of the 2nd International Workshop on Mobile Development
    Lifecycle. ACM, 2014, pp. 25–26.
    (4) D. Pagano and W. Maalej, “User feedback in the appstore: An empirical study,” in Proceedings of the 21st International Requirements Engineering
    Conference. IEEE, 2013, pp. 125–134.
    (5) D. Pagano and B. Bruegge, “User involvement in software evolution practice: a case study,” in Proceedings of the 2013 international conference on
    Software engineering. IEEE, 2013, pp. 953–962.
    (6) K. Schwaber and M. Beedle, Agile software development with Scrum. Prentice Hall, 2002.
    (7) VersionOne, “8th annual state of agile survey,” 2014, http://www.versionone.com/pdf/2013-state-of-agile-survey.pdf.
    (8) R. Pichler, Agile product management with scrum: Creating products that customers love. Addison-Wesley Professional, 2010.
    (9) J. Bosch, “Continuous software engineering: An introduction,” in Continuous Software Engineering. Springer, 2014, pp. 3–13.
    (10) J. Humble and D. Farley, Continuous Delivery: Reliable Software Releases through Build, Test, and Deployment Automation. Addison Wesley, 2010.
    (11) C. Handy, Understanding organizations: managing differentiation and integration. Oxford University Press, 1993.
    (12) W. Brown, H. McCormick, T. Mowbray, and R. Malveau, AntiPatterns: refactoring software, architectures, and projects in crisis. Wiley, 1998.
    (13) O. Pedreira, M. Piattini, M. Luaces, and N. Brisaboa, “A systematic review of software process tailoring,” SIGSOFT SE Notes, vol. 32, no. 3, pp. 1–6,
    2007.
    (14) S. Krusche, L. Alperowitz, B. Bruegge, and M. Wagner, “Rugby: An Agile Process Model Based on Continuous Delivery,” in Proceedings of the 1st
    International Workshop on RCoSE. ACM, 2014, pp. 42–50.
    (15) G. Kalus and M. Kuhrmann, “Criteria for software process tailoring: A systematic review,” in Proceedings of ICSSP. ACM, 2013, pp. 171–180.
    22
    

    View Slide

23. RCoSE, ICSE’15 - Sebastian Klepper, Stephan Krusche et al. - Introducing Continuous Delivery of Mobile Apps in a Corporate Environment: A Case Study
    References (2)
    (16) I. Jacobson, G. Booch, J. Rumbaugh, J. Rumbaugh, and G. Booch, The unified software development process. Addison-Wesley, 1999.
    (17) B. Bruegge, S. Krusche, and M. Wagner, “Teaching Tornado: from communication models to releases” in Proceedings of the 8th edition of
    the Educators’ Symposium. ACM, 2012, pp. 5–12.
    (18) J. Feller, Perspectives on free and open source software. MIT, 2005.
    (19) L. Zhao and S. Elbaum, “A survey on quality related activities in open source,” SIGSOFT Software Engineering Notes, vol. 25, no. 3, pp. 54–
    57, 2000.
    (20) M. Aberdour, “Achieving quality in open-source software,” Software, vol. 24, no. 1, pp. 58–64, 2007.
    (21) S. Krusche and L. Alperowitz, “Introduction of Continuous Delivery in Multi-Customer Project Courses,” in Proceedings of the 36th ICSE.
    IEEE, 2014, pp. 335–343.
    (22) M. Poppendiek and T. Poppendiek, Implementing Lean Software Development: From Concept to Cash. Addison Wesley, 2006.
    (23) L. Van Audenhove, “Expert interviews and interview techniques for policy analysis,” Vrije Universiteit Brussel, 2013.
    (24) P. Duvall, S. Matyas, and A. Glover, Continuous integration: improving software quality and reducing risk. Pearson Education, 2007.
    (25) L. Crispin and J. Gregory, Agile Testing: A Practical Guide for Testers and Agile Teams. Addison Wesley, 2009.
    (26) B.Marick,“Agile testing directions: tests and examples,”2003,retrieved February 26, 2015 from http://www.exampler.com/old-blog/2003/08/22.
    (27) N. Fenton and J. Bieman, Software metrics: a rigorous and practical approach, 3rd ed. CRC Press, 2014.
    (28) M. Lorenz and J. Kidd, Object-oriented software metrics: a practical guide. Prentice-Hall, 1994.
    (29) M. Poppendiek and T. Poppendiek, Lean Software Development: An Agile Toolkit. Addison Wesley, 2003.
    (30) D. Willimack et al., “Evolution and adaption of questionnaire development, evaluation, and testing methods for establishment surveys,”
    Methods for Testing and Evaluating Questionnaires, pp. 385–407, 2004.
    (31) P. Gorans and P. Kruchten, “A guide to critical success factors in agile delivery,” IBM Center for The Business of Government, 2014.
    23
    

    View Slide