Introducing Continuous Delivery of Mobile Apps in a Corporate Environment: A Case Study

Presentation at RCoSE workshop 2015 in Florence, Italy

Transcript

1. 1 Sebastian Klepper, Stephan Krusche, Sebastian Peters, Bernd Bruegge and Lukas Alperowitz Technische Universität München, Chair for Applied Software Engineering Introducing Continuous Delivery of Mobile Apps
 in a Corporate Environment: A Case Study 23 May 2015 RCoSE 2015 Stephan Krusche Twitter: @skrusche Email: s.krusche@tum.de Web: www.skrusche.de
2. RCoSE, ICSE’15 - Sebastian Klepper, Stephan Krusche et al. - Introducing Continuous Delivery of Mobile Apps in a Corporate Environment: A Case Study 2 Digital technologies become key factors for companies, in particular mobile computing Mobile-Computing is a strong driver for digital transformation Agile Development is the key to an effective realization
3. RCoSE, ICSE’15 - Sebastian Klepper, Stephan Krusche et al. - Introducing Continuous Delivery of Mobile Apps in a Corporate Environment: A Case Study Rugby’s Release Management Workflow 3 Version Control Server Developer 1 notify upload build 5 download 6 Issue Tracker notify store feedback 10 release 4 Release
 Manager checkout, compile and test build 2 upload feedback give feedback 9 Continuous Integration Server Continuous Delivery Server 7 Device 8 inform about build status commit 3 11 User RCoSE 2014 Evaluation in university projects —> leads to more releases & feedback
4. RCoSE, ICSE’15 - Sebastian Klepper, Stephan Krusche et al. - Introducing Continuous Delivery of Mobile Apps in a Corporate Environment: A Case Study Situation at Capgemini revealed with interviews Continuous integration and delivery of mobile apps in a (heterogeneous) project- based organization is complex and expensive and therefore neglected at Capgemini 4 Process No standardized development process for mobile applications, often isolated custom solutions Continuity Continuity is missing in the development Infrastructure Existing project environments are not sufficient Prioritization Automatic delivery and testing is neglected in favor of development speed
5. RCoSE, ICSE’15 - Sebastian Klepper, Stephan Krusche et al. - Introducing Continuous Delivery of Mobile Apps in a Corporate Environment: A Case Study Additional Requirements at Capgemini • Support of multiple mobile platforms (including cross-platform) • High security for business critical applications • Consideration of access control and data privacy • Support of different project environments • Consideration of dependency management • Ability to distinguish between user-generated and automatic feedback • Ability to manually deliver an application ➡Project managers want the ability to tailor the workflow to their own needs 5
6. RCoSE, ICSE’15 - Sebastian Klepper, Stephan Krusche et al. - Introducing Continuous Delivery of Mobile Apps in a Corporate Environment: A Case Study 6 User Developer Delivery Integration Configuration Management Feedback Components of the release management workflow
7. RCoSE, ICSE’15 - Sebastian Klepper, Stephan Krusche et al. - Introducing Continuous Delivery of Mobile Apps in a Corporate Environment: A Case Study Rugby’s extended and tailorable release management workflow 7 Developer Test 2.2 Delivery Service Integration System Repository Push changes 1.1 Observe/notify to trigger build 1.2 Fetch changes 1.3 Upload app 3.1 Package 2.3 Collect metrics 2.4 Issue Tracker Observe changes 1.5 Observe/notify about build 2.5 Resolve dependencies 1.4 Update status 4.4 Build 2.1 Release Manager Send email 3.2 Update status 4.2 Integration Configuration Management Delivery Feedback Send automatic feedback 4.1 User Forward email 3.3 Send user-generated feedback 4.3 Download app 3.4
8. RCoSE, ICSE’15 - Sebastian Klepper, Stephan Krusche et al. - Introducing Continuous Delivery of Mobile Apps in a Corporate Environment: A Case Study 1) Example of tailored workflow for a larger project 8 Developer Build 2.1 Upload app 3.1 Test 2.2 Delivery Service User Download app 3.4 Send email 3.2 Integration System Repository Push changes 1.1 Package 2.3 Collect metrics 2.4 Issue Tracker Observe changes 1.5 Resolve dependencies 1.4 Send automatic feedback 4.1 Forward email 3.3 Update status 4.2 Release Manager Trigger build 1.2 Email Server Fetch changes 1.3 Notify about build 2.5 Integration Configuration Management Delivery Feedback manual build only automatic feedback automatic distribution
9. RCoSE, ICSE’15 - Sebastian Klepper, Stephan Krusche et al. - Introducing Continuous Delivery of Mobile Apps in a Corporate Environment: A Case Study 2) Example of tailored workflow for a smaller project 9 Developer Build 2.1 Test 2.2 Delivery Service Integration System Repository Push changes 1.1 Trigger build 1.2 Fetch changes 1.3 Upload app 3.1 Package 2.3 Send user-generated feedback 4.3 Issue Tracker Forward email 3.3 Update status 4.4 Release Manager Send email 3.2 User Download app 3.4 Download app 3.0 Integration Configuration Management Delivery Feedback no dependency management manual distribution only user- generated feedback no metrics
10. RCoSE, ICSE’15 - Sebastian Klepper, Stephan Krusche et al. - Introducing Continuous Delivery of Mobile Apps in a Corporate Environment: A Case Study Evaluation 10
11. RCoSE, ICSE’15 - Sebastian Klepper, Stephan Krusche et al. - Introducing Continuous Delivery of Mobile Apps in a Corporate Environment: A Case Study Research Questions 1) Integration: How are projects building and testing changes and how often and timely are they doing it? 2) Testing: Which types of tests are projects employing and how are these tests run respectively? 3) Metrics: Which metrics are collected, how are they collected and how is the information used? 4) Delivery: How are projects delivering new builds to users, how much time and effort does it cost and how many users can be reached? 5) Feedback: Which feedback channels are used, how frequently is feedback collected and what is the quality of feedback? 11
12. RCoSE, ICSE’15 - Sebastian Klepper, Stephan Krusche et al. - Introducing Continuous Delivery of Mobile Apps in a Corporate Environment: A Case Study Study Design 1) Investigation of 8 projects developing mobile applications (mostly iOS and Android) at Capgemini with customers from different industrial sectors 2) Introduction of Rugby’s extended and tailorable release management workflow 3) Conduction of personal interviews about the use of the workflow • Interviewees with multiple years of experience in the mobile domain • Familiar with agile methodologies and continuous integration • Little experience with automated delivery 12
13. RCoSE, ICSE’15 - Sebastian Klepper, Stephan Krusche et al. - Introducing Continuous Delivery of Mobile Apps in a Corporate Environment: A Case Study Findings: Integration • More projects use integration systems (88% instead of 25% before) • Frequency of integration increased • More projects use dependency management (75% instead of 13% before) ➡Reduced time for integration and immediate feedback about the build state 13 38%$ 13%$ 38%$ 88%$ 25%$ 0%$ 10%$ 20%$ 30%$ 40%$ 50%$ 60%$ 70%$ 80%$ 90%$ 100%$ Before$ After$ Immediately$ Regularly$ Sporadically$
14. RCoSE, ICSE’15 - Sebastian Klepper, Stephan Krusche et al. - Introducing Continuous Delivery of Mobile Apps in a Corporate Environment: A Case Study Findings: Testing • Effort and implementation of tests remain unchanged • Automation of test cases greatly increased ➡High time savings and immediate visibility of regression failures • Acceptance, usability and exploratory test execution remain manually 14 13%$ 38%$ 38%$ 13%$ 13%$ 38%$ 13%$ 38%$ 13%$ 38%$ 50%$ 63%$ 13%$ 75%$ 38%$ 0%$ 10%$ 20%$ 30%$ 40%$ 50%$ 60%$ 70%$ 80%$ 90%$ 100%$ Before$After$ Before$After$ Before$After$ Unit$ Integration$ System$ Automated$ Semi@automated$ Manual$
15. RCoSE, ICSE’15 - Sebastian Klepper, Stephan Krusche et al. - Introducing Continuous Delivery of Mobile Apps in a Corporate Environment: A Case Study Findings: Metrics Majority of projects makes use of automatic metrics: • Static analysis tools provide codebase metrics like code quality and test coverage ➡ Continuous quality assurance • The integration system reports build success rate and duration of build and test steps ➡ Status monitoring • Project metrics (e.g. velocity, cycle time) are obtained from the issue tracker ➡Decision support 15 8%# 65%# 75%# 38%# 13%# 5%# 6%# 8%# 5%# 0%# 10%# 20%# 30%# 40%# 50%# 60%# 70%# 80%# 90%# 100%# Before#After# Before#After# Before#After# Codebase# Integration# Project# Automated# SemiCautomated# Manual#
16. RCoSE, ICSE’15 - Sebastian Klepper, Stephan Krusche et al. - Introducing Continuous Delivery of Mobile Apps in a Corporate Environment: A Case Study Findings: Delivery • About 60% use automated delivery, about 40% semi-automated delivery (before 0%) • Involvement of team members reduced by 25% (median) • Delivery time reduced to 5 min (median) ➡High time savings and immediate availability of changes in releases 16 43% 38% 57% 50% 100% 100% 13% 0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100% Before After Before After Internal External Automated Semi>automated Manual
17. RCoSE, ICSE’15 - Sebastian Klepper, Stephan Krusche et al. - Introducing Continuous Delivery of Mobile Apps in a Corporate Environment: A Case Study Findings: Feedback • The feedback collection did not significantly change ➡Unstructured feedback channels are preferred • The quality of the feedback did not significantly change ➡Synchronous communication of feedback has the highest quality 17 88% 88% 13% 50% 25% 13% 38% 13% 13% 25% 25% 0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100% Phone Email Virtual: meetings Real:meetings Custom:tools Dedicated: tools At:least:weekly At:least:monthly Less:than:monthly 25% 38% 25% 50% 13% 38% 75% 13% 13% 38% 13% 25% 25% 0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100% Phone Email Virtual: meetings Real:meetings Custom:tools Dedicated: tools Excellent Good Acceptable Insufficient Useless
18. RCoSE, ICSE’15 - Sebastian Klepper, Stephan Krusche et al. - Introducing Continuous Delivery of Mobile Apps in a Corporate Environment: A Case Study Threats to Validity • Reliability: Duration of the evaluation period, number of metrics, or level of detail may have influence on the reliability of our results • Generalizability: Number of projects and variation of project characteristics may be too low in order to achieve generalizable results • Selection bias: Projects participating in our survey may have already worked in an agile fashion and had interest in automated integration and delivery • Researcher bias: An appreciation for agile in general or our solution in particular and positive results of our previous study may have influenced our evaluation 18
19. RCoSE, ICSE’15 - Sebastian Klepper, Stephan Krusche et al. - Introducing Continuous Delivery of Mobile Apps in a Corporate Environment: A Case Study Conclusion • We showed how to introduce continuous delivery in a corporate environment where no standardized workflow for mobile projects existed before • Duration of integration and delivery decreased while their frequency increased • Automated testing and metrics collection improved • Due to the close evaluation after the introduction of the workflows we could not find significant changes in the feedback collection ➡The tailorable workflow provides a good compromise between technical promises and actual business needs ➡The value of the introduced workflow is appreciated, especially low time and effort required for setup that allows projects to get started quickly 19
20. RCoSE, ICSE’15 - Sebastian Klepper, Stephan Krusche et al. - Introducing Continuous Delivery of Mobile Apps in a Corporate Environment: A Case Study Outlook • Rugby’s components have the potential for highly complex and individual setups, e.g. • Finer control over the build process with build matrices and branch filters • Automated user interface tests • Integration tests with a backend built in parallel • Longer observation period in the evaluation • Improve the incorporation of user needs • Evaluate whether development behavior and feedback change over time 20
21. 21 Introducing Continuous Delivery of Mobile Apps
 in a Corporate Environment: A Case Study Thank you! Sebastian Klepper, Stephan Krusche, Sebastian Peters, Bernd Bruegge and Lukas Alperowitz Technische Universität München, Chair for Applied Software Engineering RCoSE 2015 Stephan Krusche Twitter: @skrusche Email: s.krusche@tum.de Web: www.skrusche.de
22. RCoSE, ICSE’15 - Sebastian Klepper, Stephan Krusche et al. - Introducing Continuous Delivery of Mobile Apps in a Corporate Environment: A Case Study References (1) (1) M. Fitzgerald, N. Kruschwitz, D. Bonnet, and M. Welch, “Embracing digital technology: A new strategic imperative,” MIT Sloan Management Review in collaboration with Capgemini Consulting, 2013. (2) S. Khalaf, “Flurry analytics insights,” 2015, retrieved February 26, 2015 from http://www.flurry.com/blog/flurry-insights. (3) S. Krusche and B. Bruegge, “User feedback in mobile development,” in Proceedings of the 2nd International Workshop on Mobile Development Lifecycle. ACM, 2014, pp. 25–26. (4) D. Pagano and W. Maalej, “User feedback in the appstore: An empirical study,” in Proceedings of the 21st International Requirements Engineering Conference. IEEE, 2013, pp. 125–134. (5) D. Pagano and B. Bruegge, “User involvement in software evolution practice: a case study,” in Proceedings of the 2013 international conference on Software engineering. IEEE, 2013, pp. 953–962. (6) K. Schwaber and M. Beedle, Agile software development with Scrum. Prentice Hall, 2002. (7) VersionOne, “8th annual state of agile survey,” 2014, http://www.versionone.com/pdf/2013-state-of-agile-survey.pdf. (8) R. Pichler, Agile product management with scrum: Creating products that customers love. Addison-Wesley Professional, 2010. (9) J. Bosch, “Continuous software engineering: An introduction,” in Continuous Software Engineering. Springer, 2014, pp. 3–13. (10) J. Humble and D. Farley, Continuous Delivery: Reliable Software Releases through Build, Test, and Deployment Automation. Addison Wesley, 2010. (11) C. Handy, Understanding organizations: managing differentiation and integration. Oxford University Press, 1993. (12) W. Brown, H. McCormick, T. Mowbray, and R. Malveau, AntiPatterns: refactoring software, architectures, and projects in crisis. Wiley, 1998. (13) O. Pedreira, M. Piattini, M. Luaces, and N. Brisaboa, “A systematic review of software process tailoring,” SIGSOFT SE Notes, vol. 32, no. 3, pp. 1–6, 2007. (14) S. Krusche, L. Alperowitz, B. Bruegge, and M. Wagner, “Rugby: An Agile Process Model Based on Continuous Delivery,” in Proceedings of the 1st International Workshop on RCoSE. ACM, 2014, pp. 42–50. (15) G. Kalus and M. Kuhrmann, “Criteria for software process tailoring: A systematic review,” in Proceedings of ICSSP. ACM, 2013, pp. 171–180. 22
23. RCoSE, ICSE’15 - Sebastian Klepper, Stephan Krusche et al. - Introducing Continuous Delivery of Mobile Apps in a Corporate Environment: A Case Study References (2) (16) I. Jacobson, G. Booch, J. Rumbaugh, J. Rumbaugh, and G. Booch, The unified software development process. Addison-Wesley, 1999. (17) B. Bruegge, S. Krusche, and M. Wagner, “Teaching Tornado: from communication models to releases” in Proceedings of the 8th edition of the Educators’ Symposium. ACM, 2012, pp. 5–12. (18) J. Feller, Perspectives on free and open source software. MIT, 2005. (19) L. Zhao and S. Elbaum, “A survey on quality related activities in open source,” SIGSOFT Software Engineering Notes, vol. 25, no. 3, pp. 54– 57, 2000. (20) M. Aberdour, “Achieving quality in open-source software,” Software, vol. 24, no. 1, pp. 58–64, 2007. (21) S. Krusche and L. Alperowitz, “Introduction of Continuous Delivery in Multi-Customer Project Courses,” in Proceedings of the 36th ICSE. IEEE, 2014, pp. 335–343. (22) M. Poppendiek and T. Poppendiek, Implementing Lean Software Development: From Concept to Cash. Addison Wesley, 2006. (23) L. Van Audenhove, “Expert interviews and interview techniques for policy analysis,” Vrije Universiteit Brussel, 2013. (24) P. Duvall, S. Matyas, and A. Glover, Continuous integration: improving software quality and reducing risk. Pearson Education, 2007. (25) L. Crispin and J. Gregory, Agile Testing: A Practical Guide for Testers and Agile Teams. Addison Wesley, 2009. (26) B.Marick,“Agile testing directions: tests and examples,”2003,retrieved February 26, 2015 from http://www.exampler.com/old-blog/2003/08/22. (27) N. Fenton and J. Bieman, Software metrics: a rigorous and practical approach, 3rd ed. CRC Press, 2014. (28) M. Lorenz and J. Kidd, Object-oriented software metrics: a practical guide. Prentice-Hall, 1994. (29) M. Poppendiek and T. Poppendiek, Lean Software Development: An Agile Toolkit. Addison Wesley, 2003. (30) D. Willimack et al., “Evolution and adaption of questionnaire development, evaluation, and testing methods for establishment surveys,” Methods for Testing and Evaluating Questionnaires, pp. 385–407, 2004. (31) P. Gorans and P. Kruchten, “A guide to critical success factors in agile delivery,” IBM Center for The Business of Government, 2014. 23