Toward supporting DevOps during Continuous Software Development

Slide of my talk at the 10th DevOps Enthusiast Meetup in Zurich (Switzerland).

Abstract: Continuous Integration (CI) and Continuous Delivery (CD) are widespread in both industrial and open-source software (OSS) projects. We investigated the successful adoption of CD in a large financial organisation (ING Nederland) and the increasing number of OSS developers that rely on a CD pipeline to build their software applications. However, despite the claimed advantages of Continuous Delivery (e.g. reliable and fast release), its core part (i.e. CI) can be broken by trivial build failures. With the aim of understanding the types of build failure affecting both open and closed source software we categorised the build failures occurred in 349 OSS projects and 418 ING projects. On one hand the results revealed that OSS and ING projects exhibit substantially different distributions of build failure types. On the other hand our study pointed out that DevOps need a better support during CD breaks and measures for preventing CD degradation.

Transcript

1. Toward supporting devops during continuous software development Carmine Vassallo
2. DevOps represents a change in IT culture, focusing on rapid IT service delivery through the adoption of agile, lean practices in the context of a system-oriented approach. DevOps emphasizes people (and culture), and seeks to improve collaboration between operations and development teams. DevOps implementations utilize technology — especially automation tools that can leverage an increasingly programmable and dynamic infrastructure from a life cycle perspective. — gartner.com
3. DevOps represents a change in IT culture, focusing on rapid IT service delivery through the adoption of agile, lean practices in the context of a system-oriented approach. DevOps emphasizes people (and culture), and seeks to improve collaboration between operations and development teams. DevOps implementations utilize technology — especially automation tools that can leverage an increasingly programmable and dynamic infrastructure from a life cycle perspective. — gartner.com
4. DevOps represents a change in IT culture, focusing on rapid IT service delivery through the adoption of agile, lean practices in the context of a system-oriented approach. DevOps emphasizes people (and culture), and seeks to improve collaboration between operations and development teams. DevOps implementations utilize technology — especially automation tools that can leverage an increasingly programmable and dynamic infrastructure from a life cycle perspective. — gartner.com
5. DevOps represents a change in IT culture, focusing on rapid IT service delivery through the adoption of agile, lean practices in the context of a system-oriented approach. DevOps emphasizes people (and culture), and seeks to improve collaboration between operations and development teams. DevOps implementations utilize technology — especially automation tools that can leverage an increasingly programmable and dynamic infrastructure from a life cycle perspective. — gartner.com
6. DevOps represents a change in IT culture, focusing on rapid IT service delivery through the adoption of agile, lean practices in the context of a system-oriented approach. DevOps emphasizes people (and culture), and seeks to improve collaboration between operations and development teams. DevOps implementations utilize technology — especially automation tools that can leverage an increasingly programmable and dynamic infrastructure from a life cycle perspective. — gartner.com
8. “Implementing automation is definitely a part of what it means to be practicing DevOps, but it’s maybe 5-10%.” process people (Automation is not DevOps, Chris Dodds)
9. Who am I? • My name is Carmine Vassallo • Research Intern in the Continuous Delivery team at ING Nederland (2015) • PhD Student in the Seal group at the University of Zurich (Since September 2016) • My current research focus is on supporting DevOps while dealing with CD breaks (e.g., build failures) and developing new approaches to prevent stop in the pipeline. http://www.ifi.uzh.ch/en/seal/people/vassallo.html @ccvassallo vassallo@ifi.uzh.ch
10. Continuous Delivery is a software development discipline where you build software in such a way that the software can be released to production at any time. (Martin Fowler)
11. Reliable Pipeline
12. Early Feedback
13. Fast Release
14. Case Study at ING Nederland • We surveyed 152 ING NL DevOps to investigate their development practices. • In three years, from 2011 to 2013, ING NL has increased the number of delivered function points by 300% and reduced the cost of a single function point to one third. • Additionally, between 2012 and 2014, the release frequency has doubled, reaching one release every four days. Carmine Vassallo, Fiorella Zampetti, Daniele Romano, Moritz Beller, Annibale Panichella, Massimiliano Di Penta, Andy Zaidman, Continuous Delivery Practice in a Large Financial Organisation. International Conference on Software Maintenance and Evolution (ICSME), Raleigh (NC, USA), September, 2016
15. Almost always Often Sometimes Seldom Almost never % of respondents 0% 10% 20% 30% 40% 50% 5% 12% 37% 30% 16% How frequently are refactoring tasks included in other tasks? ING DevOps tend to “self-admit” technical debt
16. Always When I have time Only for certain kinds of systems No % of respondents 0% 10% 20% 30% 40% 50% 22% 12% 33% 34% Do you use TDD (Test Driven Development)? The majority of respondents declare to use TDD
17. Continuous Delivery is a software development discipline where you build software in such a way that the software can be released to production at any time. (Martin Fowler)
18. Continuous Delivery is a software development discipline where you build software in such a way that the software can be released to production at any time. (Martin Fowler)
19. Source Deploy Testing Build (CI) Continuous Delivery pipeline Continuous Integration
20. Build Failures
21. • On average, a build failure takes 57 min to fix. • The overall cost of build failures ranging from 904.64 to 2034.92 man-hours (over a period of 6 months). • They monitored roughly 7200 man-hours. . . Kerzazi, F. Khomh, and B. Adams, Why do automated builds break? an empirical study, in 30th IEEE International Conference on Software Maintenance and Evolution (ICSME), pp. 41–50, IEEE, 2014. 
 Relevance of Build Breakage
22. • When builds fail, due to compilation errors, it requires programmers to take extra time and brainpower to find and fix the problem, reducing their productivity. • A better understanding of the cause of frequent software build errors, then, could help lead to new or improved development tools that would reduce these errors and increase developer output. H. Seo, C. Sadowski, S. Elbaum, E. Aftandilian, and R. Bowdidge, Programmers’ build errors: A case study (at Google), in Proc. Int’l Conference on Software Engineering (ICSE), pp. 724–734, ACM, 2014. Relevance of Build Breakage
23. “Oops my build has failed…” How to support DevOps during build fixing?
24. Open Source (OSS) Closed Source (Industry)
25. Toward supporting devops during continuous software development
26. What types of build failures stop Continuous Delivery?
27. What types of build failures stop Continuous Delivery? A case study in OSS and ING projects
28. RQ1 What types of failures affect builds of OSS and ING Projects? RQ2 How frequent are the different types of build failures in the observed OSS and ING Projects? Carmine Vassallo, Gerald Schermann, Fiorella Zampetti, Daniele Romano, Philipp Leitner, Andy Zaidman, Massimiliano Di Penta, and Sebastiano Panichella, A Tale of CI Build Failures: an Open Source and a Financial Organisation Perspective. In Proceedings of the 33rd International Conference on Software Maintenance and Evolution (ICSME), Shanghai, China, 2017.
29. 1) Data Selection 2) Data Extraction 3) Build Failure Catalog Keyword identification Failed Goal Keyword grouping Failed Goals grouped into categories Build Log classification Each Build Failure Log belongs to at least one category Result Validation Results compared to the ones of the manual classification ! " # $ % 349 Maven Projects (using Travis CI) 418 Maven Projects (using Jenkins) 34,182 Build Failure Logs Study Procedure
30. RQ1 What type of failures affect builds of OSS and ING Projects? [ERROR] Failed to execute goal org.apache.maven.plugins:maven -surefire-plugin:2.12.4:test (default-test) on project openwayback-core: There are test failures. [ERROR] Failed to execute goal org.apache.maven.plugins:maven -compiler-plugin:2.3.2:compile (default-compile) on project version: Compilation failure [ERROR] Failed to execute goal on project barge-core: Could not resolve dependencies for project org.robotninjas.barge:barge-core:jar:0.1.0-SNAPSHOT: Could not find artifact org.robotninjas:netty-protobuf- rpc:jar:2.0.0-SNAPSHOT in sonatype-snapshots [ERROR] Failed to execute goal org.apache.maven.plugins:maven-release-plugin: 2.5.1:prepare (default-cli) on project gTelephonyAPI: You don't have a SNAPSHOT project in the reactor projects list. Testing (Unit) Compilation (Production) Dependencies Release Preparation
31. CLEAN VALIDATION PRE-PROCESSING (RESOURCES) COMPILATION TESTING PACKAGING DOCUMENTATION SUPPORT EXTERNAL TASKS CODE ANALYSIS RELEASE PREPARATION DEPLOYMENT DEPENDENCIES PRODUCTION TEST SUPPORT UNIT TESTING INTEGRATION TESTING NON FUNCTIONAL TESTING CROSSCUTTING STATIC DYNAMIC LOCAL REMOTE RQ1 What type of failures affect builds of OSS and ING Projects?
32. RQ1 What type of failures affect builds of OSS and ING Projects? • Inter-rater agreement • ING: k=0.8 (strong agreement) • OSS: k=0.62 (strong agreement)
33. Clean Validation Preprocessing Compilation (production) Compilation (test) Compilation (support) Testing (unit) Testing (integration) Testing (non functional) Testing (crosscutting) Packaging Static Analysis Dynamic Analysis Deployment (Local) Deployment (Remote) Documentation Release Preparation Support External Tasks Dependencies % of build failures 0% 5% 10% 15% 20% 25% 7.1% 1.4% 0.9% 0.0% 0.9% 0.5% 0.0% 0.2% 4.2% 0.8% 8.3% 0.0% 5.0% 28.0% 0.2% 1.8% 7.1% 1.3% 0.5% 0.0% 6.3% 8.8% 0.0% 21.1% 0.3% 10.0% 0.4% 0.0% 16.4% 2.1% 18.3% 2.7% 13.3% 5.2% 0.0% 2.3% 4.2% 0.0% 0.0% 0.0% Org OSS RQ2 How frequent are the different types of build failures in the observed OSS and ING Projects? Ing
34. Clean Validation Preprocessing Compilation (production) Compilation (test) Compilation (support) Testing (unit) Testing (integration) Testing (non functional) Testing (crosscutting) Packaging Static Analysis Dynamic Analysis 0.2% 4.2% 0.8% 8.3% 0.0% 5.0% 28.0% 0.2% 1.8% 7.1% 1.3% 0.5% 0.0% 0.4% 0.0% 16.4% 2.1% 18.3% 2.7% 13.3% 5.2% 0.0% 2.3% 4.2% 0.0% 0.0% 0.0% Org OSS Clean Validation Preprocessing Compilation (production) Compilation (test) Compilation (support) Testing (unit) Testing (integration) Testing (non functional) Testing (crosscutting) Packaging Static Analysis Dynamic Analysis Deployment (Local) Org OSS Ing Testing (non functional) Testing (crosscutting) Packaging Static Analysis Dynamic Analysis Deployment (Local) Deployment (Remote) Documentation Release Preparation Support External Tasks Dependencies % of build failures 0% 5% 10% 15% 20% 25% Compilation Errors are typically fixed in private builds
35. Clean Validation Preprocessing Compilation (production) Compilation (test) Compilation (support) Testing (unit) Testing (integration) Testing (non functional) Testing (crosscutting) Packaging Static Analysis Dynamic Analysis 0.2% 4.2% 0.8% 8.3% 0.0% 5.0% 28.0% 0.2% 1.8% 7.1% 1.3% 0.5% 0.0% 0.4% 0.0% 16.4% 2.1% 18.3% 2.7% 13.3% 5.2% 0.0% 2.3% 4.2% 0.0% 0.0% 0.0% Org OSS OSS projects run every test on CI servers, ING mostly integration testing Clean Validation Preprocessing Compilation (production) Compilation (test) Compilation (support) Testing (unit) Testing (integration) Testing (non functional) Testing (crosscutting) Packaging Static Analysis Dynamic Analysis Deployment (Local) Org OSS Ing Testing (non functional) Testing (crosscutting) Packaging Static Analysis Dynamic Analysis Deployment (Local) Deployment (Remote) Documentation Release Preparation Support External Tasks Dependencies % of build failures 0% 5% 10% 15% 20% 25%
36. Clean Validation Preprocessing Compilation (production) Compilation (test) Compilation (support) Testing (unit) Testing (integration) Testing (non functional) Testing (crosscutting) Packaging Static Analysis Dynamic Analysis 0.2% 4.2% 0.8% 8.3% 0.0% 5.0% 28.0% 0.2% 1.8% 7.1% 1.3% 0.5% 0.0% 0.4% 0.0% 16.4% 2.1% 18.3% 2.7% 13.3% 5.2% 0.0% 2.3% 4.2% 0.0% 0.0% 0.0% Org OSS Early discovery of non-functional failures in ING Clean Validation Preprocessing Compilation (production) Compilation (test) Compilation (support) Testing (unit) Testing (integration) Testing (non functional) Testing (crosscutting) Packaging Static Analysis Dynamic Analysis Deployment (Local) Org OSS Ing Testing (non functional) Testing (crosscutting) Packaging Static Analysis Dynamic Analysis Deployment (Local) Deployment (Remote) Documentation Release Preparation Support External Tasks Dependencies % of build failures 0% 5% 10% 15% 20% 25%
37. Testing (non functional) Testing (crosscutting) Packaging Static Analysis Dynamic Analysis Deployment (Local) Deployment (Remote) Documentation Release Preparation Support External Tasks Dependencies % of build failures 0% 5% 10% 15% 20% 25% 7.1% 1.4% 0.9% 0.0% 0.9% 0.5% 0.0% 0.2% 4.2% 0.8% 8.3% 0.0% 5.0% 6.3% 8.8% 0.0% 21.1% 0.3% 10.0% 0.4% 0.0% 16.4% 2.1% 18.3% 2.7% Clean Validation Preprocessing Compilation (production) Compilation (test) Compilation (support) Testing (unit) Testing (integration) Testing (non functional) Testing (crosscutting) Packaging Static Analysis Dynamic Analysis Deployment (Local) Org OSS Ing Static Analysis tools: on CI server in OSS, remotely in ING.
38. Testing (non functional) Testing (crosscutting) Packaging Static Analysis Dynamic Analysis Deployment (Local) Deployment (Remote) Documentation Release Preparation Support External Tasks Dependencies % of build failures 0% 5% 10% 15% 20% 25% 7.1% 1.4% 0.9% 0.0% 0.9% 0.5% 0.0% 0.2% 4.2% 0.8% 8.3% 0.0% 5.0% 6.3% 8.8% 0.0% 21.1% 0.3% 10.0% 0.4% 0.0% 16.4% 2.1% 18.3% 2.7% Clean Validation Preprocessing Compilation (production) Compilation (test) Compilation (support) Testing (unit) Testing (integration) Testing (non functional) Testing (crosscutting) Packaging Static Analysis Dynamic Analysis Deployment (Local) Org OSS Ing Release preparation and deployment failures are very common in industry, less so in OSS
39. RQ2 How frequent are the different types of build failures in the observed OSS and ING Projects? OSS projects run every test on CI servers, ING mostly integration testing Early discovery of non-functional failures in ING Release preparation and deployment failures are very common in industry, less so in OSS Static Analysis tools: on CI server in OSS, remotely in ING. Compilation Errors are typically fixed in private builds
40. Source Build What’s next? How to support DevOps during Continuous Delivery? Deploy Testing
41. Industry VS Academia Contact me! Let’s fill the gap!
42. http://www.ifi.uzh.ch/en/seal/people/vassallo.html @ccvassallo vassallo@ifi.uzh.ch DevOps represents a change in IT culture, focusing on rapid IT service delivery through the adoption of agile, lean practices in the context of a system-oriented approach. DevOps emphasizes people (and culture), and seeks to improve collaboration between operations and development teams. DevOps implementations utilize technology — especially automation tools that can leverage an increasingly programmable and dynamic infrastructure from a life cycle perspective. — gartner.com Case Study at ING Nederland • We surveyed 152 ING NL DevOps to investigate their development practices. • In three years, from 2011 to 2013, ING NL has increased the number of delivered function points by 300% and reduced the cost of a single function point to one third. • Additionally, between 2012 and 2014, the release frequency has doubled, reaching one release every four days. Carmine Vassallo, Fiorella Zampetti, Daniele Romano, Moritz Beller, Annibale Panichella, Massimiliano Di Penta, Andy Zaidman, Continuous Delivery Practice in a Large Financial Organisation. International Conference on Software Maintenance and Evolution (ICSME), Raleigh (NC, USA), September, 2016 RQ1 What types of failures affect builds of OSS and ING Projects? RQ2 How frequent are the different types of build failures in the observed OSS and ING Projects? Carmine Vassallo, Gerald Schermann, Fiorella Zampetti, Daniele Romano, Philipp Leitner, Andy Zaidman, Massimiliano Di Penta, and Sebastiano Panichella, A Tale of CI Build Failures: an Open Source and a Financial Organisation Perspective. In Proceedings of the 33rd International Conference on Software Maintenance and Evolution (ICSME), Shanghai, China, 2017. Source Build What’s next? How to support DevOps during Continuous Delivery? Deploy Testing