Lock in $30 Savings on PRO—Offer Ends Soon! ⏳

Toward supporting DevOps during Continuous Soft...

Toward supporting DevOps during Continuous Software Development

Slide of my talk at the 10th DevOps Enthusiast Meetup in Zurich (Switzerland).

Abstract: Continuous Integration (CI) and Continuous Delivery (CD) are widespread in both industrial and open-source software (OSS) projects. We investigated the successful adoption of CD in a large financial organisation (ING Nederland) and the increasing number of OSS developers that rely on a CD pipeline to build their software applications. However, despite the claimed advantages of Continuous Delivery (e.g. reliable and fast release), its core part (i.e. CI) can be broken by trivial build failures. With the aim of understanding the types of build failure affecting both open and closed source software we categorised the build failures occurred in 349 OSS projects and 418 ING projects. On one hand the results revealed that OSS and ING projects exhibit substantially different distributions of build failure types. On the other hand our study pointed out that DevOps need a better support during CD breaks and measures for preventing CD degradation.

Carmine Vassallo

July 12, 2017
Tweet

More Decks by Carmine Vassallo

Other Decks in Technology

Transcript

  1. DevOps represents a change in IT culture, focusing on rapid

    IT service delivery through the adoption of agile, lean practices in the context of a system-oriented approach. DevOps emphasizes people (and culture), and seeks to improve collaboration between operations and development teams. DevOps implementations utilize technology — especially automation tools that can leverage an increasingly programmable and dynamic infrastructure from a life cycle perspective. — gartner.com
  2. DevOps represents a change in IT culture, focusing on rapid

    IT service delivery through the adoption of agile, lean practices in the context of a system-oriented approach. DevOps emphasizes people (and culture), and seeks to improve collaboration between operations and development teams. DevOps implementations utilize technology — especially automation tools that can leverage an increasingly programmable and dynamic infrastructure from a life cycle perspective. — gartner.com
  3. DevOps represents a change in IT culture, focusing on rapid

    IT service delivery through the adoption of agile, lean practices in the context of a system-oriented approach. DevOps emphasizes people (and culture), and seeks to improve collaboration between operations and development teams. DevOps implementations utilize technology — especially automation tools that can leverage an increasingly programmable and dynamic infrastructure from a life cycle perspective. — gartner.com
  4. DevOps represents a change in IT culture, focusing on rapid

    IT service delivery through the adoption of agile, lean practices in the context of a system-oriented approach. DevOps emphasizes people (and culture), and seeks to improve collaboration between operations and development teams. DevOps implementations utilize technology — especially automation tools that can leverage an increasingly programmable and dynamic infrastructure from a life cycle perspective. — gartner.com
  5. DevOps represents a change in IT culture, focusing on rapid

    IT service delivery through the adoption of agile, lean practices in the context of a system-oriented approach. DevOps emphasizes people (and culture), and seeks to improve collaboration between operations and development teams. DevOps implementations utilize technology — especially automation tools that can leverage an increasingly programmable and dynamic infrastructure from a life cycle perspective. — gartner.com
  6. “Implementing automation is definitely a part of what it means

    to be practicing DevOps, but it’s maybe 5-10%.” process people (Automation is not DevOps, Chris Dodds)
  7. Who am I? • My name is Carmine Vassallo •

    Research Intern in the Continuous Delivery team at ING Nederland (2015) • PhD Student in the Seal group at the University of Zurich (Since September 2016) • My current research focus is on supporting DevOps while dealing with CD breaks (e.g., build failures) and developing new approaches to prevent stop in the pipeline. http://www.ifi.uzh.ch/en/seal/people/vassallo.html @ccvassallo [email protected]
  8. Continuous Delivery is a software development discipline where you build

    software in such a way that the software can be released to production at any time. (Martin Fowler)
  9. Case Study at ING Nederland • We surveyed 152 ING

    NL DevOps to investigate their development practices. • In three years, from 2011 to 2013, ING NL has increased the number of delivered function points by 300% and reduced the cost of a single function point to one third. • Additionally, between 2012 and 2014, the release frequency has doubled, reaching one release every four days. Carmine Vassallo, Fiorella Zampetti, Daniele Romano, Moritz Beller, Annibale Panichella, Massimiliano Di Penta, Andy Zaidman, Continuous Delivery Practice in a Large Financial Organisation. International Conference on Software Maintenance and Evolution (ICSME), Raleigh (NC, USA), September, 2016
  10. Almost always Often Sometimes Seldom Almost never % of respondents

    0% 10% 20% 30% 40% 50% 5% 12% 37% 30% 16% How frequently are refactoring tasks included in other tasks? ING DevOps tend to “self-admit” technical debt
  11. Always When I have time Only for certain kinds of

    systems No % of respondents 0% 10% 20% 30% 40% 50% 22% 12% 33% 34% Do you use TDD (Test Driven Development)? The majority of respondents declare to use TDD
  12. Continuous Delivery is a software development discipline where you build

    software in such a way that the software can be released to production at any time. (Martin Fowler)
  13. Continuous Delivery is a software development discipline where you build

    software in such a way that the software can be released to production at any time. (Martin Fowler)
  14. • On average, a build failure takes 57 min to

    fix. • The overall cost of build failures ranging from 904.64 to 2034.92 man-hours (over a period of 6 months). • They monitored roughly 7200 man-hours. . . Kerzazi, F. Khomh, and B. Adams, Why do automated builds break? an empirical study, in 30th IEEE International Conference on Software Maintenance and Evolution (ICSME), pp. 41–50, IEEE, 2014. 
 Relevance of Build Breakage
  15. • When builds fail, due to compilation errors, it requires

    programmers to take extra time and brainpower to find and fix the problem, reducing their productivity. • A better understanding of the cause of frequent software build errors, then, could help lead to new or improved development tools that would reduce these errors and increase developer output. H. Seo, C. Sadowski, S. Elbaum, E. Aftandilian, and R. Bowdidge, Programmers’ build errors: A case study (at Google), in Proc. Int’l Conference on Software Engineering (ICSE), pp. 724–734, ACM, 2014. Relevance of Build Breakage
  16. RQ1 What types of failures affect builds of OSS and

    ING Projects? RQ2 How frequent are the different types of build failures in the observed OSS and ING Projects? Carmine Vassallo, Gerald Schermann, Fiorella Zampetti, Daniele Romano, Philipp Leitner, Andy Zaidman, Massimiliano Di Penta, and Sebastiano Panichella, A Tale of CI Build Failures: an Open Source and a Financial Organisation Perspective. In Proceedings of the 33rd International Conference on Software Maintenance and Evolution (ICSME), Shanghai, China, 2017.
  17. 1) Data Selection 2) Data Extraction 3) Build Failure Catalog

    Keyword identification Failed Goal Keyword grouping Failed Goals grouped into categories Build Log classification Each Build Failure Log belongs to at least one category Result Validation Results compared to the ones of the manual classification ! " # $ % 349 Maven Projects (using Travis CI) 418 Maven Projects (using Jenkins) 34,182 Build Failure Logs Study Procedure
  18. RQ1 What type of failures affect builds of OSS and

    ING Projects? [ERROR] Failed to execute goal org.apache.maven.plugins:maven -surefire-plugin:2.12.4:test (default-test) on project openwayback-core: There are test failures. [ERROR] Failed to execute goal org.apache.maven.plugins:maven -compiler-plugin:2.3.2:compile (default-compile) on project version: Compilation failure [ERROR] Failed to execute goal on project barge-core: Could not resolve dependencies for project org.robotninjas.barge:barge-core:jar:0.1.0-SNAPSHOT: Could not find artifact org.robotninjas:netty-protobuf- rpc:jar:2.0.0-SNAPSHOT in sonatype-snapshots [ERROR] Failed to execute goal org.apache.maven.plugins:maven-release-plugin: 2.5.1:prepare (default-cli) on project gTelephonyAPI: You don't have a SNAPSHOT project in the reactor projects list. Testing (Unit) Compilation (Production) Dependencies Release Preparation
  19. CLEAN VALIDATION PRE-PROCESSING (RESOURCES) COMPILATION TESTING PACKAGING DOCUMENTATION SUPPORT EXTERNAL

    TASKS CODE ANALYSIS RELEASE PREPARATION DEPLOYMENT DEPENDENCIES PRODUCTION TEST SUPPORT UNIT TESTING INTEGRATION TESTING NON FUNCTIONAL TESTING CROSSCUTTING STATIC DYNAMIC LOCAL REMOTE RQ1 What type of failures affect builds of OSS and ING Projects?
  20. RQ1 What type of failures affect builds of OSS and

    ING Projects? • Inter-rater agreement • ING: k=0.8 (strong agreement) • OSS: k=0.62 (strong agreement)
  21. Clean Validation Preprocessing Compilation (production) Compilation (test) Compilation (support) Testing

    (unit) Testing (integration) Testing (non functional) Testing (crosscutting) Packaging Static Analysis Dynamic Analysis Deployment (Local) Deployment (Remote) Documentation Release Preparation Support External Tasks Dependencies % of build failures 0% 5% 10% 15% 20% 25% 7.1% 1.4% 0.9% 0.0% 0.9% 0.5% 0.0% 0.2% 4.2% 0.8% 8.3% 0.0% 5.0% 28.0% 0.2% 1.8% 7.1% 1.3% 0.5% 0.0% 6.3% 8.8% 0.0% 21.1% 0.3% 10.0% 0.4% 0.0% 16.4% 2.1% 18.3% 2.7% 13.3% 5.2% 0.0% 2.3% 4.2% 0.0% 0.0% 0.0% Org OSS RQ2 How frequent are the different types of build failures in the observed OSS and ING Projects? Ing
  22. Clean Validation Preprocessing Compilation (production) Compilation (test) Compilation (support) Testing

    (unit) Testing (integration) Testing (non functional) Testing (crosscutting) Packaging Static Analysis Dynamic Analysis 0.2% 4.2% 0.8% 8.3% 0.0% 5.0% 28.0% 0.2% 1.8% 7.1% 1.3% 0.5% 0.0% 0.4% 0.0% 16.4% 2.1% 18.3% 2.7% 13.3% 5.2% 0.0% 2.3% 4.2% 0.0% 0.0% 0.0% Org OSS Clean Validation Preprocessing Compilation (production) Compilation (test) Compilation (support) Testing (unit) Testing (integration) Testing (non functional) Testing (crosscutting) Packaging Static Analysis Dynamic Analysis Deployment (Local) Org OSS Ing Testing (non functional) Testing (crosscutting) Packaging Static Analysis Dynamic Analysis Deployment (Local) Deployment (Remote) Documentation Release Preparation Support External Tasks Dependencies % of build failures 0% 5% 10% 15% 20% 25% Compilation Errors are typically fixed in private builds
  23. Clean Validation Preprocessing Compilation (production) Compilation (test) Compilation (support) Testing

    (unit) Testing (integration) Testing (non functional) Testing (crosscutting) Packaging Static Analysis Dynamic Analysis 0.2% 4.2% 0.8% 8.3% 0.0% 5.0% 28.0% 0.2% 1.8% 7.1% 1.3% 0.5% 0.0% 0.4% 0.0% 16.4% 2.1% 18.3% 2.7% 13.3% 5.2% 0.0% 2.3% 4.2% 0.0% 0.0% 0.0% Org OSS OSS projects run every test on CI servers, ING mostly integration testing Clean Validation Preprocessing Compilation (production) Compilation (test) Compilation (support) Testing (unit) Testing (integration) Testing (non functional) Testing (crosscutting) Packaging Static Analysis Dynamic Analysis Deployment (Local) Org OSS Ing Testing (non functional) Testing (crosscutting) Packaging Static Analysis Dynamic Analysis Deployment (Local) Deployment (Remote) Documentation Release Preparation Support External Tasks Dependencies % of build failures 0% 5% 10% 15% 20% 25%
  24. Clean Validation Preprocessing Compilation (production) Compilation (test) Compilation (support) Testing

    (unit) Testing (integration) Testing (non functional) Testing (crosscutting) Packaging Static Analysis Dynamic Analysis 0.2% 4.2% 0.8% 8.3% 0.0% 5.0% 28.0% 0.2% 1.8% 7.1% 1.3% 0.5% 0.0% 0.4% 0.0% 16.4% 2.1% 18.3% 2.7% 13.3% 5.2% 0.0% 2.3% 4.2% 0.0% 0.0% 0.0% Org OSS Early discovery of non-functional failures in ING Clean Validation Preprocessing Compilation (production) Compilation (test) Compilation (support) Testing (unit) Testing (integration) Testing (non functional) Testing (crosscutting) Packaging Static Analysis Dynamic Analysis Deployment (Local) Org OSS Ing Testing (non functional) Testing (crosscutting) Packaging Static Analysis Dynamic Analysis Deployment (Local) Deployment (Remote) Documentation Release Preparation Support External Tasks Dependencies % of build failures 0% 5% 10% 15% 20% 25%
  25. Testing (non functional) Testing (crosscutting) Packaging Static Analysis Dynamic Analysis

    Deployment (Local) Deployment (Remote) Documentation Release Preparation Support External Tasks Dependencies % of build failures 0% 5% 10% 15% 20% 25% 7.1% 1.4% 0.9% 0.0% 0.9% 0.5% 0.0% 0.2% 4.2% 0.8% 8.3% 0.0% 5.0% 6.3% 8.8% 0.0% 21.1% 0.3% 10.0% 0.4% 0.0% 16.4% 2.1% 18.3% 2.7% Clean Validation Preprocessing Compilation (production) Compilation (test) Compilation (support) Testing (unit) Testing (integration) Testing (non functional) Testing (crosscutting) Packaging Static Analysis Dynamic Analysis Deployment (Local) Org OSS Ing Static Analysis tools: on CI server in OSS, remotely in ING.
  26. Testing (non functional) Testing (crosscutting) Packaging Static Analysis Dynamic Analysis

    Deployment (Local) Deployment (Remote) Documentation Release Preparation Support External Tasks Dependencies % of build failures 0% 5% 10% 15% 20% 25% 7.1% 1.4% 0.9% 0.0% 0.9% 0.5% 0.0% 0.2% 4.2% 0.8% 8.3% 0.0% 5.0% 6.3% 8.8% 0.0% 21.1% 0.3% 10.0% 0.4% 0.0% 16.4% 2.1% 18.3% 2.7% Clean Validation Preprocessing Compilation (production) Compilation (test) Compilation (support) Testing (unit) Testing (integration) Testing (non functional) Testing (crosscutting) Packaging Static Analysis Dynamic Analysis Deployment (Local) Org OSS Ing Release preparation and deployment failures are very common in industry, less so in OSS
  27. RQ2 How frequent are the different types of build failures

    in the observed OSS and ING Projects? OSS projects run every test on CI servers, ING mostly integration testing Early discovery of non-functional failures in ING Release preparation and deployment failures are very common in industry, less so in OSS Static Analysis tools: on CI server in OSS, remotely in ING. Compilation Errors are typically fixed in private builds
  28. http://www.ifi.uzh.ch/en/seal/people/vassallo.html @ccvassallo [email protected] DevOps represents a change in IT culture,

    focusing on rapid IT service delivery through the adoption of agile, lean practices in the context of a system-oriented approach. DevOps emphasizes people (and culture), and seeks to improve collaboration between operations and development teams. DevOps implementations utilize technology — especially automation tools that can leverage an increasingly programmable and dynamic infrastructure from a life cycle perspective. — gartner.com Case Study at ING Nederland • We surveyed 152 ING NL DevOps to investigate their development practices. • In three years, from 2011 to 2013, ING NL has increased the number of delivered function points by 300% and reduced the cost of a single function point to one third. • Additionally, between 2012 and 2014, the release frequency has doubled, reaching one release every four days. Carmine Vassallo, Fiorella Zampetti, Daniele Romano, Moritz Beller, Annibale Panichella, Massimiliano Di Penta, Andy Zaidman, Continuous Delivery Practice in a Large Financial Organisation. International Conference on Software Maintenance and Evolution (ICSME), Raleigh (NC, USA), September, 2016 RQ1 What types of failures affect builds of OSS and ING Projects? RQ2 How frequent are the different types of build failures in the observed OSS and ING Projects? Carmine Vassallo, Gerald Schermann, Fiorella Zampetti, Daniele Romano, Philipp Leitner, Andy Zaidman, Massimiliano Di Penta, and Sebastiano Panichella, A Tale of CI Build Failures: an Open Source and a Financial Organisation Perspective. In Proceedings of the 33rd International Conference on Software Maintenance and Evolution (ICSME), Shanghai, China, 2017. Source Build What’s next? How to support DevOps during Continuous Delivery? Deploy Testing