Toward supporting DevOps during Continuous Software Development

Slide of my talk at the 10th DevOps Enthusiast Meetup in Zurich (Switzerland).

Abstract: Continuous Integration (CI) and Continuous Delivery (CD) are widespread in both industrial and open-source software (OSS) projects. We investigated the successful adoption of CD in a large financial organisation (ING Nederland) and the increasing number of OSS developers that rely on a CD pipeline to build their software applications. However, despite the claimed advantages of Continuous Delivery (e.g. reliable and fast release), its core part (i.e. CI) can be broken by trivial build failures. With the aim of understanding the types of build failure affecting both open and closed source software we categorised the build failures occurred in 349 OSS projects and 418 ING projects. On one hand the results revealed that OSS and ING projects exhibit substantially different distributions of build failure types. On the other hand our study pointed out that DevOps need a better support during CD breaks and measures for preventing CD degradation.

Transcript

1. Toward supporting
   devops during
   continuous software
   development
   Carmine Vassallo
   

   View Slide

2. DevOps
   represents a change in IT culture, focusing on rapid IT service
   delivery through the adoption of agile, lean practices in the context of
   a system-oriented approach. DevOps emphasizes people (and
   culture), and
   seeks to improve collaboration between operations and development
   teams. DevOps implementations utilize technology — especially
   automation
   tools that can leverage an increasingly programmable and dynamic
   infrastructure from a life cycle perspective. — gartner.com
   

   View Slide

3. DevOps
   represents a change in IT culture, focusing on rapid IT service
   delivery through the adoption of agile, lean practices in the context of
   a system-oriented approach. DevOps emphasizes people (and
   culture), and
   seeks to improve collaboration between operations and development
   teams. DevOps implementations utilize technology — especially
   automation
   tools that can leverage an increasingly programmable and dynamic
   infrastructure from a life cycle perspective. — gartner.com
   

   View Slide

4. DevOps
   represents a change in IT culture, focusing on rapid IT service
   delivery through the adoption of agile, lean practices in the context of
   a system-oriented approach. DevOps emphasizes people (and
   culture), and
   seeks to improve collaboration between operations and development
   teams. DevOps implementations utilize technology — especially
   automation
   tools that can leverage an increasingly programmable and dynamic
   infrastructure from a life cycle perspective. — gartner.com
   

   View Slide

5. DevOps
   represents a change in IT culture, focusing on rapid IT service
   delivery through the adoption of agile, lean practices in the context of
   a system-oriented approach. DevOps emphasizes people (and
   culture), and
   seeks to improve collaboration between operations and development
   teams. DevOps implementations utilize technology — especially
   automation
   tools that can leverage an increasingly programmable and dynamic
   infrastructure from a life cycle perspective. — gartner.com
   

   View Slide

6. DevOps
   represents a change in IT culture, focusing on rapid IT service
   delivery through the adoption of agile, lean practices in the context of
   a system-oriented approach. DevOps emphasizes people (and
   culture), and
   seeks to improve collaboration between operations and development
   teams. DevOps implementations utilize technology — especially
   automation
   tools that can leverage an increasingly programmable and dynamic
   infrastructure from a life cycle perspective. — gartner.com
   

   View Slide

7. View Slide

8. “Implementing automation is definitely a part of what it means to be practicing
   DevOps, but it’s maybe 5-10%.”
   process
   people
   (Automation is not DevOps, Chris Dodds)
   

   View Slide

9. Who am I?
   • My name is Carmine Vassallo
   • Research Intern in the Continuous Delivery team
   at ING Nederland (2015)
   • PhD Student in the Seal group at the
   University of Zurich (Since September 2016)
   • My current research focus is on supporting
   DevOps while dealing with CD breaks
   (e.g., build failures) and developing new
   approaches to prevent stop in the pipeline.
   http://www.ifi.uzh.ch/en/seal/people/vassallo.html
   @ccvassallo
   vassallo@ifi.uzh.ch
   

   View Slide

10. Continuous Delivery is a software
    development discipline where you
    build software in such a way that
    the software can be released to
    production at any time.
    (Martin Fowler)
    

    View Slide

11. Reliable Pipeline
    

    View Slide

12. Early Feedback
    

    View Slide

13. Fast Release
    

    View Slide

14. Case Study at ING Nederland
    • We surveyed 152 ING NL DevOps to
    investigate their development practices.
    • In three years, from 2011 to 2013, ING NL has
    increased the number of delivered function
    points by 300% and reduced the cost of a
    single function point to one third.
    • Additionally, between 2012 and 2014, the
    release frequency has doubled, reaching one
    release every four days.
    Carmine Vassallo, Fiorella Zampetti, Daniele Romano, Moritz Beller, Annibale Panichella, Massimiliano Di Penta, Andy Zaidman, Continuous Delivery Practice in
    a Large Financial Organisation. International Conference on Software Maintenance and Evolution (ICSME), Raleigh (NC, USA), September, 2016
    

    View Slide

15. Almost always
    Often
    Sometimes
    Seldom
    Almost never
    % of respondents
    0% 10% 20% 30% 40% 50%
    5%
    12%
    37%
    30%
    16%
    How frequently are refactoring tasks included in other tasks?
    ING DevOps tend to “self-admit” technical debt
    

    View Slide

16. Always
    When I have time
    Only for certain kinds of systems
    No
    % of respondents
    0% 10% 20% 30% 40% 50%
    22%
    12%
    33%
    34%
    Do you use TDD (Test Driven Development)?
    The majority of respondents declare to use TDD
    

    View Slide

17. Continuous Delivery is a software
    development discipline where you
    build software in such a way that
    the software can be released to
    production at any time.
    (Martin Fowler)
    

    View Slide

18. Continuous Delivery is a software
    development discipline where you
    build software in such a way that
    the software can be released to
    production at any time.
    (Martin Fowler)
    

    View Slide

19. Source Deploy Testing
    Build (CI)
    Continuous Delivery pipeline
    Continuous Integration
    

    View Slide

20. Build Failures
    

    View Slide

21. • On average, a build failure takes
    57 min to fix.
    • The overall cost of build failures
    ranging from 904.64 to 2034.92
    man-hours (over a period of 6
    months).
    • They monitored roughly 7200
    man-hours.
    . . Kerzazi, F. Khomh, and B. Adams, Why do automated builds break? an empirical study, in 30th IEEE International Conference on Software
    Maintenance and Evolution (ICSME), pp. 41–50, IEEE, 2014. 

    Relevance of Build Breakage
    

    View Slide

22. • When builds fail, due to compilation errors, it
    requires programmers to take extra time and
    brainpower to find and fix the problem,
    reducing their productivity.
    • A better understanding of the cause of
    frequent software build errors, then, could help
    lead to new or improved development tools
    that would reduce these errors and increase
    developer output.
    H. Seo, C. Sadowski, S. Elbaum, E. Aftandilian, and R. Bowdidge, Programmers’ build errors: A case study (at Google), in Proc. Int’l Conference on Software
    Engineering (ICSE), pp. 724–734, ACM, 2014.
    Relevance of Build Breakage
    

    View Slide

23. “Oops my build has failed…”
    How to support DevOps during
    build fixing?
    

    View Slide

24. Open
    Source
    (OSS)
    Closed
    Source
    (Industry)
    

    View Slide

25. Toward supporting
    devops during
    continuous software
    development
    

    View Slide

26. What types of
    build failures
    stop
    Continuous Delivery?
    

    View Slide

27. What types of
    build failures
    stop
    Continuous Delivery?
    A case study in OSS
    and ING projects
    

    View Slide

28. RQ1 What types of failures affect builds
    of OSS and ING Projects?
    RQ2 How frequent are the different
    types of build failures in the observed
    OSS and ING Projects?
    Carmine Vassallo, Gerald Schermann, Fiorella Zampetti, Daniele Romano, Philipp Leitner, Andy Zaidman, Massimiliano Di Penta, and Sebastiano Panichella,
    A Tale of CI Build Failures: an Open Source and a Financial Organisation Perspective.
    In Proceedings of the 33rd International Conference on Software Maintenance and Evolution (ICSME), Shanghai, China, 2017.
    

    View Slide

29. 1) Data Selection
    2) Data Extraction
    3) Build Failure Catalog
    Keyword
    identification
    Failed Goal
    Keyword grouping
    Failed Goals grouped into categories
    Build Log classification
    Each Build Failure Log belongs to
    at least one category
    Result Validation
    Results compared to the ones of
    the manual classification
    !
    "
    # $
    %
    349 Maven Projects
    (using Travis CI)
    418 Maven Projects
    (using Jenkins)
    34,182
    Build Failure Logs
    Study Procedure
    

    View Slide

30. RQ1 What type of failures affect builds of OSS and ING
    Projects?
    [ERROR] Failed to execute goal
    org.apache.maven.plugins:maven
    -surefire-plugin:2.12.4:test
    (default-test) on project
    openwayback-core: There are
    test failures.
    [ERROR] Failed to execute goal
    org.apache.maven.plugins:maven
    -compiler-plugin:2.3.2:compile
    (default-compile) on project
    version: Compilation failure
    [ERROR] Failed to execute goal on project barge-core:
    Could not resolve dependencies for project
    org.robotninjas.barge:barge-core:jar:0.1.0-SNAPSHOT:
    Could not find artifact org.robotninjas:netty-protobuf-
    rpc:jar:2.0.0-SNAPSHOT in sonatype-snapshots
    [ERROR] Failed to execute goal
    org.apache.maven.plugins:maven-release-plugin:
    2.5.1:prepare (default-cli)
    on project gTelephonyAPI: You don't have a
    SNAPSHOT project in the reactor projects list.
    Testing (Unit)
    Compilation (Production)
    Dependencies
    Release Preparation
    

    View Slide

31. CLEAN
    VALIDATION
    PRE-PROCESSING (RESOURCES)
    COMPILATION
    TESTING
    PACKAGING
    DOCUMENTATION
    SUPPORT
    EXTERNAL TASKS
    CODE ANALYSIS
    RELEASE PREPARATION
    DEPLOYMENT
    DEPENDENCIES
    PRODUCTION
    TEST
    SUPPORT
    UNIT TESTING
    INTEGRATION TESTING
    NON FUNCTIONAL TESTING
    CROSSCUTTING
    STATIC
    DYNAMIC
    LOCAL
    REMOTE
    RQ1 What type of failures affect builds of OSS and ING
    Projects?
    

    View Slide

32. RQ1 What type of failures affect builds of OSS and ING
    Projects?
    • Inter-rater agreement
    • ING: k=0.8 (strong agreement)
    • OSS: k=0.62 (strong agreement)
    

    View Slide

33. Clean
    Validation
    Preprocessing
    Compilation (production)
    Compilation (test)
    Compilation (support)
    Testing (unit)
    Testing (integration)
    Testing (non functional)
    Testing (crosscutting)
    Packaging
    Static Analysis
    Dynamic Analysis
    Deployment (Local)
    Deployment (Remote)
    Documentation
    Release Preparation
    Support
    External Tasks
    Dependencies
    % of build failures
    0% 5% 10% 15% 20% 25%
    7.1%
    1.4%
    0.9%
    0.0%
    0.9%
    0.5%
    0.0%
    0.2%
    4.2%
    0.8%
    8.3%
    0.0%
    5.0%
    28.0%
    0.2%
    1.8%
    7.1%
    1.3%
    0.5%
    0.0%
    6.3%
    8.8%
    0.0%
    21.1%
    0.3%
    10.0%
    0.4%
    0.0%
    16.4%
    2.1%
    18.3%
    2.7%
    13.3%
    5.2%
    0.0%
    2.3%
    4.2%
    0.0%
    0.0%
    0.0%
    Org OSS
    RQ2 How frequent are
    the different types of
    build failures in the
    observed OSS and ING
    Projects?
    Ing
    

    View Slide

34. Clean
    Validation
    Preprocessing
    Compilation (production)
    Compilation (test)
    Compilation (support)
    Testing (unit)
    Testing (integration)
    Testing (non functional)
    Testing (crosscutting)
    Packaging
    Static Analysis
    Dynamic Analysis 0.2%
    4.2%
    0.8%
    8.3%
    0.0%
    5.0%
    28.0%
    0.2%
    1.8%
    7.1%
    1.3%
    0.5%
    0.0%
    0.4%
    0.0%
    16.4%
    2.1%
    18.3%
    2.7%
    13.3%
    5.2%
    0.0%
    2.3%
    4.2%
    0.0%
    0.0%
    0.0%
    Org OSS
    Clean
    Validation
    Preprocessing
    Compilation (production)
    Compilation (test)
    Compilation (support)
    Testing (unit)
    Testing (integration)
    Testing (non functional)
    Testing (crosscutting)
    Packaging
    Static Analysis
    Dynamic Analysis
    Deployment (Local)
    Org OSS
    Ing
    Testing (non functional)
    Testing (crosscutting)
    Packaging
    Static Analysis
    Dynamic Analysis
    Deployment (Local)
    Deployment (Remote)
    Documentation
    Release Preparation
    Support
    External Tasks
    Dependencies
    % of build failures
    0% 5% 10% 15% 20% 25%
    Compilation Errors are typically fixed in
    private builds
    

    View Slide

35. Clean
    Validation
    Preprocessing
    Compilation (production)
    Compilation (test)
    Compilation (support)
    Testing (unit)
    Testing (integration)
    Testing (non functional)
    Testing (crosscutting)
    Packaging
    Static Analysis
    Dynamic Analysis 0.2%
    4.2%
    0.8%
    8.3%
    0.0%
    5.0%
    28.0%
    0.2%
    1.8%
    7.1%
    1.3%
    0.5%
    0.0%
    0.4%
    0.0%
    16.4%
    2.1%
    18.3%
    2.7%
    13.3%
    5.2%
    0.0%
    2.3%
    4.2%
    0.0%
    0.0%
    0.0%
    Org OSS
    OSS projects run every test on CI servers,
    ING mostly integration testing
    Clean
    Validation
    Preprocessing
    Compilation (production)
    Compilation (test)
    Compilation (support)
    Testing (unit)
    Testing (integration)
    Testing (non functional)
    Testing (crosscutting)
    Packaging
    Static Analysis
    Dynamic Analysis
    Deployment (Local)
    Org OSS
    Ing
    Testing (non functional)
    Testing (crosscutting)
    Packaging
    Static Analysis
    Dynamic Analysis
    Deployment (Local)
    Deployment (Remote)
    Documentation
    Release Preparation
    Support
    External Tasks
    Dependencies
    % of build failures
    0% 5% 10% 15% 20% 25%
    

    View Slide

36. Clean
    Validation
    Preprocessing
    Compilation (production)
    Compilation (test)
    Compilation (support)
    Testing (unit)
    Testing (integration)
    Testing (non functional)
    Testing (crosscutting)
    Packaging
    Static Analysis
    Dynamic Analysis 0.2%
    4.2%
    0.8%
    8.3%
    0.0%
    5.0%
    28.0%
    0.2%
    1.8%
    7.1%
    1.3%
    0.5%
    0.0%
    0.4%
    0.0%
    16.4%
    2.1%
    18.3%
    2.7%
    13.3%
    5.2%
    0.0%
    2.3%
    4.2%
    0.0%
    0.0%
    0.0%
    Org OSS
    Early discovery of non-functional failures in
    ING
    Clean
    Validation
    Preprocessing
    Compilation (production)
    Compilation (test)
    Compilation (support)
    Testing (unit)
    Testing (integration)
    Testing (non functional)
    Testing (crosscutting)
    Packaging
    Static Analysis
    Dynamic Analysis
    Deployment (Local)
    Org OSS
    Ing
    Testing (non functional)
    Testing (crosscutting)
    Packaging
    Static Analysis
    Dynamic Analysis
    Deployment (Local)
    Deployment (Remote)
    Documentation
    Release Preparation
    Support
    External Tasks
    Dependencies
    % of build failures
    0% 5% 10% 15% 20% 25%
    

    View Slide

37. Testing (non functional)
    Testing (crosscutting)
    Packaging
    Static Analysis
    Dynamic Analysis
    Deployment (Local)
    Deployment (Remote)
    Documentation
    Release Preparation
    Support
    External Tasks
    Dependencies
    % of build failures
    0% 5% 10% 15% 20% 25%
    7.1%
    1.4%
    0.9%
    0.0%
    0.9%
    0.5%
    0.0%
    0.2%
    4.2%
    0.8%
    8.3%
    0.0%
    5.0%
    6.3%
    8.8%
    0.0%
    21.1%
    0.3%
    10.0%
    0.4%
    0.0%
    16.4%
    2.1%
    18.3%
    2.7%
    Clean
    Validation
    Preprocessing
    Compilation (production)
    Compilation (test)
    Compilation (support)
    Testing (unit)
    Testing (integration)
    Testing (non functional)
    Testing (crosscutting)
    Packaging
    Static Analysis
    Dynamic Analysis
    Deployment (Local)
    Org OSS
    Ing
    Static Analysis tools: on CI server in OSS,
    remotely in ING.
    

    View Slide

38. Testing (non functional)
    Testing (crosscutting)
    Packaging
    Static Analysis
    Dynamic Analysis
    Deployment (Local)
    Deployment (Remote)
    Documentation
    Release Preparation
    Support
    External Tasks
    Dependencies
    % of build failures
    0% 5% 10% 15% 20% 25%
    7.1%
    1.4%
    0.9%
    0.0%
    0.9%
    0.5%
    0.0%
    0.2%
    4.2%
    0.8%
    8.3%
    0.0%
    5.0%
    6.3%
    8.8%
    0.0%
    21.1%
    0.3%
    10.0%
    0.4%
    0.0%
    16.4%
    2.1%
    18.3%
    2.7%
    Clean
    Validation
    Preprocessing
    Compilation (production)
    Compilation (test)
    Compilation (support)
    Testing (unit)
    Testing (integration)
    Testing (non functional)
    Testing (crosscutting)
    Packaging
    Static Analysis
    Dynamic Analysis
    Deployment (Local)
    Org OSS
    Ing
    Release preparation and deployment failures
    are very common in industry, less so in OSS
    

    View Slide

39. RQ2 How frequent are the different types of build
    failures in the observed OSS and ING Projects?
    OSS projects run every test on CI servers,
    ING mostly integration testing
    Early discovery of non-functional failures in
    ING
    Release preparation and deployment failures
    are very common in industry, less so in OSS
    Static Analysis tools: on CI server in OSS,
    remotely in ING.
    Compilation Errors are typically fixed in
    private builds
    

    View Slide

40. Source Build
    What’s next? How to support DevOps
    during Continuous Delivery?
    Deploy Testing
    

    View Slide

41. Industry VS Academia
    Contact me! Let’s fill the gap!
    

    View Slide

42. http://www.ifi.uzh.ch/en/seal/people/vassallo.html @ccvassallo
    vassallo@ifi.uzh.ch
    DevOps
    represents a change in IT culture, focusing on rapid IT service
    delivery through the adoption of agile, lean practices in the context of
    a system-oriented approach. DevOps emphasizes people (and
    culture), and
    seeks to improve collaboration between operations and development
    teams. DevOps implementations utilize technology — especially
    automation
    tools that can leverage an increasingly programmable and dynamic
    infrastructure from a life cycle perspective. — gartner.com
    Case Study at ING Nederland
    • We surveyed 152 ING NL DevOps to
    investigate their development practices.
    • In three years, from 2011 to 2013, ING NL has
    increased the number of delivered function
    points by 300% and reduced the cost of a
    single function point to one third.
    • Additionally, between 2012 and 2014, the
    release frequency has doubled, reaching one
    release every four days.
    Carmine Vassallo, Fiorella Zampetti, Daniele Romano, Moritz Beller, Annibale Panichella, Massimiliano Di Penta, Andy Zaidman, Continuous Delivery Practice in
    a Large Financial Organisation. International Conference on Software Maintenance and Evolution (ICSME), Raleigh (NC, USA), September, 2016
    RQ1 What types of failures affect builds
    of OSS and ING Projects?
    RQ2 How frequent are the different
    types of build failures in the observed
    OSS and ING Projects?
    Carmine Vassallo, Gerald Schermann, Fiorella Zampetti, Daniele Romano, Philipp Leitner, Andy Zaidman, Massimiliano Di Penta, and Sebastiano Panichella,
    A Tale of CI Build Failures: an Open Source and a Financial Organisation Perspective.
    In Proceedings of the 33rd International Conference on Software Maintenance and Evolution (ICSME), Shanghai, China, 2017.
    Source Build
    What’s next? How to support DevOps
    during Continuous Delivery?
    Deploy Testing
    

    View Slide