$30 off During Our Annual Pro Sale. View Details »

Toward supporting DevOps during Continuous Software Development

Toward supporting DevOps during Continuous Software Development

Slide of my talk at the 10th DevOps Enthusiast Meetup in Zurich (Switzerland).

Abstract: Continuous Integration (CI) and Continuous Delivery (CD) are widespread in both industrial and open-source software (OSS) projects. We investigated the successful adoption of CD in a large financial organisation (ING Nederland) and the increasing number of OSS developers that rely on a CD pipeline to build their software applications. However, despite the claimed advantages of Continuous Delivery (e.g. reliable and fast release), its core part (i.e. CI) can be broken by trivial build failures. With the aim of understanding the types of build failure affecting both open and closed source software we categorised the build failures occurred in 349 OSS projects and 418 ING projects. On one hand the results revealed that OSS and ING projects exhibit substantially different distributions of build failure types. On the other hand our study pointed out that DevOps need a better support during CD breaks and measures for preventing CD degradation.

Carmine Vassallo

July 12, 2017
Tweet

More Decks by Carmine Vassallo

Other Decks in Technology

Transcript

  1. Toward supporting
    devops during
    continuous software
    development
    Carmine Vassallo

    View Slide

  2. DevOps
    represents a change in IT culture, focusing on rapid IT service
    delivery through the adoption of agile, lean practices in the context of
    a system-oriented approach. DevOps emphasizes people (and
    culture), and
    seeks to improve collaboration between operations and development
    teams. DevOps implementations utilize technology — especially
    automation
    tools that can leverage an increasingly programmable and dynamic
    infrastructure from a life cycle perspective. — gartner.com

    View Slide

  3. DevOps
    represents a change in IT culture, focusing on rapid IT service
    delivery through the adoption of agile, lean practices in the context of
    a system-oriented approach. DevOps emphasizes people (and
    culture), and
    seeks to improve collaboration between operations and development
    teams. DevOps implementations utilize technology — especially
    automation
    tools that can leverage an increasingly programmable and dynamic
    infrastructure from a life cycle perspective. — gartner.com

    View Slide

  4. DevOps
    represents a change in IT culture, focusing on rapid IT service
    delivery through the adoption of agile, lean practices in the context of
    a system-oriented approach. DevOps emphasizes people (and
    culture), and
    seeks to improve collaboration between operations and development
    teams. DevOps implementations utilize technology — especially
    automation
    tools that can leverage an increasingly programmable and dynamic
    infrastructure from a life cycle perspective. — gartner.com

    View Slide

  5. DevOps
    represents a change in IT culture, focusing on rapid IT service
    delivery through the adoption of agile, lean practices in the context of
    a system-oriented approach. DevOps emphasizes people (and
    culture), and
    seeks to improve collaboration between operations and development
    teams. DevOps implementations utilize technology — especially
    automation
    tools that can leverage an increasingly programmable and dynamic
    infrastructure from a life cycle perspective. — gartner.com

    View Slide

  6. DevOps
    represents a change in IT culture, focusing on rapid IT service
    delivery through the adoption of agile, lean practices in the context of
    a system-oriented approach. DevOps emphasizes people (and
    culture), and
    seeks to improve collaboration between operations and development
    teams. DevOps implementations utilize technology — especially
    automation
    tools that can leverage an increasingly programmable and dynamic
    infrastructure from a life cycle perspective. — gartner.com

    View Slide

  7. View Slide

  8. “Implementing automation is definitely a part of what it means to be practicing
    DevOps, but it’s maybe 5-10%.”
    process
    people
    (Automation is not DevOps, Chris Dodds)

    View Slide

  9. Who am I?
    • My name is Carmine Vassallo
    • Research Intern in the Continuous Delivery team
    at ING Nederland (2015)
    • PhD Student in the Seal group at the
    University of Zurich (Since September 2016)
    • My current research focus is on supporting
    DevOps while dealing with CD breaks
    (e.g., build failures) and developing new
    approaches to prevent stop in the pipeline.
    http://www.ifi.uzh.ch/en/seal/people/vassallo.html
    @ccvassallo
    [email protected]

    View Slide

  10. Continuous Delivery is a software
    development discipline where you
    build software in such a way that
    the software can be released to
    production at any time.
    (Martin Fowler)

    View Slide

  11. Reliable Pipeline

    View Slide

  12. Early Feedback

    View Slide

  13. Fast Release

    View Slide

  14. Case Study at ING Nederland
    • We surveyed 152 ING NL DevOps to
    investigate their development practices.
    • In three years, from 2011 to 2013, ING NL has
    increased the number of delivered function
    points by 300% and reduced the cost of a
    single function point to one third.
    • Additionally, between 2012 and 2014, the
    release frequency has doubled, reaching one
    release every four days.
    Carmine Vassallo, Fiorella Zampetti, Daniele Romano, Moritz Beller, Annibale Panichella, Massimiliano Di Penta, Andy Zaidman, Continuous Delivery Practice in
    a Large Financial Organisation. International Conference on Software Maintenance and Evolution (ICSME), Raleigh (NC, USA), September, 2016

    View Slide

  15. Almost always
    Often
    Sometimes
    Seldom
    Almost never
    % of respondents
    0% 10% 20% 30% 40% 50%
    5%
    12%
    37%
    30%
    16%
    How frequently are refactoring tasks included in other tasks?
    ING DevOps tend to “self-admit” technical debt

    View Slide

  16. Always
    When I have time
    Only for certain kinds of systems
    No
    % of respondents
    0% 10% 20% 30% 40% 50%
    22%
    12%
    33%
    34%
    Do you use TDD (Test Driven Development)?
    The majority of respondents declare to use TDD

    View Slide

  17. Continuous Delivery is a software
    development discipline where you
    build software in such a way that
    the software can be released to
    production at any time.
    (Martin Fowler)

    View Slide

  18. Continuous Delivery is a software
    development discipline where you
    build software in such a way that
    the software can be released to
    production at any time.
    (Martin Fowler)

    View Slide

  19. Source Deploy Testing
    Build (CI)
    Continuous Delivery pipeline
    Continuous Integration

    View Slide

  20. Build Failures

    View Slide

  21. • On average, a build failure takes
    57 min to fix.
    • The overall cost of build failures
    ranging from 904.64 to 2034.92
    man-hours (over a period of 6
    months).
    • They monitored roughly 7200
    man-hours.
    . . Kerzazi, F. Khomh, and B. Adams, Why do automated builds break? an empirical study, in 30th IEEE International Conference on Software
    Maintenance and Evolution (ICSME), pp. 41–50, IEEE, 2014. 

    Relevance of Build Breakage

    View Slide

  22. • When builds fail, due to compilation errors, it
    requires programmers to take extra time and
    brainpower to find and fix the problem,
    reducing their productivity.
    • A better understanding of the cause of
    frequent software build errors, then, could help
    lead to new or improved development tools
    that would reduce these errors and increase
    developer output.
    H. Seo, C. Sadowski, S. Elbaum, E. Aftandilian, and R. Bowdidge, Programmers’ build errors: A case study (at Google), in Proc. Int’l Conference on Software
    Engineering (ICSE), pp. 724–734, ACM, 2014.
    Relevance of Build Breakage

    View Slide

  23. “Oops my build has failed…”
    How to support DevOps during
    build fixing?

    View Slide

  24. Open
    Source
    (OSS)
    Closed
    Source
    (Industry)

    View Slide

  25. Toward supporting
    devops during
    continuous software
    development

    View Slide

  26. What types of
    build failures
    stop
    Continuous Delivery?

    View Slide

  27. What types of
    build failures
    stop
    Continuous Delivery?
    A case study in OSS
    and ING projects

    View Slide

  28. RQ1 What types of failures affect builds
    of OSS and ING Projects?
    RQ2 How frequent are the different
    types of build failures in the observed
    OSS and ING Projects?
    Carmine Vassallo, Gerald Schermann, Fiorella Zampetti, Daniele Romano, Philipp Leitner, Andy Zaidman, Massimiliano Di Penta, and Sebastiano Panichella,
    A Tale of CI Build Failures: an Open Source and a Financial Organisation Perspective.
    In Proceedings of the 33rd International Conference on Software Maintenance and Evolution (ICSME), Shanghai, China, 2017.

    View Slide

  29. 1) Data Selection
    2) Data Extraction
    3) Build Failure Catalog
    Keyword
    identification
    Failed Goal
    Keyword grouping
    Failed Goals grouped into categories
    Build Log classification
    Each Build Failure Log belongs to
    at least one category
    Result Validation
    Results compared to the ones of
    the manual classification
    !
    "
    # $
    %
    349 Maven Projects
    (using Travis CI)
    418 Maven Projects
    (using Jenkins)
    34,182
    Build Failure Logs
    Study Procedure

    View Slide

  30. RQ1 What type of failures affect builds of OSS and ING
    Projects?
    [ERROR] Failed to execute goal
    org.apache.maven.plugins:maven
    -surefire-plugin:2.12.4:test
    (default-test) on project
    openwayback-core: There are
    test failures.
    [ERROR] Failed to execute goal
    org.apache.maven.plugins:maven
    -compiler-plugin:2.3.2:compile
    (default-compile) on project
    version: Compilation failure
    [ERROR] Failed to execute goal on project barge-core:
    Could not resolve dependencies for project
    org.robotninjas.barge:barge-core:jar:0.1.0-SNAPSHOT:
    Could not find artifact org.robotninjas:netty-protobuf-
    rpc:jar:2.0.0-SNAPSHOT in sonatype-snapshots
    [ERROR] Failed to execute goal
    org.apache.maven.plugins:maven-release-plugin:
    2.5.1:prepare (default-cli)
    on project gTelephonyAPI: You don't have a
    SNAPSHOT project in the reactor projects list.
    Testing (Unit)
    Compilation (Production)
    Dependencies
    Release Preparation

    View Slide

  31. CLEAN
    VALIDATION
    PRE-PROCESSING (RESOURCES)
    COMPILATION
    TESTING
    PACKAGING
    DOCUMENTATION
    SUPPORT
    EXTERNAL TASKS
    CODE ANALYSIS
    RELEASE PREPARATION
    DEPLOYMENT
    DEPENDENCIES
    PRODUCTION
    TEST
    SUPPORT
    UNIT TESTING
    INTEGRATION TESTING
    NON FUNCTIONAL TESTING
    CROSSCUTTING
    STATIC
    DYNAMIC
    LOCAL
    REMOTE
    RQ1 What type of failures affect builds of OSS and ING
    Projects?

    View Slide

  32. RQ1 What type of failures affect builds of OSS and ING
    Projects?
    • Inter-rater agreement
    • ING: k=0.8 (strong agreement)
    • OSS: k=0.62 (strong agreement)

    View Slide

  33. Clean
    Validation
    Preprocessing
    Compilation (production)
    Compilation (test)
    Compilation (support)
    Testing (unit)
    Testing (integration)
    Testing (non functional)
    Testing (crosscutting)
    Packaging
    Static Analysis
    Dynamic Analysis
    Deployment (Local)
    Deployment (Remote)
    Documentation
    Release Preparation
    Support
    External Tasks
    Dependencies
    % of build failures
    0% 5% 10% 15% 20% 25%
    7.1%
    1.4%
    0.9%
    0.0%
    0.9%
    0.5%
    0.0%
    0.2%
    4.2%
    0.8%
    8.3%
    0.0%
    5.0%
    28.0%
    0.2%
    1.8%
    7.1%
    1.3%
    0.5%
    0.0%
    6.3%
    8.8%
    0.0%
    21.1%
    0.3%
    10.0%
    0.4%
    0.0%
    16.4%
    2.1%
    18.3%
    2.7%
    13.3%
    5.2%
    0.0%
    2.3%
    4.2%
    0.0%
    0.0%
    0.0%
    Org OSS
    RQ2 How frequent are
    the different types of
    build failures in the
    observed OSS and ING
    Projects?
    Ing

    View Slide

  34. Clean
    Validation
    Preprocessing
    Compilation (production)
    Compilation (test)
    Compilation (support)
    Testing (unit)
    Testing (integration)
    Testing (non functional)
    Testing (crosscutting)
    Packaging
    Static Analysis
    Dynamic Analysis 0.2%
    4.2%
    0.8%
    8.3%
    0.0%
    5.0%
    28.0%
    0.2%
    1.8%
    7.1%
    1.3%
    0.5%
    0.0%
    0.4%
    0.0%
    16.4%
    2.1%
    18.3%
    2.7%
    13.3%
    5.2%
    0.0%
    2.3%
    4.2%
    0.0%
    0.0%
    0.0%
    Org OSS
    Clean
    Validation
    Preprocessing
    Compilation (production)
    Compilation (test)
    Compilation (support)
    Testing (unit)
    Testing (integration)
    Testing (non functional)
    Testing (crosscutting)
    Packaging
    Static Analysis
    Dynamic Analysis
    Deployment (Local)
    Org OSS
    Ing
    Testing (non functional)
    Testing (crosscutting)
    Packaging
    Static Analysis
    Dynamic Analysis
    Deployment (Local)
    Deployment (Remote)
    Documentation
    Release Preparation
    Support
    External Tasks
    Dependencies
    % of build failures
    0% 5% 10% 15% 20% 25%
    Compilation Errors are typically fixed in
    private builds

    View Slide

  35. Clean
    Validation
    Preprocessing
    Compilation (production)
    Compilation (test)
    Compilation (support)
    Testing (unit)
    Testing (integration)
    Testing (non functional)
    Testing (crosscutting)
    Packaging
    Static Analysis
    Dynamic Analysis 0.2%
    4.2%
    0.8%
    8.3%
    0.0%
    5.0%
    28.0%
    0.2%
    1.8%
    7.1%
    1.3%
    0.5%
    0.0%
    0.4%
    0.0%
    16.4%
    2.1%
    18.3%
    2.7%
    13.3%
    5.2%
    0.0%
    2.3%
    4.2%
    0.0%
    0.0%
    0.0%
    Org OSS
    OSS projects run every test on CI servers,
    ING mostly integration testing
    Clean
    Validation
    Preprocessing
    Compilation (production)
    Compilation (test)
    Compilation (support)
    Testing (unit)
    Testing (integration)
    Testing (non functional)
    Testing (crosscutting)
    Packaging
    Static Analysis
    Dynamic Analysis
    Deployment (Local)
    Org OSS
    Ing
    Testing (non functional)
    Testing (crosscutting)
    Packaging
    Static Analysis
    Dynamic Analysis
    Deployment (Local)
    Deployment (Remote)
    Documentation
    Release Preparation
    Support
    External Tasks
    Dependencies
    % of build failures
    0% 5% 10% 15% 20% 25%

    View Slide

  36. Clean
    Validation
    Preprocessing
    Compilation (production)
    Compilation (test)
    Compilation (support)
    Testing (unit)
    Testing (integration)
    Testing (non functional)
    Testing (crosscutting)
    Packaging
    Static Analysis
    Dynamic Analysis 0.2%
    4.2%
    0.8%
    8.3%
    0.0%
    5.0%
    28.0%
    0.2%
    1.8%
    7.1%
    1.3%
    0.5%
    0.0%
    0.4%
    0.0%
    16.4%
    2.1%
    18.3%
    2.7%
    13.3%
    5.2%
    0.0%
    2.3%
    4.2%
    0.0%
    0.0%
    0.0%
    Org OSS
    Early discovery of non-functional failures in
    ING
    Clean
    Validation
    Preprocessing
    Compilation (production)
    Compilation (test)
    Compilation (support)
    Testing (unit)
    Testing (integration)
    Testing (non functional)
    Testing (crosscutting)
    Packaging
    Static Analysis
    Dynamic Analysis
    Deployment (Local)
    Org OSS
    Ing
    Testing (non functional)
    Testing (crosscutting)
    Packaging
    Static Analysis
    Dynamic Analysis
    Deployment (Local)
    Deployment (Remote)
    Documentation
    Release Preparation
    Support
    External Tasks
    Dependencies
    % of build failures
    0% 5% 10% 15% 20% 25%

    View Slide

  37. Testing (non functional)
    Testing (crosscutting)
    Packaging
    Static Analysis
    Dynamic Analysis
    Deployment (Local)
    Deployment (Remote)
    Documentation
    Release Preparation
    Support
    External Tasks
    Dependencies
    % of build failures
    0% 5% 10% 15% 20% 25%
    7.1%
    1.4%
    0.9%
    0.0%
    0.9%
    0.5%
    0.0%
    0.2%
    4.2%
    0.8%
    8.3%
    0.0%
    5.0%
    6.3%
    8.8%
    0.0%
    21.1%
    0.3%
    10.0%
    0.4%
    0.0%
    16.4%
    2.1%
    18.3%
    2.7%
    Clean
    Validation
    Preprocessing
    Compilation (production)
    Compilation (test)
    Compilation (support)
    Testing (unit)
    Testing (integration)
    Testing (non functional)
    Testing (crosscutting)
    Packaging
    Static Analysis
    Dynamic Analysis
    Deployment (Local)
    Org OSS
    Ing
    Static Analysis tools: on CI server in OSS,
    remotely in ING.

    View Slide

  38. Testing (non functional)
    Testing (crosscutting)
    Packaging
    Static Analysis
    Dynamic Analysis
    Deployment (Local)
    Deployment (Remote)
    Documentation
    Release Preparation
    Support
    External Tasks
    Dependencies
    % of build failures
    0% 5% 10% 15% 20% 25%
    7.1%
    1.4%
    0.9%
    0.0%
    0.9%
    0.5%
    0.0%
    0.2%
    4.2%
    0.8%
    8.3%
    0.0%
    5.0%
    6.3%
    8.8%
    0.0%
    21.1%
    0.3%
    10.0%
    0.4%
    0.0%
    16.4%
    2.1%
    18.3%
    2.7%
    Clean
    Validation
    Preprocessing
    Compilation (production)
    Compilation (test)
    Compilation (support)
    Testing (unit)
    Testing (integration)
    Testing (non functional)
    Testing (crosscutting)
    Packaging
    Static Analysis
    Dynamic Analysis
    Deployment (Local)
    Org OSS
    Ing
    Release preparation and deployment failures
    are very common in industry, less so in OSS

    View Slide

  39. RQ2 How frequent are the different types of build
    failures in the observed OSS and ING Projects?
    OSS projects run every test on CI servers,
    ING mostly integration testing
    Early discovery of non-functional failures in
    ING
    Release preparation and deployment failures
    are very common in industry, less so in OSS
    Static Analysis tools: on CI server in OSS,
    remotely in ING.
    Compilation Errors are typically fixed in
    private builds

    View Slide

  40. Source Build
    What’s next? How to support DevOps
    during Continuous Delivery?
    Deploy Testing

    View Slide

  41. Industry VS Academia
    Contact me! Let’s fill the gap!

    View Slide

  42. http://www.ifi.uzh.ch/en/seal/people/vassallo.html @ccvassallo
    [email protected]
    DevOps
    represents a change in IT culture, focusing on rapid IT service
    delivery through the adoption of agile, lean practices in the context of
    a system-oriented approach. DevOps emphasizes people (and
    culture), and
    seeks to improve collaboration between operations and development
    teams. DevOps implementations utilize technology — especially
    automation
    tools that can leverage an increasingly programmable and dynamic
    infrastructure from a life cycle perspective. — gartner.com
    Case Study at ING Nederland
    • We surveyed 152 ING NL DevOps to
    investigate their development practices.
    • In three years, from 2011 to 2013, ING NL has
    increased the number of delivered function
    points by 300% and reduced the cost of a
    single function point to one third.
    • Additionally, between 2012 and 2014, the
    release frequency has doubled, reaching one
    release every four days.
    Carmine Vassallo, Fiorella Zampetti, Daniele Romano, Moritz Beller, Annibale Panichella, Massimiliano Di Penta, Andy Zaidman, Continuous Delivery Practice in
    a Large Financial Organisation. International Conference on Software Maintenance and Evolution (ICSME), Raleigh (NC, USA), September, 2016
    RQ1 What types of failures affect builds
    of OSS and ING Projects?
    RQ2 How frequent are the different
    types of build failures in the observed
    OSS and ING Projects?
    Carmine Vassallo, Gerald Schermann, Fiorella Zampetti, Daniele Romano, Philipp Leitner, Andy Zaidman, Massimiliano Di Penta, and Sebastiano Panichella,
    A Tale of CI Build Failures: an Open Source and a Financial Organisation Perspective.
    In Proceedings of the 33rd International Conference on Software Maintenance and Evolution (ICSME), Shanghai, China, 2017.
    Source Build
    What’s next? How to support DevOps
    during Continuous Delivery?
    Deploy Testing

    View Slide