Upgrade to Pro — share decks privately, control downloads, hide ads and more …

A Tale of CI Build Failures: an Open Source and a Financial Organization Perspective

Carmine Vassallo
September 21, 2017

A Tale of CI Build Failures: an Open Source and a Financial Organization Perspective

Our study about CI build failures presented at the 33rd International Conference on Software Maintenance and Evolution (ICSME '17) in Shanghai, Cina. Preprint of the corresponding paper available at http://www.ifi.uzh.ch/seal/people/vassallo/vassalloICSME17.pdf

Carmine Vassallo

September 21, 2017
Tweet

More Decks by Carmine Vassallo

Other Decks in Research

Transcript

  1. A tale of ci build failures: an Open Source and a Financial Organization
    Perspective
    Carmine Vassallo, Gerald Schermann, Fiorella Zampetti, Daniele Romano, Philipp Leitner,
    Andy Zaidman, Massimiliano Di Penta, Sebastiano Panichella
    1
    @ccvassallo
    [email protected]

    View full-size slide

  2. Continuous Delivery is a
    software development
    discipline where you
    build software in such a
    way that the software
    can be released to
    production at any time.
    (Martin Fowler)
    2

    View full-size slide

  3. Continuous Delivery is a
    software development
    discipline where you
    build software in such a
    way that the software
    can be released to
    production at any time.
    (Martin Fowler)
    3

    View full-size slide

  4. Software building: Continuous Integration 4
    4
    Build(CI)
    Source

    View full-size slide

  5. Software building: Continuous Integration 5
    5
    Build(CI)
    Source

    View full-size slide

  6. • On average, a build failure takes
    57 min to fix.
    • The overall cost of build failures
    ranging from 904.64 to 2034.92
    man-hours (over a period of 6
    months).
    • They monitored roughly 7200
    man-hours.
    . . Kerzazi, F. Khomh, and B. Adams, Why do automated builds break? an empirical study, in 30th IEEE International Conference on Software
    Maintenance and Evolution (ICSME), pp. 41–50, IEEE, 2014. 

    Relevance of Build Breakage 6
    6

    View full-size slide

  7. Build breaks types 7
    Failing tests are the dominant reason for unsuccessful builds
    (Rausch et al., MSR 2017) (Beller et al., MSR 2017)
    Industry OSS
    40% of the failures occur during static analysis
    (Miller et al., AGILE 2008)
    Dependencies between components are the most relevant
    cause of compilation related failures
    (Seo et al., ICSE 2014)
    7

    View full-size slide

  8. 8
    What are differences and
    commonalities in the
    distribution of build failure
    types occurring in OSS and
    industry?
    8

    View full-size slide

  9. 10
    RQ1 What types of failures affect builds
    of OSS and industrial projects?
    RQ2 How frequent are the different
    types of build failures in the observed
    OSS and industrial projects?
    10

    View full-size slide

  10. Data Selection 11
    • 418 Maven (mostly Java)
    projects
    • 12,871 builds of which 3,390
    (≈ 26%) failed.
    • 349 Maven (Java) projects
    • 116,741 builds, of which 30,792
    (≈ 26%) failed
    11
    Industry OSS

    View full-size slide

  11. 12
    Data Selection
    • 418 Maven (mostly Java)
    projects
    • 12,871 builds of which 3,390
    (≈ 26%) failed.
    • 349 Maven (Java) projects
    • 116,741 builds, of which 30,792
    (≈ 26%) failed
    Industry OSS
    Build failure logs were the only resources we could
    access
    12

    View full-size slide

  12. 13
    Data Preprocessing
    13

    View full-size slide

  13. 14
    Data Preprocessing
    14
    [ERROR] Failed to execute goal org.apache.maven.plugins:maven-compiler-plugin:3.0:compile (default-compile) on project es-common:
    Compilation failure
    [ERROR] /home/travis/build/zhangkaitao/es/common/src/main/java/com/sishuok/es/common/utils/html/HTMLUtils.java:[14,8] class
    HtmlUtils is public, should be declared in a file named HtmlUtils.java
    [ERROR] -> [Help 1]
    [ERROR]
    [ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch.
    [ERROR] Re-run Maven using the -X switch to enable full debug logging.
    [ERROR]
    [ERROR] For more information about the errors and possible solutions, please read the following articles:
    [ERROR] [Help 1] http:/
    /cwiki.apache.org/confluence/display/MAVEN/MojoFailureException

    View full-size slide

  14. Keyword grouping
    Build Log classification
    Result Validation
    !
    " #
    $
    Build Failure Catalog 15
    15
    CODE ANALYSIS (STATIC)
    org.codehaus.mojo:sonar-maven-plugin:sonar
    org.codehaus.mojo:findbugs-maven-plugin:findbugs
    Inter-rater agreement
    • Industry: k=0.8 (strong agreement)
    • OSS: k=0.62 (strong agreement)

    View full-size slide

  15. 16
    CLEAN
    VALIDATION
    PRE-PROCESSING (RESOURCES)
    COMPILATION
    TESTING
    PACKAGING
    DOCUMENTATION
    SUPPORT
    EXTERNAL TASKS
    CODE ANALYSIS
    RELEASE PREPARATION
    DEPLOYMENT
    DEPENDENCIES
    PRODUCTION
    TEST
    SUPPORT
    UNIT TESTING
    INTEGRATION TESTING
    NON FUNCTIONAL TESTING
    CROSSCUTTING
    STATIC
    DYNAMIC
    LOCAL
    REMOTE
    RQ1 What type of failures affect builds of OSS and
    industrial projects?
    16

    View full-size slide

  16. 17
    Clean
    Validation
    Preprocessing
    Compilation (production)
    Compilation (test)
    Compilation (support)
    Testing (unit)
    Testing (integration)
    Testing (non functional)
    Testing (crosscutting)
    Packaging
    Static Analysis
    Dynamic Analysis
    Deployment (Local)
    Deployment (Remote)
    Documentation
    Release Preparation
    Support
    External Tasks
    Dependencies
    % of build failures
    0% 5% 10% 15% 20% 25%
    7.1%
    1.4%
    0.9%
    0.0%
    0.9%
    0.5%
    0.0%
    0.2%
    4.2%
    0.8%
    8.3%
    0.0%
    5.0%
    28.0%
    0.2%
    1.8%
    7.1%
    1.3%
    0.5%
    0.0%
    6.3%
    8.8%
    0.0%
    21.1%
    0.3%
    10.0%
    0.4%
    0.0%
    16.4%
    2.1%
    18.3%
    2.7%
    13.3%
    5.2%
    0.0%
    2.3%
    4.2%
    0.0%
    0.0%
    0.0%
    Org OSS
    RQ2 How frequent are
    the different types of
    build failures in the
    observed OSS and
    industrial projects?
    Industry
    17

    View full-size slide

  17. 18
    Compilation (production)
    Compilation (test)
    Compilation (support)
    % of build failures
    0% 5% 10% 15% 20% 25%
    0.2%
    1.8%
    7.1%
    0.0%
    2.3%
    4.2%
    ING OSS
    Compilation Errors are fairly limited
    Industry
    RQ2 How frequent are the different types of build
    failures in the observed OSS and industrial projects?
    18

    View full-size slide

  18. 19
    Testing (unit)
    Testing (integration)
    Testing (non functional)
    % of build failures
    0% 5% 10% 15% 20% 25%
    0.0%
    5.0%
    28.0%
    2.7%
    13.3%
    5.2%
    ING OSS
    OSS projects exhibit more unit than
    integration testing related failures.
    In industry it’s the opposite.
    Early discovery of non-functional failures in
    industry.
    RQ2 How frequent are the different types of build
    failures in the observed OSS and industrial projects?
    Industry
    19

    View full-size slide

  19. 20
    Static Analysis
    % of build failures
    0% 5% 10% 15% 20% 25%
    4.2%
    16.4%
    ING OSS
    Static Analysis tools: on CI server in OSS,
    remotely in industry.
    RQ2 How frequent are the different types of build
    failures in the observed OSS and industrial projects?
    Industry
    20

    View full-size slide

  20. 21
    Deployment (Local)
    Deployment (Remote)
    Release Preparation
    % of build failures
    0% 5% 10% 15% 20% 25%
    0.0%
    0.5%
    0.0%
    21.1%
    10.0%
    0.4%
    ING OSS
    Release preparation and deployment failures
    are very common in industry, less so in OSS
    RQ2 How frequent are the different types of build
    failures in the observed OSS and industrial projects?
    Industry
    21

    View full-size slide

  21. 22
    • Projects clustering (using K-means
    algorithm)
    • Optimal value of silhouette statistic: 6
    • Each cluster dominated either by
    industrial or oss projects
    • except Dependencies
    # projects
    0
    45
    90
    135
    180
    Cluster
    Code Analysis
    Release Preparation
    Dependencies
    Crosscutting Testing
    Unit Testing
    Compilation
    ING OSS
    25%
    75% 100% 59%
    41% 85%
    31%
    69%
    94%
    6%
    15%
    22
    RQ2 How frequent are the different types of build
    failures in the observed OSS and industrial project?
    Industry

    View full-size slide

  22. 23
    • Dependencies related failures occur with the same frequency.
    Key findings
    • In OSS projects a lot of failures are due to unit testing: try to catch those
    issues earlier!
    • Need for a better release/deployment strategy in OSS.
    • Static analysis on separate server: well collected data and less overloading of
    CI server.
    • Towards early discovery of non functional testing failures.
    23

    View full-size slide

  23. @ccvassallo
    [email protected]
    24
    Build breaks types X
    Failing tests are the dominant reason for unsuccessful builds
    (Rausch et al., MSR 2017) (Beller et al., MSR 2017)
    Industry OSS
    40% of the failures occur during static analysis
    (Miller et al., AGILE 2008)
    Dependencies between components are the most relevant
    cause of compilation related failures
    (Seo et al., ICSE 2014)
    X
    X
    RQ1 What types of failures affect builds
    of OSS and industrial projects?
    RQ2 How frequent are the different
    types of build failures in the observed
    OSS and industrial projects?
    X
    X
    CLEAN
    VALIDATION
    PRE-PROCESSING (RESOURCES)
    COMPILATION
    TESTING
    PACKAGING
    DOCUMENTATION
    SUPPORT
    EXTERNAL TASKS
    CODE ANALYSIS
    RELEASE PREPARATION
    DEPLOYMENT
    DEPENDENCIES
    PRODUCTION
    TEST
    SUPPORT
    UNIT TESTING
    INTEGRATION TESTING
    NON FUNCTIONAL TESTING
    CROSSCUTTING
    STATIC
    DYNAMIC
    LOCAL
    REMOTE
    RQ1 What type of failures affect builds of OSS and
    industrial projects?
    X
    X
    Clean
    Validation
    Preprocessing
    Compilation (production)
    Compilation (test)
    Compilation (support)
    Testing (unit)
    Testing (integration)
    Testing (non functional)
    Testing (crosscutting)
    Packaging
    Static Analysis
    Dynamic Analysis
    Deployment (Local)
    Deployment (Remote)
    Documentation
    Release Preparation
    Support
    External Tasks
    Dependencies
    % of build failures
    0% 5% 10% 15% 20% 25%
    7.1%
    1.4%
    0.9%
    0.0%
    0.9%
    0.5%
    0.0%
    0.2%
    4.2%
    0.8%
    8.3%
    0.0%
    5.0%
    28.0%
    0.2%
    1.8%
    7.1%
    1.3%
    0.5%
    0.0%
    6.3%
    8.8%
    0.0%
    21.1%
    0.3%
    10.0%
    0.4%
    0.0%
    16.4%
    2.1%
    18.3%
    2.7%
    13.3%
    5.2%
    0.0%
    2.3%
    4.2%
    0.0%
    0.0%
    0.0%
    Org OSS
    RQ2 How frequent are
    the different types of
    build failures in the
    observed OSS and
    industrial projects?
    Industry
    X

    View full-size slide