$30 off During Our Annual Pro Sale. View Details »

Un-Break My Build: Assisting Developers with Build Repair Hints.

Un-Break My Build: Assisting Developers with Build Repair Hints.

Our study on developer-oriented assistance for build failures presented at the IEEE/ACM International Conference on Program Comprehension (Technical Track, ICPC '18) in Gothenburg, Sweden. Preprint of the corresponding paper available at http://www.ifi.uzh.ch/seal/people/vassallo/VassalloICPC18.pdf

Carmine Vassallo

May 27, 2018
Tweet

More Decks by Carmine Vassallo

Other Decks in Research

Transcript

  1. Un-Break My Build:
    Assisting Developers with Build Repair Hints.
    Carmine Vassallo, Sebastian Proksch, Timothy Zemp, and Harald C. Gall.
    @ccvassallo
    IEEE/ACM International Conference on Program Comprehension (Technical Track), Gothenburg, Sweden 2018

    View Slide

  2. Continuous Integration (CI)
    2
    Developer
    Commit
    Developer Repository Build Server
    Commit Poll

    View Slide

  3. 3
    Developer
    Commit
    Build Pass?
    Developer Repository Build Server
    Yes
    No
    Commit
    Build Success
    Build Failure
    Poll
    Continuous Integration (CI)

    View Slide

  4. Early Error Detection in CI: Build Failures
    4
    Developer
    Commit
    Build Pass?
    Developer Repository Build Server
    Yes
    No
    Commit
    Build Success
    Build Failure
    Read
    Poll
    [INFO] Scanning for projects...
    [INFO] Inspecting build with
    total of 1 modules...
    [ERROR] To see the full stack
    trace of the errors, re-run
    Maven with the -e switch.
    Build Log

    View Slide

  5. Why Do We Need to Study
    Build Failures?
    5

    View Slide

  6. 6
    Google Guava
    Build Server: TravisCI
    Build Tool: Maven
    40431 lines (Job #3.1)
    https://travis-ci.org/google/guava/jobs/40227447

    View Slide

  7. 7
    ‑1572 lines

    View Slide

  8. 8
    Modules being built

    View Slide

  9. 9
    ‑38828 lines

    View Slide

  10. 10
    Build Outcome
    Failed Build Step

    View Slide

  11. 11
    ‐4219 lines

    View Slide

  12. 12
    Test Execution

    View Slide

  13. 13
    Barriers when Solving Build Failures
    Hilton et al.,
    “Trade-offs in continuous integration:
    assurance, security, and flexibility”
    FSE 2017
    Build Logs are quite large in size
    (100k lines)
    Build logs lack of important
    informa;on (e.g., full test report)

    View Slide

  14. 14
    Build Un-Breaking takes long time
    Kerzazi et al.,
    “Why do Automated Builds Break?
    An Empirical Study”
    ICSME 2014
    On average, a build break takes
    1 hour to be fixed
    Significant amount of working
    hours spent on build fixing

    View Slide

  15. How Can We Mitigate the
    Build-Failure Overhead?
    15

    View Slide

  16. 16
    From BIG To small

    View Slide

  17. 17
    Developer-Oriented Assistance for Build Failure

    View Slide

  18. 18

    View Slide

  19. 19
    Automated Build Fix
    (Macho et al., SANER 2018)

    View Slide

  20. 20
    Automated Build Fix
    (Macho et al., SANER 2018)
    Developer-Oriented Assistance
    (BART)

    View Slide

  21. 21
    Developer
    Commit
    Build Pass?
    Developer Repository Build Server
    Yes
    No
    Commit
    [INFO] Scanning for projects...
    [INFO] Inspecting build with
    total of 1 modules...
    [ERROR] To see the full stack
    trace of the errors, re-run
    Maven with the -e switch.
    Build Success
    Build Failure
    Build log
    Poll
    Read
    Data Extraction
    Extract Generate
    External Resources
    Hint
    Build Summary
    BART (Build Abstraction and Recovery Tool)
    Meta-Model
    Data Enrichment

    View Slide

  22. 22
    From BIG To small

    View Slide

  23. 23
    From BIG To small
    Build Summary
    Build Status Failed
    Failed Goal org.apache.maven.plugins:maven-compiler-plugin:2.1:compile
    Error Cause Compilation Failure
    Reactor
    ActiveJpa SUCCESS
    test SUCCESS
    core FAILURE
    utils SKIPPED
    Reason for Build Failure: Compilation Failure
    Hint: Compilation
    Your build contains a compilation error. Please check the following
    file:
    File: Model.java
    Line: 224
    Reason: Model is abstract; cannot be instantiated
    Done
    Build Server
    Build
    Passed?
    Repository
    Build
    Log
    Notification
    Hint Generators
    1 2
    3
    4
    [INFO] Scanning for projects...
    [INFO] Inspecting build with
    total of 1 modules...
    [ERROR] To see the full stack
    trace of the errors, re-run
    Maven with the -e switch.
    5
    7
    Developer
    Compilation
    Testing
    Code Analysis
    Dependencies
    6
    BART
    8
    StackOverflow Discussions
    9
    14 lines
    ≈ 40k lines

    View Slide

  24. 24
    From BIG To small
    Build Summary
    Build Status Failed
    Failed Goal org.apache.maven.plugins:maven-compiler-plugin:2.1:compile
    Error Cause Compilation Failure
    Reactor
    ActiveJpa SUCCESS
    test SUCCESS
    core FAILURE
    utils SKIPPED
    Reason for Build Failure: Compilation Failure
    Hint: Compilation
    Your build contains a compilation error. Please check the following
    file:
    File: Model.java
    Line: 224
    Reason: Model is abstract; cannot be instantiated
    Done
    Build Server
    Build
    Passed?
    Repository
    Build
    Log
    Notification
    Hint Generators
    1 2
    3
    4
    [INFO] Scanning for projects...
    [INFO] Inspecting build with
    total of 1 modules...
    [ERROR] To see the full stack
    trace of the errors, re-run
    Maven with the -e switch.
    5
    7
    Developer
    Compilation
    Testing
    Code Analysis
    Dependencies
    6
    BART
    8
    StackOverflow Discussions
    9
    14 lines
    ≈ 40k lines

    View Slide

  25. Hint Generators
    Hints for the most 4 recurrent build-
    failure types (Vassallo et al., ICSME 2017)
    • Compilation
    • Dependencies
    • Code Analysis
    • Testing
    External Resource: StackOverflow
    25
    Build Summary
    Build Status Failed
    Failed Goal org.apache.maven.plugins:maven-compiler-plugin:2.1:compile
    Error Cause Compilation Failure
    Reactor
    ActiveJpa SUCCESS
    test SUCCESS
    core FAILURE
    utils SKIPPED
    Reason for Build Failure: Compilation Failure
    Hint: Compilation
    Your build contains a compilation error. Please check the following
    file:
    File: Model.java
    Line: 224
    Reason: Model is abstract; cannot be instantiated
    Done
    Build Server
    Build
    Passed?
    Repository
    Build
    Log
    Notification
    Hint Generators
    1 2
    3
    4
    [INFO] Scanning for projects...
    [INFO] Inspecting build with
    total of 1 modules...
    [ERROR] To see the full stack
    trace of the errors, re-run
    Maven with the -e switch.
    5
    7
    Developer
    Compilation
    Testing
    Code Analysis
    Dependencies
    6
    BART
    8
    StackOverflow Discussions
    9
    This is normal behaviour. Abstract classes are not supposed to be instantiated. You
    should test the classes which inherit from the abstract class, not the abstract class itself.
    Full Discussion: https://stackoverflow.com/questions/5028082/rails-3-activerecord-
    abstract-objects
    StackOverflow Analysis
    Related Post 1:

    View Slide

  26. Empirical Study
    RQ1 Are summarised logs more understandable?
    RQ2 Does a semi-automated support system
    influence the time that is required to fix a broken
    build?
    26

    View Slide

  27. Study Context
    3 Open-source systems
    • ActiveJPA (39,335 LOC, 143 GitHub stars, 123 builds)
    • Sentry-Java (113,332 LOC, 312 GitHub stars, 509 builds)
    • Fongo (31,088 LOC, 374 GitHub stars, 404 builds)
    8 Participants
    27

    View Slide

  28. Study Procedure
    Bugs injection to provoke Compilation, Dependencies, Code Analysis, and
    Testing build failures.
    Task 1: Build Summaries (Hints) VS Logs
    • Understandability
    • Relevance and Applicability of Proposed Solutions
    Task 2: Developer Performance With and Without BART
    • Build-Fix Time
    28

    View Slide

  29. Hint Understandability
    29
    Very Low Below Average Average Above Average Very High
    0%
    0%
    0%
    0%
    100%
    100%
    100%
    88%
    0%
    0%
    0%
    12%
    100 50 0 50 100
    Percentage
    Response VL BA A AA VH
    Code Analysis
    Testing
    Compilation
    Dependencies
    “A structured summary is way easier to grasp than

    many unstructured lines of text” (S4)

    View Slide

  30. 30
    Very Low Below Average Average Above Average Very High
    25%
    0%
    12%
    75%
    50%
    75%
    75%
    0%
    100 50 0 50 100
    Percentage
    Response VL BA A AA VH
    12%
    25%
    25%
    25%
    Code Analysis
    Compilation
    Dependencies
    Testing
    38%
    12%
    25%
    75%
    50%
    75%
    75%
    0%
    100 50 0 50 100
    Percentage
    25%
    0%
    12%
    12%
    Code Analysis
    Compilation
    Dependencies
    Testing
    Solution Hints
    Relevance
    Applicability

    View Slide

  31. 31
    Build-Fix Performance

    View Slide

  32. 32
    Build-Fix Time with BART
    %
    -27
    in case of Compilation
    %
    -33
    in case of Code Analysis
    %
    -42
    in case of Testing
    %
    -62
    in case of Dependencies

    View Slide

  33. 33
    Average Build-Fix Time with BART
    %
    -41
    “Less searching for relevant part, hence fast bug resolution” (S5)
    “Maven logs tend to be verbose, having a summary greatly reduces the time
    needed to find and correct a build failure” (S5)

    View Slide

  34. 34
    Holis;c Build Failures
    Build summaries as support for
    build-failures inspec;on

    View Slide

  35. 35
    Un-Break My Build:
    Assisting Developers with Build Repair Hints.
    Carmine Vassallo, Sebastian Proksch, Timothy Zemp,
    and Harald C. Gall.
    @ccvassallo
    [email protected]
    X
    Developer
    Commit
    Build Pass?
    Developer Repository Build Server
    Yes
    No
    Commit
    [INFO] Scanning for projects...
    [INFO] Inspecting build with
    total of 1 modules...
    [ERROR] To see the full stack
    trace of the errors, re-run
    Maven with the -e switch.
    Build Success
    Build Failure
    Build log
    Poll
    Read
    Data Extraction
    Extract Generate
    External Resources
    Hint
    Build Summary
    BART (Build Abstraction and Recovery Tool)
    Meta-Model
    Data Enrichment
    Hint Understandability
    X
    Very Low Below Average Average Above Average Very High
    0%
    0%
    0%
    0%
    100%
    100%
    100%
    88%
    0%
    0%
    0%
    12%
    100 50 0 50 100
    Percentage
    Response VL BA A AA VH
    Code Analysis
    Testing
    Compilation
    Dependencies
    “A structured summary is way easier to grasp than

    many unstructured lines of text” (S4)
    X
    Average Build-Fix Time with BART
    %
    -41
    “Less searching for relevant part, hence fast bug resolution” (S5)
    “Maven logs tend to be verbose, having a summary greatly reduces the time
    needed to find and correct a build failure” (S5)

    View Slide