Un-Break My Build: Assisting Developers with Build Repair Hints.

Our study on developer-oriented assistance for build failures presented at the IEEE/ACM International Conference on Program Comprehension (Technical Track, ICPC '18) in Gothenburg, Sweden. Preprint of the corresponding paper available at http://www.ifi.uzh.ch/seal/people/vassallo/VassalloICPC18.pdf

Transcript

1. Un-Break My Build:
   Assisting Developers with Build Repair Hints.
   Carmine Vassallo, Sebastian Proksch, Timothy Zemp, and Harald C. Gall.
   @ccvassallo
   IEEE/ACM International Conference on Program Comprehension (Technical Track), Gothenburg, Sweden 2018
   

   View Slide

2. Continuous Integration (CI)
   2
   Developer
   Commit
   Developer Repository Build Server
   Commit Poll
   

   View Slide

3. 3
   Developer
   Commit
   Build Pass?
   Developer Repository Build Server
   Yes
   No
   Commit
   Build Success
   Build Failure
   Poll
   Continuous Integration (CI)
   

   View Slide

4. Early Error Detection in CI: Build Failures
   4
   Developer
   Commit
   Build Pass?
   Developer Repository Build Server
   Yes
   No
   Commit
   Build Success
   Build Failure
   Read
   Poll
   [INFO] Scanning for projects...
   [INFO] Inspecting build with
   total of 1 modules...
   [ERROR] To see the full stack
   trace of the errors, re-run
   Maven with the -e switch.
   Build Log
   

   View Slide

5. Why Do We Need to Study
   Build Failures?
   5
   

   View Slide

6. 6
   Google Guava
   Build Server: TravisCI
   Build Tool: Maven
   40431 lines (Job #3.1)
   https://travis-ci.org/google/guava/jobs/40227447
   

   View Slide

7. 7
   ‑1572 lines
   

   View Slide

8. 8
   Modules being built
   

   View Slide

9. 9
   ‑38828 lines
   

   View Slide

10. 10
    Build Outcome
    Failed Build Step
    

    View Slide

11. 11
    ‐4219 lines
    

    View Slide

12. 12
    Test Execution
    

    View Slide

13. 13
    Barriers when Solving Build Failures
    Hilton et al.,
    “Trade-offs in continuous integration:
    assurance, security, and flexibility”
    FSE 2017
    Build Logs are quite large in size
    (100k lines)
    Build logs lack of important
    informa;on (e.g., full test report)
    

    View Slide

14. 14
    Build Un-Breaking takes long time
    Kerzazi et al.,
    “Why do Automated Builds Break?
    An Empirical Study”
    ICSME 2014
    On average, a build break takes
    1 hour to be fixed
    Significant amount of working
    hours spent on build fixing
    

    View Slide

15. How Can We Mitigate the
    Build-Failure Overhead?
    15
    

    View Slide

16. 16
    From BIG To small
    

    View Slide

17. 17
    Developer-Oriented Assistance for Build Failure
    

    View Slide

18. View Slide

19. 19
    Automated Build Fix
    (Macho et al., SANER 2018)
    

    View Slide

20. 20
    Automated Build Fix
    (Macho et al., SANER 2018)
    Developer-Oriented Assistance
    (BART)
    

    View Slide

21. 21
    Developer
    Commit
    Build Pass?
    Developer Repository Build Server
    Yes
    No
    Commit
    [INFO] Scanning for projects...
    [INFO] Inspecting build with
    total of 1 modules...
    [ERROR] To see the full stack
    trace of the errors, re-run
    Maven with the -e switch.
    Build Success
    Build Failure
    Build log
    Poll
    Read
    Data Extraction
    Extract Generate
    External Resources
    Hint
    Build Summary
    BART (Build Abstraction and Recovery Tool)
    Meta-Model
    Data Enrichment
    

    View Slide

22. 22
    From BIG To small
    

    View Slide

23. 23
    From BIG To small
    Build Summary
    Build Status Failed
    Failed Goal org.apache.maven.plugins:maven-compiler-plugin:2.1:compile
    Error Cause Compilation Failure
    Reactor
    ActiveJpa SUCCESS
    test SUCCESS
    core FAILURE
    utils SKIPPED
    Reason for Build Failure: Compilation Failure
    Hint: Compilation
    Your build contains a compilation error. Please check the following
    file:
    File: Model.java
    Line: 224
    Reason: Model is abstract; cannot be instantiated
    Done
    Build Server
    Build
    Passed?
    Repository
    Build
    Log
    Notification
    Hint Generators
    1 2
    3
    4
    [INFO] Scanning for projects...
    [INFO] Inspecting build with
    total of 1 modules...
    [ERROR] To see the full stack
    trace of the errors, re-run
    Maven with the -e switch.
    5
    7
    Developer
    Compilation
    Testing
    Code Analysis
    Dependencies
    6
    BART
    8
    StackOverflow Discussions
    9
    14 lines
    ≈ 40k lines
    

    View Slide

24. 24
    From BIG To small
    Build Summary
    Build Status Failed
    Failed Goal org.apache.maven.plugins:maven-compiler-plugin:2.1:compile
    Error Cause Compilation Failure
    Reactor
    ActiveJpa SUCCESS
    test SUCCESS
    core FAILURE
    utils SKIPPED
    Reason for Build Failure: Compilation Failure
    Hint: Compilation
    Your build contains a compilation error. Please check the following
    file:
    File: Model.java
    Line: 224
    Reason: Model is abstract; cannot be instantiated
    Done
    Build Server
    Build
    Passed?
    Repository
    Build
    Log
    Notification
    Hint Generators
    1 2
    3
    4
    [INFO] Scanning for projects...
    [INFO] Inspecting build with
    total of 1 modules...
    [ERROR] To see the full stack
    trace of the errors, re-run
    Maven with the -e switch.
    5
    7
    Developer
    Compilation
    Testing
    Code Analysis
    Dependencies
    6
    BART
    8
    StackOverflow Discussions
    9
    14 lines
    ≈ 40k lines
    

    View Slide

25. Hint Generators
    Hints for the most 4 recurrent build-
    failure types (Vassallo et al., ICSME 2017)
    • Compilation
    • Dependencies
    • Code Analysis
    • Testing
    External Resource: StackOverflow
    25
    Build Summary
    Build Status Failed
    Failed Goal org.apache.maven.plugins:maven-compiler-plugin:2.1:compile
    Error Cause Compilation Failure
    Reactor
    ActiveJpa SUCCESS
    test SUCCESS
    core FAILURE
    utils SKIPPED
    Reason for Build Failure: Compilation Failure
    Hint: Compilation
    Your build contains a compilation error. Please check the following
    file:
    File: Model.java
    Line: 224
    Reason: Model is abstract; cannot be instantiated
    Done
    Build Server
    Build
    Passed?
    Repository
    Build
    Log
    Notification
    Hint Generators
    1 2
    3
    4
    [INFO] Scanning for projects...
    [INFO] Inspecting build with
    total of 1 modules...
    [ERROR] To see the full stack
    trace of the errors, re-run
    Maven with the -e switch.
    5
    7
    Developer
    Compilation
    Testing
    Code Analysis
    Dependencies
    6
    BART
    8
    StackOverflow Discussions
    9
    This is normal behaviour. Abstract classes are not supposed to be instantiated. You
    should test the classes which inherit from the abstract class, not the abstract class itself.
    Full Discussion: https://stackoverflow.com/questions/5028082/rails-3-activerecord-
    abstract-objects
    StackOverflow Analysis
    Related Post 1:
    

    View Slide

26. Empirical Study
    RQ1 Are summarised logs more understandable?
    RQ2 Does a semi-automated support system
    influence the time that is required to fix a broken
    build?
    26
    

    View Slide

27. Study Context
    3 Open-source systems
    • ActiveJPA (39,335 LOC, 143 GitHub stars, 123 builds)
    • Sentry-Java (113,332 LOC, 312 GitHub stars, 509 builds)
    • Fongo (31,088 LOC, 374 GitHub stars, 404 builds)
    8 Participants
    27
    

    View Slide

28. Study Procedure
    Bugs injection to provoke Compilation, Dependencies, Code Analysis, and
    Testing build failures.
    Task 1: Build Summaries (Hints) VS Logs
    • Understandability
    • Relevance and Applicability of Proposed Solutions
    Task 2: Developer Performance With and Without BART
    • Build-Fix Time
    28
    

    View Slide

29. Hint Understandability
    29
    Very Low Below Average Average Above Average Very High
    0%
    0%
    0%
    0%
    100%
    100%
    100%
    88%
    0%
    0%
    0%
    12%
    100 50 0 50 100
    Percentage
    Response VL BA A AA VH
    Code Analysis
    Testing
    Compilation
    Dependencies
    “A structured summary is way easier to grasp than
    
    many unstructured lines of text” (S4)
    

    View Slide

30. 30
    Very Low Below Average Average Above Average Very High
    25%
    0%
    12%
    75%
    50%
    75%
    75%
    0%
    100 50 0 50 100
    Percentage
    Response VL BA A AA VH
    12%
    25%
    25%
    25%
    Code Analysis
    Compilation
    Dependencies
    Testing
    38%
    12%
    25%
    75%
    50%
    75%
    75%
    0%
    100 50 0 50 100
    Percentage
    25%
    0%
    12%
    12%
    Code Analysis
    Compilation
    Dependencies
    Testing
    Solution Hints
    Relevance
    Applicability
    

    View Slide

31. 31
    Build-Fix Performance
    

    View Slide

32. 32
    Build-Fix Time with BART
    %
    -27
    in case of Compilation
    %
    -33
    in case of Code Analysis
    %
    -42
    in case of Testing
    %
    -62
    in case of Dependencies
    

    View Slide

33. 33
    Average Build-Fix Time with BART
    %
    -41
    “Less searching for relevant part, hence fast bug resolution” (S5)
    “Maven logs tend to be verbose, having a summary greatly reduces the time
    needed to find and correct a build failure” (S5)
    

    View Slide

34. 34
    Holis;c Build Failures
    Build summaries as support for
    build-failures inspec;on
    

    View Slide

35. 35
    Un-Break My Build:
    Assisting Developers with Build Repair Hints.
    Carmine Vassallo, Sebastian Proksch, Timothy Zemp,
    and Harald C. Gall.
    @ccvassallo
    vassallo@ifi.uzh.ch
    X
    Developer
    Commit
    Build Pass?
    Developer Repository Build Server
    Yes
    No
    Commit
    [INFO] Scanning for projects...
    [INFO] Inspecting build with
    total of 1 modules...
    [ERROR] To see the full stack
    trace of the errors, re-run
    Maven with the -e switch.
    Build Success
    Build Failure
    Build log
    Poll
    Read
    Data Extraction
    Extract Generate
    External Resources
    Hint
    Build Summary
    BART (Build Abstraction and Recovery Tool)
    Meta-Model
    Data Enrichment
    Hint Understandability
    X
    Very Low Below Average Average Above Average Very High
    0%
    0%
    0%
    0%
    100%
    100%
    100%
    88%
    0%
    0%
    0%
    12%
    100 50 0 50 100
    Percentage
    Response VL BA A AA VH
    Code Analysis
    Testing
    Compilation
    Dependencies
    “A structured summary is way easier to grasp than
    
    many unstructured lines of text” (S4)
    X
    Average Build-Fix Time with BART
    %
    -41
    “Less searching for relevant part, hence fast bug resolution” (S5)
    “Maven logs tend to be verbose, having a summary greatly reduces the time
    needed to find and correct a build failure” (S5)
    

    View Slide