Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Developer-Oriented Assistance for Build Failure Resolution

Developer-Oriented Assistance for Build Failure Resolution

Slides on Developer-Oriented Assistance for Build Failure Resolution discussed during In-Vivo Analytics for Big Software Quality at Lorentz Center in Leiden (The Netherlands).

84581630350da72785f7d2448549ca06?s=128

Carmine Vassallo

September 18, 2018
Tweet

Transcript

  1. Developer-Oriented Assistance for Build Failure Resolution Carmine Vassallo @ccvassallo In-Vivo

    Analytics for Big Software Quality, Sep. 17-21 2018, Lorentz Center, Leiden (The Netherlands)
  2. 2 CI Analytics CI My Research Overview CI Failures Debugging

    CI Configuration in review ;) A Tale of CI Build Failures: an Open Source and a Financial Organisation Perspective. (ICSME ‘17) Context Is King: The Developer Perspective on the Usage of Static Analysis Tools. (SANER ‘18) Un-Break My Build: Assisting Developers with Build Repair Hints.(ICPC ‘18) Continuous Code Quality: Are We (Really) Doing That? (ASE ‘18) Continuous Refactoring in CI: A Preliminary Study On the Perceived Advantages and Barriers. (ICSME ‘18)
  3. Continuous Integration (CI) 3 Developer Commit Developer Repository Build Server

    Commit Poll
  4. 4 Developer Commit Build Pass? Developer Repository Build Server Yes

    No Commit Build Success Build Failure Poll Continuous Integration (CI)
  5. Early Error Detection: Build Failures 5 Developer Commit Build Pass?

    Developer Repository Build Server Yes No Commit Build Success Build Failure Read Poll [INFO] Scanning for projects... [INFO] Inspecting build with total of 1 modules... [ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch. Build Log
  6. Why Do We Need to Study Build Failures? 6

  7. 7 Google Guava Build Server: TravisCI Build Tool: Maven 40431

    lines (Job #3.1) https://travis-ci.org/google/guava/jobs/40227447
  8. 8 ‑1572 lines

  9. 9 Modules being built

  10. 10 ‑38828 lines

  11. 11 Build Outcome Failed Build Step

  12. 12 ‐4219 lines

  13. 13 Test Execution

  14. 14 Barriers when Solving Build Failures Hilton et al., “Trade-offs

    in continuous integration: assurance, security, and flexibility” FSE 2017 Build Logs are quite large in size (100k lines) Build logs lack of important information (e.g., full test report)
  15. 15 Build Un-Breaking takes long time Kerzazi et al., “Why

    do Automated Builds Break? An Empirical Study” ICSME 2014 On average, a build break takes 1 hour to be fixed Significant amount of working hours spent on build fixing
  16. How Can We Mitigate the Build-Failure Overhead? 16

  17. 17 From BIG To small

  18. 18 Developer-Oriented Assistance for Build Failure

  19. 19

  20. 20 Automated Build Fix (Macho et al., SANER 2018)

  21. 21 Automated Build Fix (Macho et al., SANER 2018) Developer-Oriented

    Assistance (BART)
  22. 22 Developer Commit Build Pass? Developer Repository Build Server Yes

    No Commit [INFO] Scanning for projects... [INFO] Inspecting build with total of 1 modules... [ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch. Build Success Build Failure Build log Poll Read Data Extraction Extract Generate External Resources Hint Build Summary BART (Build Abstraction and Recovery Tool) Meta-Model Data Enrichment
  23. 23 From BIG To small

  24. 24 From BIG To small Build Summary Build Status Failed

    Failed Goal org.apache.maven.plugins:maven-compiler-plugin:2.1:compile Error Cause Compilation Failure Reactor ActiveJpa SUCCESS test SUCCESS core FAILURE utils SKIPPED Reason for Build Failure: Compilation Failure Hint: Compilation Your build contains a compilation error. Please check the following file: File: Model.java Line: 224 Reason: Model is abstract; cannot be instantiated Done Build Server Build Passed? Repository Build Log Notification Hint Generators 1 2 3 4 [INFO] Scanning for projects... [INFO] Inspecting build with total of 1 modules... [ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch. 5 7 Developer Compilation Testing Code Analysis Dependencies 6 BART 8 StackOverflow Discussions 9 14 lines ≈ 40k lines
  25. 25 From BIG To small Build Summary Build Status Failed

    Failed Goal org.apache.maven.plugins:maven-compiler-plugin:2.1:compile Error Cause Compilation Failure Reactor ActiveJpa SUCCESS test SUCCESS core FAILURE utils SKIPPED Reason for Build Failure: Compilation Failure Hint: Compilation Your build contains a compilation error. Please check the following file: File: Model.java Line: 224 Reason: Model is abstract; cannot be instantiated Done Build Server Build Passed? Repository Build Log Notification Hint Generators 1 2 3 4 [INFO] Scanning for projects... [INFO] Inspecting build with total of 1 modules... [ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch. 5 7 Developer Compilation Testing Code Analysis Dependencies 6 BART 8 StackOverflow Discussions 9 14 lines ≈ 40k lines
  26. Hint Generators Hints for the most 4 recurrent build- failure

    types (Vassallo et al., ICSME 2017) • Compilation • Dependencies • Code Analysis • Testing External Resource: StackOverflow 26 Build Summary Build Status Failed Failed Goal org.apache.maven.plugins:maven-compiler-plugin:2.1:compile Error Cause Compilation Failure Reactor ActiveJpa SUCCESS test SUCCESS core FAILURE utils SKIPPED Reason for Build Failure: Compilation Failure Hint: Compilation Your build contains a compilation error. Please check the following file: File: Model.java Line: 224 Reason: Model is abstract; cannot be instantiated Done Build Server Build Passed? Repository Build Log Notification Hint Generators 1 2 3 4 [INFO] Scanning for projects... [INFO] Inspecting build with total of 1 modules... [ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch. 5 7 Developer Compilation Testing Code Analysis Dependencies 6 BART 8 StackOverflow Discussions 9 This is normal behaviour. Abstract classes are not supposed to be instantiated. You should test the classes which inherit from the abstract class, not the abstract class itself. Full Discussion: https://stackoverflow.com/questions/5028082/rails-3-activerecord- abstract-objects StackOverflow Analysis Related Post 1:
  27. Empirical Study RQ1 Are summarised logs more understandable? RQ2 Does

    a semi-automated support system influence the time that is required to fix a broken build? 27
  28. Study Context 3 Open-source systems • ActiveJPA (39,335 LOC, 143

    GitHub stars, 123 builds) • Sentry-Java (113,332 LOC, 312 GitHub stars, 509 builds) • Fongo (31,088 LOC, 374 GitHub stars, 404 builds) 8 Participants 28
  29. Study Procedure Bugs injection to provoke Compilation, Dependencies, Code Analysis,

    and Testing build failures. Task 1: Build Summaries (Hints) VS Logs • Understandability • Relevance and Applicability of Proposed Solutions Task 2: Developer Performance With and Without BART • Build-Fix Time 29
  30. Hint Understandability 30 Very Low Below Average Average Above Average

    Very High 0% 0% 0% 0% 100% 100% 100% 88% 0% 0% 0% 12% 100 50 0 50 100 Percentage Response VL BA A AA VH Code Analysis Testing Compilation Dependencies “A structured summary is way easier to grasp than many unstructured lines of text” (S4)
  31. 31 Very Low Below Average Average Above Average Very High

    25% 0% 12% 75% 50% 75% 75% 0% 100 50 0 50 100 Percentage Response VL BA A AA VH 12% 25% 25% 25% Code Analysis Compilation Dependencies Testing 38% 12% 25% 75% 50% 75% 75% 0% 100 50 0 50 100 Percentage 25% 0% 12% 12% Code Analysis Compilation Dependencies Testing Solution Hints Relevance Applicability
  32. 32 Build-Fix Performance

  33. 33 Build-Fix Time with BART % -27 in case of

    Compilation % -33 in case of Code Analysis % -42 in case of Testing % -62 in case of Dependencies
  34. 34 Average Build-Fix Time with BART % -41 “Less searching

    for relevant part, hence fast bug resolution” (S5) “Maven logs tend to be verbose, having a summary greatly reduces the time needed to find and correct a build failure” (S5)
  35. 35 Holistic Build Failures Build summaries as support for build-failures

    inspection
  36. 36 Developer-oriented Assistance for Build Failure Resolution. Carmine Vassallo. @ccvassallo

    vassallo@ifi.uzh.ch X Developer Commit Build Pass? Developer Repository Build Server Yes No Commit [INFO] Scanning for projects... [INFO] Inspecting build with total of 1 modules... [ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch. Build Success Build Failure Build log Poll Read Data Extraction Extract Generate External Resources Hint Build Summary BART (Build Abstraction and Recovery Tool) Meta-Model Data Enrichment Hint Understandability X Very Low Below Average Average Above Average Very High 0% 0% 0% 0% 100% 100% 100% 88% 0% 0% 0% 12% 100 50 0 50 100 Percentage Response VL BA A AA VH Code Analysis Testing Compilation Dependencies “A structured summary is way easier to grasp than many unstructured lines of text” (S4) X CI Analytics CI My Research Overview CI Failures Debugging CI Configuration in review ;) A Tale of CI Build Failures: an Open Source and a Financial Organisation Perspective. (ICSME ‘17) Context Is King: The Developer Perspective on the Usage of Static Analysis Tools. (SANER ‘18) Un-Break My Build: Assisting Developers with Build Repair Hints.(ICPC ‘18) Continuous Code Quality: Are We (Really) Doing That? (ASE ‘18) Continuous Refactoring in CI: A Preliminary Study On the Perceived Advantages and Barriers. (ICSME ‘18)