Save 37% off PRO during our Black Friday Sale! »

Un-Break My Build: Assisting Developers with Build Repair Hints.

Un-Break My Build: Assisting Developers with Build Repair Hints.

Our study on developer-oriented assistance for build failures presented at the IEEE/ACM International Conference on Program Comprehension (Technical Track, ICPC '18) in Gothenburg, Sweden. Preprint of the corresponding paper available at http://www.ifi.uzh.ch/seal/people/vassallo/VassalloICPC18.pdf

84581630350da72785f7d2448549ca06?s=128

Carmine Vassallo

May 27, 2018
Tweet

Transcript

  1. Un-Break My Build: Assisting Developers with Build Repair Hints. Carmine

    Vassallo, Sebastian Proksch, Timothy Zemp, and Harald C. Gall. @ccvassallo IEEE/ACM International Conference on Program Comprehension (Technical Track), Gothenburg, Sweden 2018
  2. Continuous Integration (CI) 2 Developer Commit Developer Repository Build Server

    Commit Poll
  3. 3 Developer Commit Build Pass? Developer Repository Build Server Yes

    No Commit Build Success Build Failure Poll Continuous Integration (CI)
  4. Early Error Detection in CI: Build Failures 4 Developer Commit

    Build Pass? Developer Repository Build Server Yes No Commit Build Success Build Failure Read Poll [INFO] Scanning for projects... [INFO] Inspecting build with total of 1 modules... [ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch. Build Log
  5. Why Do We Need to Study Build Failures? 5

  6. 6 Google Guava Build Server: TravisCI Build Tool: Maven 40431

    lines (Job #3.1) https://travis-ci.org/google/guava/jobs/40227447
  7. 7 ‑1572 lines

  8. 8 Modules being built

  9. 9 ‑38828 lines

  10. 10 Build Outcome Failed Build Step

  11. 11 ‐4219 lines

  12. 12 Test Execution

  13. 13 Barriers when Solving Build Failures Hilton et al., “Trade-offs

    in continuous integration: assurance, security, and flexibility” FSE 2017 Build Logs are quite large in size (100k lines) Build logs lack of important informa;on (e.g., full test report)
  14. 14 Build Un-Breaking takes long time Kerzazi et al., “Why

    do Automated Builds Break? An Empirical Study” ICSME 2014 On average, a build break takes 1 hour to be fixed Significant amount of working hours spent on build fixing
  15. How Can We Mitigate the Build-Failure Overhead? 15

  16. 16 From BIG To small

  17. 17 Developer-Oriented Assistance for Build Failure

  18. 18

  19. 19 Automated Build Fix (Macho et al., SANER 2018)

  20. 20 Automated Build Fix (Macho et al., SANER 2018) Developer-Oriented

    Assistance (BART)
  21. 21 Developer Commit Build Pass? Developer Repository Build Server Yes

    No Commit [INFO] Scanning for projects... [INFO] Inspecting build with total of 1 modules... [ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch. Build Success Build Failure Build log Poll Read Data Extraction Extract Generate External Resources Hint Build Summary BART (Build Abstraction and Recovery Tool) Meta-Model Data Enrichment
  22. 22 From BIG To small

  23. 23 From BIG To small Build Summary Build Status Failed

    Failed Goal org.apache.maven.plugins:maven-compiler-plugin:2.1:compile Error Cause Compilation Failure Reactor ActiveJpa SUCCESS test SUCCESS core FAILURE utils SKIPPED Reason for Build Failure: Compilation Failure Hint: Compilation Your build contains a compilation error. Please check the following file: File: Model.java Line: 224 Reason: Model is abstract; cannot be instantiated Done Build Server Build Passed? Repository Build Log Notification Hint Generators 1 2 3 4 [INFO] Scanning for projects... [INFO] Inspecting build with total of 1 modules... [ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch. 5 7 Developer Compilation Testing Code Analysis Dependencies 6 BART 8 StackOverflow Discussions 9 14 lines ≈ 40k lines
  24. 24 From BIG To small Build Summary Build Status Failed

    Failed Goal org.apache.maven.plugins:maven-compiler-plugin:2.1:compile Error Cause Compilation Failure Reactor ActiveJpa SUCCESS test SUCCESS core FAILURE utils SKIPPED Reason for Build Failure: Compilation Failure Hint: Compilation Your build contains a compilation error. Please check the following file: File: Model.java Line: 224 Reason: Model is abstract; cannot be instantiated Done Build Server Build Passed? Repository Build Log Notification Hint Generators 1 2 3 4 [INFO] Scanning for projects... [INFO] Inspecting build with total of 1 modules... [ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch. 5 7 Developer Compilation Testing Code Analysis Dependencies 6 BART 8 StackOverflow Discussions 9 14 lines ≈ 40k lines
  25. Hint Generators Hints for the most 4 recurrent build- failure

    types (Vassallo et al., ICSME 2017) • Compilation • Dependencies • Code Analysis • Testing External Resource: StackOverflow 25 Build Summary Build Status Failed Failed Goal org.apache.maven.plugins:maven-compiler-plugin:2.1:compile Error Cause Compilation Failure Reactor ActiveJpa SUCCESS test SUCCESS core FAILURE utils SKIPPED Reason for Build Failure: Compilation Failure Hint: Compilation Your build contains a compilation error. Please check the following file: File: Model.java Line: 224 Reason: Model is abstract; cannot be instantiated Done Build Server Build Passed? Repository Build Log Notification Hint Generators 1 2 3 4 [INFO] Scanning for projects... [INFO] Inspecting build with total of 1 modules... [ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch. 5 7 Developer Compilation Testing Code Analysis Dependencies 6 BART 8 StackOverflow Discussions 9 This is normal behaviour. Abstract classes are not supposed to be instantiated. You should test the classes which inherit from the abstract class, not the abstract class itself. Full Discussion: https://stackoverflow.com/questions/5028082/rails-3-activerecord- abstract-objects StackOverflow Analysis Related Post 1:
  26. Empirical Study RQ1 Are summarised logs more understandable? RQ2 Does

    a semi-automated support system influence the time that is required to fix a broken build? 26
  27. Study Context 3 Open-source systems • ActiveJPA (39,335 LOC, 143

    GitHub stars, 123 builds) • Sentry-Java (113,332 LOC, 312 GitHub stars, 509 builds) • Fongo (31,088 LOC, 374 GitHub stars, 404 builds) 8 Participants 27
  28. Study Procedure Bugs injection to provoke Compilation, Dependencies, Code Analysis,

    and Testing build failures. Task 1: Build Summaries (Hints) VS Logs • Understandability • Relevance and Applicability of Proposed Solutions Task 2: Developer Performance With and Without BART • Build-Fix Time 28
  29. Hint Understandability 29 Very Low Below Average Average Above Average

    Very High 0% 0% 0% 0% 100% 100% 100% 88% 0% 0% 0% 12% 100 50 0 50 100 Percentage Response VL BA A AA VH Code Analysis Testing Compilation Dependencies “A structured summary is way easier to grasp than many unstructured lines of text” (S4)
  30. 30 Very Low Below Average Average Above Average Very High

    25% 0% 12% 75% 50% 75% 75% 0% 100 50 0 50 100 Percentage Response VL BA A AA VH 12% 25% 25% 25% Code Analysis Compilation Dependencies Testing 38% 12% 25% 75% 50% 75% 75% 0% 100 50 0 50 100 Percentage 25% 0% 12% 12% Code Analysis Compilation Dependencies Testing Solution Hints Relevance Applicability
  31. 31 Build-Fix Performance

  32. 32 Build-Fix Time with BART % -27 in case of

    Compilation % -33 in case of Code Analysis % -42 in case of Testing % -62 in case of Dependencies
  33. 33 Average Build-Fix Time with BART % -41 “Less searching

    for relevant part, hence fast bug resolution” (S5) “Maven logs tend to be verbose, having a summary greatly reduces the time needed to find and correct a build failure” (S5)
  34. 34 Holis;c Build Failures Build summaries as support for build-failures

    inspec;on
  35. 35 Un-Break My Build: Assisting Developers with Build Repair Hints.

    Carmine Vassallo, Sebastian Proksch, Timothy Zemp, and Harald C. Gall. @ccvassallo vassallo@ifi.uzh.ch X Developer Commit Build Pass? Developer Repository Build Server Yes No Commit [INFO] Scanning for projects... [INFO] Inspecting build with total of 1 modules... [ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch. Build Success Build Failure Build log Poll Read Data Extraction Extract Generate External Resources Hint Build Summary BART (Build Abstraction and Recovery Tool) Meta-Model Data Enrichment Hint Understandability X Very Low Below Average Average Above Average Very High 0% 0% 0% 0% 100% 100% 100% 88% 0% 0% 0% 12% 100 50 0 50 100 Percentage Response VL BA A AA VH Code Analysis Testing Compilation Dependencies “A structured summary is way easier to grasp than many unstructured lines of text” (S4) X Average Build-Fix Time with BART % -41 “Less searching for relevant part, hence fast bug resolution” (S5) “Maven logs tend to be verbose, having a summary greatly reduces the time needed to find and correct a build failure” (S5)