Un-Break My Build: Assisting Developers with Build Repair Hints.

Our study on developer-oriented assistance for build failures presented at the IEEE/ACM International Conference on Program Comprehension (Technical Track, ICPC '18) in Gothenburg, Sweden. Preprint of the corresponding paper available at http://www.ifi.uzh.ch/seal/people/vassallo/VassalloICPC18.pdf

Transcript

1. Un-Break My Build: Assisting Developers with Build Repair Hints. Carmine Vassallo, Sebastian Proksch, Timothy Zemp, and Harald C. Gall. @ccvassallo IEEE/ACM International Conference on Program Comprehension (Technical Track), Gothenburg, Sweden 2018
2. Continuous Integration (CI) 2 Developer Commit Developer Repository Build Server Commit Poll
3. 3 Developer Commit Build Pass? Developer Repository Build Server Yes No Commit Build Success Build Failure Poll Continuous Integration (CI)
4. Early Error Detection in CI: Build Failures 4 Developer Commit Build Pass? Developer Repository Build Server Yes No Commit Build Success Build Failure Read Poll [INFO] Scanning for projects... [INFO] Inspecting build with total of 1 modules... [ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch. Build Log
5. Why Do We Need to Study Build Failures? 5
6. 6 Google Guava Build Server: TravisCI Build Tool: Maven 40431 lines (Job #3.1) https://travis-ci.org/google/guava/jobs/40227447
7. 7 ‑1572 lines
8. 8 Modules being built
9. 9 ‑38828 lines
10. 10 Build Outcome Failed Build Step
11. 11 ‐4219 lines
12. 12 Test Execution
13. 13 Barriers when Solving Build Failures Hilton et al., “Trade-offs in continuous integration: assurance, security, and flexibility” FSE 2017 Build Logs are quite large in size (100k lines) Build logs lack of important informa;on (e.g., full test report)
14. 14 Build Un-Breaking takes long time Kerzazi et al., “Why do Automated Builds Break? An Empirical Study” ICSME 2014 On average, a build break takes 1 hour to be fixed Significant amount of working hours spent on build fixing
15. How Can We Mitigate the Build-Failure Overhead? 15
16. 16 From BIG To small
17. 17 Developer-Oriented Assistance for Build Failure
19. 19 Automated Build Fix (Macho et al., SANER 2018)
20. 20 Automated Build Fix (Macho et al., SANER 2018) Developer-Oriented Assistance (BART)
21. 21 Developer Commit Build Pass? Developer Repository Build Server Yes No Commit [INFO] Scanning for projects... [INFO] Inspecting build with total of 1 modules... [ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch. Build Success Build Failure Build log Poll Read Data Extraction Extract Generate External Resources Hint Build Summary BART (Build Abstraction and Recovery Tool) Meta-Model Data Enrichment
22. 22 From BIG To small
23. 23 From BIG To small Build Summary Build Status Failed Failed Goal org.apache.maven.plugins:maven-compiler-plugin:2.1:compile Error Cause Compilation Failure Reactor ActiveJpa SUCCESS test SUCCESS core FAILURE utils SKIPPED Reason for Build Failure: Compilation Failure Hint: Compilation Your build contains a compilation error. Please check the following file: File: Model.java Line: 224 Reason: Model is abstract; cannot be instantiated Done Build Server Build Passed? Repository Build Log Notification Hint Generators 1 2 3 4 [INFO] Scanning for projects... [INFO] Inspecting build with total of 1 modules... [ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch. 5 7 Developer Compilation Testing Code Analysis Dependencies 6 BART 8 StackOverflow Discussions 9 14 lines ≈ 40k lines
24. 24 From BIG To small Build Summary Build Status Failed Failed Goal org.apache.maven.plugins:maven-compiler-plugin:2.1:compile Error Cause Compilation Failure Reactor ActiveJpa SUCCESS test SUCCESS core FAILURE utils SKIPPED Reason for Build Failure: Compilation Failure Hint: Compilation Your build contains a compilation error. Please check the following file: File: Model.java Line: 224 Reason: Model is abstract; cannot be instantiated Done Build Server Build Passed? Repository Build Log Notification Hint Generators 1 2 3 4 [INFO] Scanning for projects... [INFO] Inspecting build with total of 1 modules... [ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch. 5 7 Developer Compilation Testing Code Analysis Dependencies 6 BART 8 StackOverflow Discussions 9 14 lines ≈ 40k lines
25. Hint Generators Hints for the most 4 recurrent build- failure types (Vassallo et al., ICSME 2017) • Compilation • Dependencies • Code Analysis • Testing External Resource: StackOverflow 25 Build Summary Build Status Failed Failed Goal org.apache.maven.plugins:maven-compiler-plugin:2.1:compile Error Cause Compilation Failure Reactor ActiveJpa SUCCESS test SUCCESS core FAILURE utils SKIPPED Reason for Build Failure: Compilation Failure Hint: Compilation Your build contains a compilation error. Please check the following file: File: Model.java Line: 224 Reason: Model is abstract; cannot be instantiated Done Build Server Build Passed? Repository Build Log Notification Hint Generators 1 2 3 4 [INFO] Scanning for projects... [INFO] Inspecting build with total of 1 modules... [ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch. 5 7 Developer Compilation Testing Code Analysis Dependencies 6 BART 8 StackOverflow Discussions 9 This is normal behaviour. Abstract classes are not supposed to be instantiated. You should test the classes which inherit from the abstract class, not the abstract class itself. Full Discussion: https://stackoverflow.com/questions/5028082/rails-3-activerecord- abstract-objects StackOverflow Analysis Related Post 1:
26. Empirical Study RQ1 Are summarised logs more understandable? RQ2 Does a semi-automated support system influence the time that is required to fix a broken build? 26
27. Study Context 3 Open-source systems • ActiveJPA (39,335 LOC, 143 GitHub stars, 123 builds) • Sentry-Java (113,332 LOC, 312 GitHub stars, 509 builds) • Fongo (31,088 LOC, 374 GitHub stars, 404 builds) 8 Participants 27
28. Study Procedure Bugs injection to provoke Compilation, Dependencies, Code Analysis, and Testing build failures. Task 1: Build Summaries (Hints) VS Logs • Understandability • Relevance and Applicability of Proposed Solutions Task 2: Developer Performance With and Without BART • Build-Fix Time 28
29. Hint Understandability 29 Very Low Below Average Average Above Average Very High 0% 0% 0% 0% 100% 100% 100% 88% 0% 0% 0% 12% 100 50 0 50 100 Percentage Response VL BA A AA VH Code Analysis Testing Compilation Dependencies “A structured summary is way easier to grasp than many unstructured lines of text” (S4)
30. 30 Very Low Below Average Average Above Average Very High 25% 0% 12% 75% 50% 75% 75% 0% 100 50 0 50 100 Percentage Response VL BA A AA VH 12% 25% 25% 25% Code Analysis Compilation Dependencies Testing 38% 12% 25% 75% 50% 75% 75% 0% 100 50 0 50 100 Percentage 25% 0% 12% 12% Code Analysis Compilation Dependencies Testing Solution Hints Relevance Applicability
31. 31 Build-Fix Performance
32. 32 Build-Fix Time with BART % -27 in case of Compilation % -33 in case of Code Analysis % -42 in case of Testing % -62 in case of Dependencies
33. 33 Average Build-Fix Time with BART % -41 “Less searching for relevant part, hence fast bug resolution” (S5) “Maven logs tend to be verbose, having a summary greatly reduces the time needed to find and correct a build failure” (S5)
34. 34 Holis;c Build Failures Build summaries as support for build-failures inspec;on
35. 35 Un-Break My Build: Assisting Developers with Build Repair Hints. Carmine Vassallo, Sebastian Proksch, Timothy Zemp, and Harald C. Gall. @ccvassallo vassallo@ifi.uzh.ch X Developer Commit Build Pass? Developer Repository Build Server Yes No Commit [INFO] Scanning for projects... [INFO] Inspecting build with total of 1 modules... [ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch. Build Success Build Failure Build log Poll Read Data Extraction Extract Generate External Resources Hint Build Summary BART (Build Abstraction and Recovery Tool) Meta-Model Data Enrichment Hint Understandability X Very Low Below Average Average Above Average Very High 0% 0% 0% 0% 100% 100% 100% 88% 0% 0% 0% 12% 100 50 0 50 100 Percentage Response VL BA A AA VH Code Analysis Testing Compilation Dependencies “A structured summary is way easier to grasp than many unstructured lines of text” (S4) X Average Build-Fix Time with BART % -41 “Less searching for relevant part, hence fast bug resolution” (S5) “Maven logs tend to be verbose, having a summary greatly reduces the time needed to find and correct a build failure” (S5)