Developer-Oriented Assistance for Build Failure Resolution

Slides on Developer-Oriented Assistance for Build Failure Resolution discussed during In-Vivo Analytics for Big Software Quality at Lorentz Center in Leiden (The Netherlands).

Transcript

1. Developer-Oriented Assistance for Build Failure Resolution Carmine Vassallo @ccvassallo In-Vivo Analytics for Big Software Quality, Sep. 17-21 2018, Lorentz Center, Leiden (The Netherlands)
2. 2 CI Analytics CI My Research Overview CI Failures Debugging CI Configuration in review ;) A Tale of CI Build Failures: an Open Source and a Financial Organisation Perspective. (ICSME ‘17) Context Is King: The Developer Perspective on the Usage of Static Analysis Tools. (SANER ‘18) Un-Break My Build: Assisting Developers with Build Repair Hints.(ICPC ‘18) Continuous Code Quality: Are We (Really) Doing That? (ASE ‘18) Continuous Refactoring in CI: A Preliminary Study On the Perceived Advantages and Barriers. (ICSME ‘18)
3. Continuous Integration (CI) 3 Developer Commit Developer Repository Build Server Commit Poll
4. 4 Developer Commit Build Pass? Developer Repository Build Server Yes No Commit Build Success Build Failure Poll Continuous Integration (CI)
5. Early Error Detection: Build Failures 5 Developer Commit Build Pass? Developer Repository Build Server Yes No Commit Build Success Build Failure Read Poll [INFO] Scanning for projects... [INFO] Inspecting build with total of 1 modules... [ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch. Build Log
6. Why Do We Need to Study Build Failures? 6
7. 7 Google Guava Build Server: TravisCI Build Tool: Maven 40431 lines (Job #3.1) https://travis-ci.org/google/guava/jobs/40227447
8. 8 ‑1572 lines
9. 9 Modules being built
10. 10 ‑38828 lines
11. 11 Build Outcome Failed Build Step
12. 12 ‐4219 lines
13. 13 Test Execution
14. 14 Barriers when Solving Build Failures Hilton et al., “Trade-offs in continuous integration: assurance, security, and flexibility” FSE 2017 Build Logs are quite large in size (100k lines) Build logs lack of important information (e.g., full test report)
15. 15 Build Un-Breaking takes long time Kerzazi et al., “Why do Automated Builds Break? An Empirical Study” ICSME 2014 On average, a build break takes 1 hour to be fixed Significant amount of working hours spent on build fixing
16. How Can We Mitigate the Build-Failure Overhead? 16
17. 17 From BIG To small
18. 18 Developer-Oriented Assistance for Build Failure
20. 20 Automated Build Fix (Macho et al., SANER 2018)
21. 21 Automated Build Fix (Macho et al., SANER 2018) Developer-Oriented Assistance (BART)
22. 22 Developer Commit Build Pass? Developer Repository Build Server Yes No Commit [INFO] Scanning for projects... [INFO] Inspecting build with total of 1 modules... [ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch. Build Success Build Failure Build log Poll Read Data Extraction Extract Generate External Resources Hint Build Summary BART (Build Abstraction and Recovery Tool) Meta-Model Data Enrichment
23. 23 From BIG To small
24. 24 From BIG To small Build Summary Build Status Failed Failed Goal org.apache.maven.plugins:maven-compiler-plugin:2.1:compile Error Cause Compilation Failure Reactor ActiveJpa SUCCESS test SUCCESS core FAILURE utils SKIPPED Reason for Build Failure: Compilation Failure Hint: Compilation Your build contains a compilation error. Please check the following file: File: Model.java Line: 224 Reason: Model is abstract; cannot be instantiated Done Build Server Build Passed? Repository Build Log Notification Hint Generators 1 2 3 4 [INFO] Scanning for projects... [INFO] Inspecting build with total of 1 modules... [ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch. 5 7 Developer Compilation Testing Code Analysis Dependencies 6 BART 8 StackOverflow Discussions 9 14 lines ≈ 40k lines
25. 25 From BIG To small Build Summary Build Status Failed Failed Goal org.apache.maven.plugins:maven-compiler-plugin:2.1:compile Error Cause Compilation Failure Reactor ActiveJpa SUCCESS test SUCCESS core FAILURE utils SKIPPED Reason for Build Failure: Compilation Failure Hint: Compilation Your build contains a compilation error. Please check the following file: File: Model.java Line: 224 Reason: Model is abstract; cannot be instantiated Done Build Server Build Passed? Repository Build Log Notification Hint Generators 1 2 3 4 [INFO] Scanning for projects... [INFO] Inspecting build with total of 1 modules... [ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch. 5 7 Developer Compilation Testing Code Analysis Dependencies 6 BART 8 StackOverflow Discussions 9 14 lines ≈ 40k lines
26. Hint Generators Hints for the most 4 recurrent build- failure types (Vassallo et al., ICSME 2017) • Compilation • Dependencies • Code Analysis • Testing External Resource: StackOverflow 26 Build Summary Build Status Failed Failed Goal org.apache.maven.plugins:maven-compiler-plugin:2.1:compile Error Cause Compilation Failure Reactor ActiveJpa SUCCESS test SUCCESS core FAILURE utils SKIPPED Reason for Build Failure: Compilation Failure Hint: Compilation Your build contains a compilation error. Please check the following file: File: Model.java Line: 224 Reason: Model is abstract; cannot be instantiated Done Build Server Build Passed? Repository Build Log Notification Hint Generators 1 2 3 4 [INFO] Scanning for projects... [INFO] Inspecting build with total of 1 modules... [ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch. 5 7 Developer Compilation Testing Code Analysis Dependencies 6 BART 8 StackOverflow Discussions 9 This is normal behaviour. Abstract classes are not supposed to be instantiated. You should test the classes which inherit from the abstract class, not the abstract class itself. Full Discussion: https://stackoverflow.com/questions/5028082/rails-3-activerecord- abstract-objects StackOverflow Analysis Related Post 1:
27. Empirical Study RQ1 Are summarised logs more understandable? RQ2 Does a semi-automated support system influence the time that is required to fix a broken build? 27
28. Study Context 3 Open-source systems • ActiveJPA (39,335 LOC, 143 GitHub stars, 123 builds) • Sentry-Java (113,332 LOC, 312 GitHub stars, 509 builds) • Fongo (31,088 LOC, 374 GitHub stars, 404 builds) 8 Participants 28
29. Study Procedure Bugs injection to provoke Compilation, Dependencies, Code Analysis, and Testing build failures. Task 1: Build Summaries (Hints) VS Logs • Understandability • Relevance and Applicability of Proposed Solutions Task 2: Developer Performance With and Without BART • Build-Fix Time 29
30. Hint Understandability 30 Very Low Below Average Average Above Average Very High 0% 0% 0% 0% 100% 100% 100% 88% 0% 0% 0% 12% 100 50 0 50 100 Percentage Response VL BA A AA VH Code Analysis Testing Compilation Dependencies “A structured summary is way easier to grasp than many unstructured lines of text” (S4)
31. 31 Very Low Below Average Average Above Average Very High 25% 0% 12% 75% 50% 75% 75% 0% 100 50 0 50 100 Percentage Response VL BA A AA VH 12% 25% 25% 25% Code Analysis Compilation Dependencies Testing 38% 12% 25% 75% 50% 75% 75% 0% 100 50 0 50 100 Percentage 25% 0% 12% 12% Code Analysis Compilation Dependencies Testing Solution Hints Relevance Applicability
32. 32 Build-Fix Performance
33. 33 Build-Fix Time with BART % -27 in case of Compilation % -33 in case of Code Analysis % -42 in case of Testing % -62 in case of Dependencies
34. 34 Average Build-Fix Time with BART % -41 “Less searching for relevant part, hence fast bug resolution” (S5) “Maven logs tend to be verbose, having a summary greatly reduces the time needed to find and correct a build failure” (S5)
35. 35 Holistic Build Failures Build summaries as support for build-failures inspection
36. 36 Developer-oriented Assistance for Build Failure Resolution. Carmine Vassallo. @ccvassallo vassallo@ifi.uzh.ch X Developer Commit Build Pass? Developer Repository Build Server Yes No Commit [INFO] Scanning for projects... [INFO] Inspecting build with total of 1 modules... [ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch. Build Success Build Failure Build log Poll Read Data Extraction Extract Generate External Resources Hint Build Summary BART (Build Abstraction and Recovery Tool) Meta-Model Data Enrichment Hint Understandability X Very Low Below Average Average Above Average Very High 0% 0% 0% 0% 100% 100% 100% 88% 0% 0% 0% 12% 100 50 0 50 100 Percentage Response VL BA A AA VH Code Analysis Testing Compilation Dependencies “A structured summary is way easier to grasp than many unstructured lines of text” (S4) X CI Analytics CI My Research Overview CI Failures Debugging CI Configuration in review ;) A Tale of CI Build Failures: an Open Source and a Financial Organisation Perspective. (ICSME ‘17) Context Is King: The Developer Perspective on the Usage of Static Analysis Tools. (SANER ‘18) Un-Break My Build: Assisting Developers with Build Repair Hints.(ICPC ‘18) Continuous Code Quality: Are We (Really) Doing That? (ASE ‘18) Continuous Refactoring in CI: A Preliminary Study On the Perceived Advantages and Barriers. (ICSME ‘18)