Share
Embed
Developer-Oriented Assistance for Build Failure Resolution
Slides on Developer-Oriented Assistance for Build Failure Resolution discussed during In-Vivo Analytics for Big Software Quality at Lorentz Center in Leiden (The Netherlands).
Carmine Vassallo
September 18, 2018
Transcript
36 slides
-
Developer-Oriented Assistance for Build Failure Resolution Carmine Vassallo @ccvassallo In-Vivo
Analytics for Big Software Quality, Sep. 17-21 2018, Lorentz Center, Leiden (The Netherlands) -
2 CI Analytics CI My Research Overview CI Failures Debugging
CI Configuration in review ;) A Tale of CI Build Failures: an Open Source and a Financial Organisation Perspective. (ICSME ‘17) Context Is King: The Developer Perspective on the Usage of Static Analysis Tools. (SANER ‘18) Un-Break My Build: Assisting Developers with Build Repair Hints.(ICPC ‘18) Continuous Code Quality: Are We (Really) Doing That? (ASE ‘18) Continuous Refactoring in CI: A Preliminary Study On the Perceived Advantages and Barriers. (ICSME ‘18) -
4 Developer Commit Build Pass? Developer Repository Build Server Yes
No Commit Build Success Build Failure Poll Continuous Integration (CI) -
Early Error Detection: Build Failures 5 Developer Commit Build Pass?
Developer Repository Build Server Yes No Commit Build Success Build Failure Read Poll [INFO] Scanning for projects... [INFO] Inspecting build with total of 1 modules... [ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch. Build Log -
7 Google Guava Build Server: TravisCI Build Tool: Maven 40431
lines (Job #3.1) https://travis-ci.org/google/guava/jobs/40227447 -
14 Barriers when Solving Build Failures Hilton et al., “Trade-offs
in continuous integration: assurance, security, and flexibility” FSE 2017 Build Logs are quite large in size (100k lines) Build logs lack of important information (e.g., full test report) -
15 Build Un-Breaking takes long time Kerzazi et al., “Why
do Automated Builds Break? An Empirical Study” ICSME 2014 On average, a build break takes 1 hour to be fixed Significant amount of working hours spent on build fixing -
21 Automated Build Fix (Macho et al., SANER 2018) Developer-Oriented
Assistance (BART) -
22 Developer Commit Build Pass? Developer Repository Build Server Yes
No Commit [INFO] Scanning for projects... [INFO] Inspecting build with total of 1 modules... [ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch. Build Success Build Failure Build log Poll Read Data Extraction Extract Generate External Resources Hint Build Summary BART (Build Abstraction and Recovery Tool) Meta-Model Data Enrichment -
24 From BIG To small Build Summary Build Status Failed
Failed Goal org.apache.maven.plugins:maven-compiler-plugin:2.1:compile Error Cause Compilation Failure Reactor ActiveJpa SUCCESS test SUCCESS core FAILURE utils SKIPPED Reason for Build Failure: Compilation Failure Hint: Compilation Your build contains a compilation error. Please check the following file: File: Model.java Line: 224 Reason: Model is abstract; cannot be instantiated Done Build Server Build Passed? Repository Build Log Notification Hint Generators 1 2 3 4 [INFO] Scanning for projects... [INFO] Inspecting build with total of 1 modules... [ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch. 5 7 Developer Compilation Testing Code Analysis Dependencies 6 BART 8 StackOverflow Discussions 9 14 lines ≈ 40k lines -
25 From BIG To small Build Summary Build Status Failed
Failed Goal org.apache.maven.plugins:maven-compiler-plugin:2.1:compile Error Cause Compilation Failure Reactor ActiveJpa SUCCESS test SUCCESS core FAILURE utils SKIPPED Reason for Build Failure: Compilation Failure Hint: Compilation Your build contains a compilation error. Please check the following file: File: Model.java Line: 224 Reason: Model is abstract; cannot be instantiated Done Build Server Build Passed? Repository Build Log Notification Hint Generators 1 2 3 4 [INFO] Scanning for projects... [INFO] Inspecting build with total of 1 modules... [ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch. 5 7 Developer Compilation Testing Code Analysis Dependencies 6 BART 8 StackOverflow Discussions 9 14 lines ≈ 40k lines -
Hint Generators Hints for the most 4 recurrent build- failure
types (Vassallo et al., ICSME 2017) • Compilation • Dependencies • Code Analysis • Testing External Resource: StackOverflow 26 Build Summary Build Status Failed Failed Goal org.apache.maven.plugins:maven-compiler-plugin:2.1:compile Error Cause Compilation Failure Reactor ActiveJpa SUCCESS test SUCCESS core FAILURE utils SKIPPED Reason for Build Failure: Compilation Failure Hint: Compilation Your build contains a compilation error. Please check the following file: File: Model.java Line: 224 Reason: Model is abstract; cannot be instantiated Done Build Server Build Passed? Repository Build Log Notification Hint Generators 1 2 3 4 [INFO] Scanning for projects... [INFO] Inspecting build with total of 1 modules... [ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch. 5 7 Developer Compilation Testing Code Analysis Dependencies 6 BART 8 StackOverflow Discussions 9 This is normal behaviour. Abstract classes are not supposed to be instantiated. You should test the classes which inherit from the abstract class, not the abstract class itself. Full Discussion: https://stackoverflow.com/questions/5028082/rails-3-activerecord- abstract-objects StackOverflow Analysis Related Post 1: -
Empirical Study RQ1 Are summarised logs more understandable? RQ2 Does
a semi-automated support system influence the time that is required to fix a broken build? 27 -
Study Context 3 Open-source systems • ActiveJPA (39,335 LOC, 143
GitHub stars, 123 builds) • Sentry-Java (113,332 LOC, 312 GitHub stars, 509 builds) • Fongo (31,088 LOC, 374 GitHub stars, 404 builds) 8 Participants 28 -
Study Procedure Bugs injection to provoke Compilation, Dependencies, Code Analysis,
and Testing build failures. Task 1: Build Summaries (Hints) VS Logs • Understandability • Relevance and Applicability of Proposed Solutions Task 2: Developer Performance With and Without BART • Build-Fix Time 29 -
Hint Understandability 30 Very Low Below Average Average Above Average
Very High 0% 0% 0% 0% 100% 100% 100% 88% 0% 0% 0% 12% 100 50 0 50 100 Percentage Response VL BA A AA VH Code Analysis Testing Compilation Dependencies “A structured summary is way easier to grasp than many unstructured lines of text” (S4) -
31 Very Low Below Average Average Above Average Very High
25% 0% 12% 75% 50% 75% 75% 0% 100 50 0 50 100 Percentage Response VL BA A AA VH 12% 25% 25% 25% Code Analysis Compilation Dependencies Testing 38% 12% 25% 75% 50% 75% 75% 0% 100 50 0 50 100 Percentage 25% 0% 12% 12% Code Analysis Compilation Dependencies Testing Solution Hints Relevance Applicability -
33 Build-Fix Time with BART % -27 in case of
Compilation % -33 in case of Code Analysis % -42 in case of Testing % -62 in case of Dependencies -
34 Average Build-Fix Time with BART % -41 “Less searching
for relevant part, hence fast bug resolution” (S5) “Maven logs tend to be verbose, having a summary greatly reduces the time needed to find and correct a build failure” (S5) -
36 Developer-oriented Assistance for Build Failure Resolution. Carmine Vassallo. @ccvassallo
[email protected] X Developer Commit Build Pass? Developer Repository Build Server Yes No Commit [INFO] Scanning for projects... [INFO] Inspecting build with total of 1 modules... [ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch. Build Success Build Failure Build log Poll Read Data Extraction Extract Generate External Resources Hint Build Summary BART (Build Abstraction and Recovery Tool) Meta-Model Data Enrichment Hint Understandability X Very Low Below Average Average Above Average Very High 0% 0% 0% 0% 100% 100% 100% 88% 0% 0% 0% 12% 100 50 0 50 100 Percentage Response VL BA A AA VH Code Analysis Testing Compilation Dependencies “A structured summary is way easier to grasp than many unstructured lines of text” (S4) X CI Analytics CI My Research Overview CI Failures Debugging CI Configuration in review ;) A Tale of CI Build Failures: an Open Source and a Financial Organisation Perspective. (ICSME ‘17) Context Is King: The Developer Perspective on the Usage of Static Analysis Tools. (SANER ‘18) Un-Break My Build: Assisting Developers with Build Repair Hints.(ICPC ‘18) Continuous Code Quality: Are We (Really) Doing That? (ASE ‘18) Continuous Refactoring in CI: A Preliminary Study On the Perceived Advantages and Barriers. (ICSME ‘18)