Share
Embed
Un-Break My Build: Assisting Developers with Build Repair Hints.
Our study on developer-oriented assistance for build failures presented at the IEEE/ACM International Conference on Program Comprehension (Technical Track, ICPC '18) in Gothenburg, Sweden. Preprint of the corresponding paper available at http://www.ifi.uzh.ch/seal/people/vassallo/VassalloICPC18.pdf
Carmine Vassallo
May 27, 2018
Transcript
35 slides
-
Un-Break My Build: Assisting Developers with Build Repair Hints. Carmine
Vassallo, Sebastian Proksch, Timothy Zemp, and Harald C. Gall. @ccvassallo IEEE/ACM International Conference on Program Comprehension (Technical Track), Gothenburg, Sweden 2018 -
3 Developer Commit Build Pass? Developer Repository Build Server Yes
No Commit Build Success Build Failure Poll Continuous Integration (CI) -
Early Error Detection in CI: Build Failures 4 Developer Commit
Build Pass? Developer Repository Build Server Yes No Commit Build Success Build Failure Read Poll [INFO] Scanning for projects... [INFO] Inspecting build with total of 1 modules... [ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch. Build Log -
6 Google Guava Build Server: TravisCI Build Tool: Maven 40431
lines (Job #3.1) https://travis-ci.org/google/guava/jobs/40227447 -
13 Barriers when Solving Build Failures Hilton et al., “Trade-offs
in continuous integration: assurance, security, and flexibility” FSE 2017 Build Logs are quite large in size (100k lines) Build logs lack of important informa;on (e.g., full test report) -
14 Build Un-Breaking takes long time Kerzazi et al., “Why
do Automated Builds Break? An Empirical Study” ICSME 2014 On average, a build break takes 1 hour to be fixed Significant amount of working hours spent on build fixing -
20 Automated Build Fix (Macho et al., SANER 2018) Developer-Oriented
Assistance (BART) -
21 Developer Commit Build Pass? Developer Repository Build Server Yes
No Commit [INFO] Scanning for projects... [INFO] Inspecting build with total of 1 modules... [ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch. Build Success Build Failure Build log Poll Read Data Extraction Extract Generate External Resources Hint Build Summary BART (Build Abstraction and Recovery Tool) Meta-Model Data Enrichment -
23 From BIG To small Build Summary Build Status Failed
Failed Goal org.apache.maven.plugins:maven-compiler-plugin:2.1:compile Error Cause Compilation Failure Reactor ActiveJpa SUCCESS test SUCCESS core FAILURE utils SKIPPED Reason for Build Failure: Compilation Failure Hint: Compilation Your build contains a compilation error. Please check the following file: File: Model.java Line: 224 Reason: Model is abstract; cannot be instantiated Done Build Server Build Passed? Repository Build Log Notification Hint Generators 1 2 3 4 [INFO] Scanning for projects... [INFO] Inspecting build with total of 1 modules... [ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch. 5 7 Developer Compilation Testing Code Analysis Dependencies 6 BART 8 StackOverflow Discussions 9 14 lines ≈ 40k lines -
24 From BIG To small Build Summary Build Status Failed
Failed Goal org.apache.maven.plugins:maven-compiler-plugin:2.1:compile Error Cause Compilation Failure Reactor ActiveJpa SUCCESS test SUCCESS core FAILURE utils SKIPPED Reason for Build Failure: Compilation Failure Hint: Compilation Your build contains a compilation error. Please check the following file: File: Model.java Line: 224 Reason: Model is abstract; cannot be instantiated Done Build Server Build Passed? Repository Build Log Notification Hint Generators 1 2 3 4 [INFO] Scanning for projects... [INFO] Inspecting build with total of 1 modules... [ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch. 5 7 Developer Compilation Testing Code Analysis Dependencies 6 BART 8 StackOverflow Discussions 9 14 lines ≈ 40k lines -
Hint Generators Hints for the most 4 recurrent build- failure
types (Vassallo et al., ICSME 2017) • Compilation • Dependencies • Code Analysis • Testing External Resource: StackOverflow 25 Build Summary Build Status Failed Failed Goal org.apache.maven.plugins:maven-compiler-plugin:2.1:compile Error Cause Compilation Failure Reactor ActiveJpa SUCCESS test SUCCESS core FAILURE utils SKIPPED Reason for Build Failure: Compilation Failure Hint: Compilation Your build contains a compilation error. Please check the following file: File: Model.java Line: 224 Reason: Model is abstract; cannot be instantiated Done Build Server Build Passed? Repository Build Log Notification Hint Generators 1 2 3 4 [INFO] Scanning for projects... [INFO] Inspecting build with total of 1 modules... [ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch. 5 7 Developer Compilation Testing Code Analysis Dependencies 6 BART 8 StackOverflow Discussions 9 This is normal behaviour. Abstract classes are not supposed to be instantiated. You should test the classes which inherit from the abstract class, not the abstract class itself. Full Discussion: https://stackoverflow.com/questions/5028082/rails-3-activerecord- abstract-objects StackOverflow Analysis Related Post 1: -
Empirical Study RQ1 Are summarised logs more understandable? RQ2 Does
a semi-automated support system influence the time that is required to fix a broken build? 26 -
Study Context 3 Open-source systems • ActiveJPA (39,335 LOC, 143
GitHub stars, 123 builds) • Sentry-Java (113,332 LOC, 312 GitHub stars, 509 builds) • Fongo (31,088 LOC, 374 GitHub stars, 404 builds) 8 Participants 27 -
Study Procedure Bugs injection to provoke Compilation, Dependencies, Code Analysis,
and Testing build failures. Task 1: Build Summaries (Hints) VS Logs • Understandability • Relevance and Applicability of Proposed Solutions Task 2: Developer Performance With and Without BART • Build-Fix Time 28 -
Hint Understandability 29 Very Low Below Average Average Above Average
Very High 0% 0% 0% 0% 100% 100% 100% 88% 0% 0% 0% 12% 100 50 0 50 100 Percentage Response VL BA A AA VH Code Analysis Testing Compilation Dependencies “A structured summary is way easier to grasp than many unstructured lines of text” (S4) -
30 Very Low Below Average Average Above Average Very High
25% 0% 12% 75% 50% 75% 75% 0% 100 50 0 50 100 Percentage Response VL BA A AA VH 12% 25% 25% 25% Code Analysis Compilation Dependencies Testing 38% 12% 25% 75% 50% 75% 75% 0% 100 50 0 50 100 Percentage 25% 0% 12% 12% Code Analysis Compilation Dependencies Testing Solution Hints Relevance Applicability -
32 Build-Fix Time with BART % -27 in case of
Compilation % -33 in case of Code Analysis % -42 in case of Testing % -62 in case of Dependencies -
33 Average Build-Fix Time with BART % -41 “Less searching
for relevant part, hence fast bug resolution” (S5) “Maven logs tend to be verbose, having a summary greatly reduces the time needed to find and correct a build failure” (S5) -
35 Un-Break My Build: Assisting Developers with Build Repair Hints.
Carmine Vassallo, Sebastian Proksch, Timothy Zemp, and Harald C. Gall. @ccvassallo [email protected] X Developer Commit Build Pass? Developer Repository Build Server Yes No Commit [INFO] Scanning for projects... [INFO] Inspecting build with total of 1 modules... [ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch. Build Success Build Failure Build log Poll Read Data Extraction Extract Generate External Resources Hint Build Summary BART (Build Abstraction and Recovery Tool) Meta-Model Data Enrichment Hint Understandability X Very Low Below Average Average Above Average Very High 0% 0% 0% 0% 100% 100% 100% 88% 0% 0% 0% 12% 100 50 0 50 100 Percentage Response VL BA A AA VH Code Analysis Testing Compilation Dependencies “A structured summary is way easier to grasp than many unstructured lines of text” (S4) X Average Build-Fix Time with BART % -41 “Less searching for relevant part, hence fast bug resolution” (S5) “Maven logs tend to be verbose, having a summary greatly reduces the time needed to find and correct a build failure” (S5)