Developer-Oriented Assistance for Build Failure Resolution
Slides on Developer-Oriented Assistance for Build Failure Resolution discussed during In-Vivo Analytics for Big Software Quality at Lorentz Center in Leiden (The Netherlands).
Developer-Oriented Assistance for Build Failure Resolution Carmine Vassallo @ccvassallo In-Vivo Analytics for Big Software Quality, Sep. 17-21 2018, Lorentz Center, Leiden (The Netherlands)
2 CI Analytics CI My Research Overview CI Failures Debugging CI Configuration in review ;) A Tale of CI Build Failures: an Open Source and a Financial Organisation Perspective. (ICSME ‘17) Context Is King: The Developer Perspective on the Usage of Static Analysis Tools. (SANER ‘18) Un-Break My Build: Assisting Developers with Build Repair Hints.(ICPC ‘18) Continuous Code Quality: Are We (Really) Doing That? (ASE ‘18) Continuous Refactoring in CI: A Preliminary Study On the Perceived Advantages and Barriers. (ICSME ‘18)
Early Error Detection: Build Failures 5 Developer Commit Build Pass? Developer Repository Build Server Yes No Commit Build Success Build Failure Read Poll [INFO] Scanning for projects... [INFO] Inspecting build with total of 1 modules... [ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch. Build Log
14 Barriers when Solving Build Failures Hilton et al., “Trade-offs in continuous integration: assurance, security, and flexibility” FSE 2017 Build Logs are quite large in size (100k lines) Build logs lack of important information (e.g., full test report)
15 Build Un-Breaking takes long time Kerzazi et al., “Why do Automated Builds Break? An Empirical Study” ICSME 2014 On average, a build break takes 1 hour to be fixed Significant amount of working hours spent on build fixing
22 Developer Commit Build Pass? Developer Repository Build Server Yes No Commit [INFO] Scanning for projects... [INFO] Inspecting build with total of 1 modules... [ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch. Build Success Build Failure Build log Poll Read Data Extraction Extract Generate External Resources Hint Build Summary BART (Build Abstraction and Recovery Tool) Meta-Model Data Enrichment
24 From BIG To small Build Summary Build Status Failed Failed Goal org.apache.maven.plugins:maven-compiler-plugin:2.1:compile Error Cause Compilation Failure Reactor ActiveJpa SUCCESS test SUCCESS core FAILURE utils SKIPPED Reason for Build Failure: Compilation Failure Hint: Compilation Your build contains a compilation error. Please check the following file: File: Model.java Line: 224 Reason: Model is abstract; cannot be instantiated Done Build Server Build Passed? Repository Build Log Notification Hint Generators 1 2 3 4 [INFO] Scanning for projects... [INFO] Inspecting build with total of 1 modules... [ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch. 5 7 Developer Compilation Testing Code Analysis Dependencies 6 BART 8 StackOverflow Discussions 9 14 lines ≈ 40k lines
25 From BIG To small Build Summary Build Status Failed Failed Goal org.apache.maven.plugins:maven-compiler-plugin:2.1:compile Error Cause Compilation Failure Reactor ActiveJpa SUCCESS test SUCCESS core FAILURE utils SKIPPED Reason for Build Failure: Compilation Failure Hint: Compilation Your build contains a compilation error. Please check the following file: File: Model.java Line: 224 Reason: Model is abstract; cannot be instantiated Done Build Server Build Passed? Repository Build Log Notification Hint Generators 1 2 3 4 [INFO] Scanning for projects... [INFO] Inspecting build with total of 1 modules... [ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch. 5 7 Developer Compilation Testing Code Analysis Dependencies 6 BART 8 StackOverflow Discussions 9 14 lines ≈ 40k lines
Hint Generators Hints for the most 4 recurrent build- failure types (Vassallo et al., ICSME 2017) • Compilation • Dependencies • Code Analysis • Testing External Resource: StackOverflow 26 Build Summary Build Status Failed Failed Goal org.apache.maven.plugins:maven-compiler-plugin:2.1:compile Error Cause Compilation Failure Reactor ActiveJpa SUCCESS test SUCCESS core FAILURE utils SKIPPED Reason for Build Failure: Compilation Failure Hint: Compilation Your build contains a compilation error. Please check the following file: File: Model.java Line: 224 Reason: Model is abstract; cannot be instantiated Done Build Server Build Passed? Repository Build Log Notification Hint Generators 1 2 3 4 [INFO] Scanning for projects... [INFO] Inspecting build with total of 1 modules... [ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch. 5 7 Developer Compilation Testing Code Analysis Dependencies 6 BART 8 StackOverflow Discussions 9 This is normal behaviour. Abstract classes are not supposed to be instantiated. You should test the classes which inherit from the abstract class, not the abstract class itself. Full Discussion: https://stackoverflow.com/questions/5028082/rails-3-activerecord- abstract-objects StackOverflow Analysis Related Post 1:
Empirical Study RQ1 Are summarised logs more understandable? RQ2 Does a semi-automated support system influence the time that is required to fix a broken build? 27
Study Procedure Bugs injection to provoke Compilation, Dependencies, Code Analysis, and Testing build failures. Task 1: Build Summaries (Hints) VS Logs • Understandability • Relevance and Applicability of Proposed Solutions Task 2: Developer Performance With and Without BART • Build-Fix Time 29
Hint Understandability 30 Very Low Below Average Average Above Average Very High 0% 0% 0% 0% 100% 100% 100% 88% 0% 0% 0% 12% 100 50 0 50 100 Percentage Response VL BA A AA VH Code Analysis Testing Compilation Dependencies “A structured summary is way easier to grasp than
34 Average Build-Fix Time with BART % -41 “Less searching for relevant part, hence fast bug resolution” (S5) “Maven logs tend to be verbose, having a summary greatly reduces the time needed to find and correct a build failure” (S5)
36 Developer-oriented Assistance for Build Failure Resolution. Carmine Vassallo. @ccvassallo [email protected] X Developer Commit Build Pass? Developer Repository Build Server Yes No Commit [INFO] Scanning for projects... [INFO] Inspecting build with total of 1 modules... [ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch. Build Success Build Failure Build log Poll Read Data Extraction Extract Generate External Resources Hint Build Summary BART (Build Abstraction and Recovery Tool) Meta-Model Data Enrichment Hint Understandability X Very Low Below Average Average Above Average Very High 0% 0% 0% 0% 100% 100% 100% 88% 0% 0% 0% 12% 100 50 0 50 100 Percentage Response VL BA A AA VH Code Analysis Testing Compilation Dependencies “A structured summary is way easier to grasp than
many unstructured lines of text” (S4) X CI Analytics CI My Research Overview CI Failures Debugging CI Configuration in review ;) A Tale of CI Build Failures: an Open Source and a Financial Organisation Perspective. (ICSME ‘17) Context Is King: The Developer Perspective on the Usage of Static Analysis Tools. (SANER ‘18) Un-Break My Build: Assisting Developers with Build Repair Hints.(ICPC ‘18) Continuous Code Quality: Are We (Really) Doing That? (ASE ‘18) Continuous Refactoring in CI: A Preliminary Study On the Perceived Advantages and Barriers. (ICSME ‘18)
carminevassallo
5g_digitalservicetmg
takuma_okamoto
yamato0811
hashi0203
arumakan
ykiyota
tomoshige_n
ariaki
hide2kano
icttitech
sorami
gamella
mclloyd
pauljervisheath
addyosmani
destraynor
sstephenson
lynnandtonic
trallard
dougneiner
nonsquared
cromwellryan
tammielis
rasmusluckow