Automated Reporting of Anti-patterns and Decay in Continuous Integration

Our study on detection of anti-patterns and decay in Continuous Integration presented at the International Conference on Software Engineering (Technical Track, ICSE '19) in Montréal, Canada. The preprint of the corresponding paper available at http://doi.org/10.5281/zenodo.2578271

Transcript

1. Automated Reporting of Anti-patterns and Decay in Continuous Integration Carmine Vassallo, Sebastian Proksch, Harald C. Gall, and Massimiliano Di Penta. @ccvassallo International Conference on Software Engineering (ICSE), Montréal, Canada, 2019 !1
2. Continuous Integration (CI) !2 Commit Poll Revisions Build Logs Build Repository Build Server Release Candidate CI History
3. Benefits of Using CI !3 Faster Releases (Hilton et al., 2016) Increased Productivity (Vasilescu et al., 2015) Improved Reliability (Vasilescu et al., 2015)
4. Principles !4 Duvall et al., 2007
5. !5 Anti-patterns Duvall et al., 2010 Principles Duvall et al., 2007
6. !6 CI Principle: Merge Frequently master
7. !7 CI Principle: Merge Frequently featureX master featureY
8. !8 CI Principle: Merge Frequently featureX master 2 hours 3.5 hours featureY
9. !9 CI Principle: Merge Frequently featureX master 2 hours 3.5 hours featureY
10. !10 CI Principle: Merge Frequently featureX master 2 hours 3.5 hours featureY
11. !11 CI Anti-pattern: Late Merging featureX master featureY
12. !12 CI Anti-pattern: Late Merging featureX master 5 days 2 weeks featureY
13. !13 CI Anti-pattern: Late Merging featureX master 5 days 2 weeks featureY
14. !14 CI Anti-pattern: Late Merging featureX master 5 days 2 weeks CI Decay featureY
15. Goals of our study !15 Investigate the existence of CI Decay Provide awareness of CI decay caused by CI anti-patterns
16. The Problem of CI Decay !16
17. Selection and Validation !17 Duvall et al., 2010
18. !18 3 authors Duvall et al., 2010 Internal Selection Anti-pattern candidates Pre-selection Selection and Validation
19. Selection and Validation !19 124 participants Feedback on Detection Strategies 3 authors Duvall et al., 2010 Internal Selection Survey Anti-pattern candidates Relevant Anti-patterns CI decay and anti-patterns Pre-selection Validation
20. !20 Late Merging Skip Failed Tests Slow Builds Broken Release Branch Relevant CI Anti-patterns
21. CI Decay and Lack of Awareness The existence of anti-patterns provoke CI Decay • 77% agree that CI benefits diminish in case anti-patterns exist Developers are not aware of anti-patterns • 77% deviate unintentionally from CI principles !21
22. !22 Detection of Anti-patterns in CI
23. CI-Odor: Detection of Anti-patterns in CI !23 Commit Poll Revisions Build Logs
24. CI-Odor: Detection of Anti-patterns in CI !24 Commit Poll Revisions Build Logs # Build Failures # Skipped Tests Compute CI Metrics …
25. CI-Odor: Detection of Anti-patterns in CI !25 Commit Poll Revisions Build Logs # Build Failures # Skipped Tests … Compute CI Metrics Detectors Generate CI Reports
26. Late Merging !26 featureX featureY master Time
27. Late Merging !27 featureX featureY master Time
28. Late Merging !28 featureX featureY master Time Last Sync With Master Last Commit Unsynced Activity
29. Late Merging !29 featureX featureY master Time Last Sync With Master Last Commit Unsynced Activity In your project, branches are typically synced with master every 2.8 days. However, branch featureY was last synced with master on Nov 23rd at 10:25 and branch master has commits that are 19 days newer that that. High
30. Late Merging !30 featureX featureY master Time Last Commit on Other Branches Last Commit Branch Deviation
31. Late Merging !31 featureX featureY master Time Last Commit on Other Branches Last Commit You latest commits were performed on branch featureX. While you typically merge branches within 1.8 days in your project, branch featureY was last changed 3.7 days ago and has not been merged into featureX yet. Medium Branch Deviation
32. Late Merging !32 featureX featureY master Time First Commit Last Commit Total Activity
33. Late Merging !33 featureX featureY master Time First Commit Last Commit Total Activity Your feature branches are typically open for 2.3 weeks. However, you have been working on featureX for 7.3 weeks now. High
34. Slow Builds !34 Average Build Duration per Week Recent Slow Builds
35. Skip Failed Tests !35 36 Time 37 Test #3 Test #1 Test #2 Test #3 Test #1
36. Skip Failed Tests !36 Affected Builds (and Test Classes) # Incidents per Month
37. !37 # Incidents per Week Average Resolution Time Broken Release Branch
38. Usefulness of CI-Odor !38
39. Evaluation !39 Generate 13 Original Developers Qualitative Analysis (Survey) CI Odor CI-Reports 29 Other Developers Quantitative Analysis Data Collection 36 Projects Compute
40. Qualitative Insights on CI-Odor CI Reports are useful and CI anti-patterns are properly identified Developers expect a positive effect of using CI Reports on their CI discipline • 67% expect an improvement of their CI practice Developers would like to use CI Reports in their teams • 55% are willing to integrate CI-Odor !40
41. Qualitative Insights on CI-Odor CI Reports should suggest how to fix anti-patterns • In case of Slow Builds developers do not know how to react to the warnings Need for configurability of CI-Odor !41
42. !42 Late Merging Skip Failed Tests Slow Builds Broken Release Branch Quantitative Analysis of CI Decay 97% of projects 67% of branches 42% of projects 0.65% of builds 55% of projects 27% of builds 100% of projects 12% of builds
43. !43 Increase awareness to avoid CI decay Facilitate the learning of CI principles Support more project-specific policies
44. 44 Automated Reporting of Anti-patterns and Decay in Continuous Integration. Carmine Vassallo, Sebastian Proksch, Harald C. Gall, and Massimiliano Di Penta. @ccvassallo vassallo@ifi.uzh.ch Selection and Validation !X 124 participants Feedback on Detection Strategies 3 authors Duvall et al., 2010 Internal Selection Survey Anti-pattern candidates Relevant Anti-patterns CI decay and anti-patterns Pre-selection Validation Evaluation !X Generate 13 Original Developers Qualitative Analysis (Survey) CI Odor CI-Reports 29 Other Developers Quantitative Analysis Data Collection 36 Projects Compute CI-Odor: Detection of Anti-patterns in CI !X Commit Poll Revisions Build Logs # Build Failures # Skipped Tests … Compute CI Metrics Detectors Generate CI Reports
45. CI-Odor: Detection of Anti-patterns in CI !X Commit Poll Revisions Build Logs # Build Failures # Skipped Tests … Compute CI Metrics Detectors Generate CI Reports 45 Automated Reporting of Anti-patterns and Decay in Continuous Integration. Carmine Vassallo, Sebastian Proksch, Harald C. Gall, and Massimiliano Di Penta. @ccvassallo vassallo@ifi.uzh.ch https://bit.ly/2Me62P0 Selection and Validation !X 124 participants Feedback on Detection Strategies 3 authors Duvall et al., 2010 Internal Selection Survey Anti-pattern candidates Relevant Anti-patterns CI decay and anti-patterns Pre-selection Validation Evaluation !X Generate 13 Original Developers Qualitative Analysis (Survey) CI Odor CI-Reports 29 Other Developers Quantitative Analysis Data Collection 36 Projects Compute