Save 37% off PRO during our Black Friday Sale! »

Automated Reporting of Anti-patterns and Decay in Continuous Integration

Automated Reporting of Anti-patterns and Decay in Continuous Integration

Our study on detection of anti-patterns and decay in Continuous Integration presented at the International Conference on Software Engineering (Technical Track, ICSE '19) in Montréal, Canada. The preprint of the corresponding paper available at http://doi.org/10.5281/zenodo.2578271

84581630350da72785f7d2448549ca06?s=128

Carmine Vassallo

May 29, 2019
Tweet

Transcript

  1. Automated Reporting of Anti-patterns and Decay in Continuous Integration Carmine

    Vassallo, Sebastian Proksch, Harald C. Gall, and Massimiliano Di Penta. @ccvassallo International Conference on Software Engineering (ICSE), Montréal, Canada, 2019 !1
  2. Continuous Integration (CI) !2 Commit Poll Revisions Build Logs Build

    Repository Build Server Release Candidate CI History
  3. Benefits of Using CI !3 Faster Releases (Hilton et al.,

    2016) Increased Productivity (Vasilescu et al., 2015) Improved Reliability (Vasilescu et al., 2015)
  4. Principles !4 Duvall et al., 2007

  5. !5 Anti-patterns Duvall et al., 2010 Principles Duvall et al.,

    2007
  6. !6 CI Principle: Merge Frequently master

  7. !7 CI Principle: Merge Frequently featureX master featureY

  8. !8 CI Principle: Merge Frequently featureX master 2 hours 3.5

    hours featureY
  9. !9 CI Principle: Merge Frequently featureX master 2 hours 3.5

    hours featureY
  10. !10 CI Principle: Merge Frequently featureX master 2 hours 3.5

    hours featureY
  11. !11 CI Anti-pattern: Late Merging featureX master featureY

  12. !12 CI Anti-pattern: Late Merging featureX master 5 days 2

    weeks featureY
  13. !13 CI Anti-pattern: Late Merging featureX master 5 days 2

    weeks featureY
  14. !14 CI Anti-pattern: Late Merging featureX master 5 days 2

    weeks CI Decay featureY
  15. Goals of our study !15 Investigate the existence of CI

    Decay Provide awareness of CI decay caused by CI anti-patterns
  16. The Problem of CI Decay !16

  17. Selection and Validation !17 Duvall et al., 2010

  18. !18 3 authors Duvall et al., 2010 Internal Selection Anti-pattern

    candidates Pre-selection Selection and Validation
  19. Selection and Validation !19 124 participants Feedback on Detection Strategies

    3 authors Duvall et al., 2010 Internal Selection Survey Anti-pattern candidates Relevant Anti-patterns CI decay and anti-patterns Pre-selection Validation
  20. !20 Late Merging Skip Failed Tests Slow Builds Broken Release

    Branch Relevant CI Anti-patterns
  21. CI Decay and Lack of Awareness The existence of anti-patterns

    provoke CI Decay • 77% agree that CI benefits diminish in case anti-patterns exist Developers are not aware of anti-patterns • 77% deviate unintentionally from CI principles !21
  22. !22 Detection of Anti-patterns in CI

  23. CI-Odor: Detection of Anti-patterns in CI !23 Commit Poll Revisions

    Build Logs
  24. CI-Odor: Detection of Anti-patterns in CI !24 Commit Poll Revisions

    Build Logs # Build Failures # Skipped Tests Compute CI Metrics …
  25. CI-Odor: Detection of Anti-patterns in CI !25 Commit Poll Revisions

    Build Logs # Build Failures # Skipped Tests … Compute CI Metrics Detectors Generate CI Reports
  26. Late Merging !26 featureX featureY master Time

  27. Late Merging !27 featureX featureY master Time

  28. Late Merging !28 featureX featureY master Time Last Sync With

    Master Last Commit Unsynced Activity
  29. Late Merging !29 featureX featureY master Time Last Sync With

    Master Last Commit Unsynced Activity In your project, branches are typically synced with master every 2.8 days. However, branch featureY was last synced with master on Nov 23rd at 10:25 and branch master has commits that are 19 days newer that that. High
  30. Late Merging !30 featureX featureY master Time Last Commit on

    Other Branches Last Commit Branch Deviation
  31. Late Merging !31 featureX featureY master Time Last Commit on

    Other Branches Last Commit You latest commits were performed on branch featureX. While you typically merge branches within 1.8 days in your project, branch featureY was last changed 3.7 days ago and has not been merged into featureX yet. Medium Branch Deviation
  32. Late Merging !32 featureX featureY master Time First Commit Last

    Commit Total Activity
  33. Late Merging !33 featureX featureY master Time First Commit Last

    Commit Total Activity Your feature branches are typically open for 2.3 weeks. However, you have been working on featureX for 7.3 weeks now. High
  34. Slow Builds !34 Average Build Duration per Week Recent Slow

    Builds
  35. Skip Failed Tests !35 36 Time 37 Test #3 Test

    #1 Test #2 Test #3 Test #1
  36. Skip Failed Tests !36 Affected Builds (and Test Classes) #

    Incidents per Month
  37. !37 # Incidents per Week Average Resolution Time Broken Release

    Branch
  38. Usefulness of CI-Odor !38

  39. Evaluation !39 Generate 13 Original Developers Qualitative Analysis (Survey) CI

    Odor CI-Reports 29 Other Developers Quantitative Analysis Data Collection 36 Projects Compute
  40. Qualitative Insights on CI-Odor CI Reports are useful and CI

    anti-patterns are properly identified Developers expect a positive effect of using CI Reports on their CI discipline • 67% expect an improvement of their CI practice Developers would like to use CI Reports in their teams • 55% are willing to integrate CI-Odor !40
  41. Qualitative Insights on CI-Odor CI Reports should suggest how to

    fix anti-patterns • In case of Slow Builds developers do not know how to react to the warnings Need for configurability of CI-Odor !41
  42. !42 Late Merging Skip Failed Tests Slow Builds Broken Release

    Branch Quantitative Analysis of CI Decay 97% of projects 67% of branches 42% of projects 0.65% of builds 55% of projects 27% of builds 100% of projects 12% of builds
  43. !43 Increase awareness to avoid CI decay Facilitate the learning

    of CI principles Support more project-specific policies
  44. 44 Automated Reporting of Anti-patterns and Decay in Continuous Integration.

    Carmine Vassallo, Sebastian Proksch, Harald C. Gall, and Massimiliano Di Penta. @ccvassallo vassallo@ifi.uzh.ch Selection and Validation !X 124 participants Feedback on Detection Strategies 3 authors Duvall et al., 2010 Internal Selection Survey Anti-pattern candidates Relevant Anti-patterns CI decay and anti-patterns Pre-selection Validation Evaluation !X Generate 13 Original Developers Qualitative Analysis (Survey) CI Odor CI-Reports 29 Other Developers Quantitative Analysis Data Collection 36 Projects Compute CI-Odor: Detection of Anti-patterns in CI !X Commit Poll Revisions Build Logs # Build Failures # Skipped Tests … Compute CI Metrics Detectors Generate CI Reports
  45. CI-Odor: Detection of Anti-patterns in CI !X Commit Poll Revisions

    Build Logs # Build Failures # Skipped Tests … Compute CI Metrics Detectors Generate CI Reports 45 Automated Reporting of Anti-patterns and Decay in Continuous Integration. Carmine Vassallo, Sebastian Proksch, Harald C. Gall, and Massimiliano Di Penta. @ccvassallo vassallo@ifi.uzh.ch https://bit.ly/2Me62P0 Selection and Validation !X 124 participants Feedback on Detection Strategies 3 authors Duvall et al., 2010 Internal Selection Survey Anti-pattern candidates Relevant Anti-patterns CI decay and anti-patterns Pre-selection Validation Evaluation !X Generate 13 Original Developers Qualitative Analysis (Survey) CI Odor CI-Reports 29 Other Developers Quantitative Analysis Data Collection 36 Projects Compute