Automated Reporting of Anti-patterns and Decay in Continuous Integration

Our study on detection of anti-patterns and decay in Continuous Integration presented at the International Conference on Software Engineering (Technical Track, ICSE '19) in Montréal, Canada. The preprint of the corresponding paper available at http://doi.org/10.5281/zenodo.2578271

Transcript

1. Automated Reporting of Anti-patterns and Decay in
   Continuous Integration
   Carmine Vassallo, Sebastian Proksch, Harald C. Gall, and Massimiliano Di Penta.
   @ccvassallo
   International Conference on Software Engineering (ICSE), Montréal, Canada, 2019
   !1
   

   View Slide

2. Continuous Integration (CI)
   !2
   Commit Poll
   Revisions Build Logs
   Build
   Repository Build Server Release Candidate
   CI History
   

   View Slide

3. Benefits of Using CI
   !3
   Faster Releases
   (Hilton et al., 2016)
   Increased Productivity
   (Vasilescu et al., 2015)
   Improved Reliability
   (Vasilescu et al., 2015)
   

   View Slide

4. Principles
   !4
   Duvall et al., 2007
   

   View Slide

5. !5
   Anti-patterns
   Duvall et al., 2010
   Principles
   Duvall et al., 2007
   

   View Slide

6. !6
   CI Principle: Merge Frequently
   master
   

   View Slide

7. !7
   CI Principle: Merge Frequently
   featureX
   master
   featureY
   

   View Slide

8. !8
   CI Principle: Merge Frequently
   featureX
   master
   2 hours
   3.5 hours
   featureY
   

   View Slide

9. !9
   CI Principle: Merge Frequently
   featureX
   master
   2 hours
   3.5 hours
   featureY
   

   View Slide

10. !10
    CI Principle: Merge Frequently
    featureX
    master
    2 hours
    3.5 hours
    featureY
    

    View Slide

11. !11
    CI Anti-pattern: Late Merging
    featureX
    master
    featureY
    

    View Slide

12. !12
    CI Anti-pattern: Late Merging
    featureX
    master
    5 days
    2 weeks
    featureY
    

    View Slide

13. !13
    CI Anti-pattern: Late Merging
    featureX
    master
    5 days
    2 weeks
    featureY
    

    View Slide

14. !14
    CI Anti-pattern: Late Merging
    featureX
    master
    5 days
    2 weeks
    CI Decay
    featureY
    

    View Slide

15. Goals of our study
    !15
    Investigate the existence of CI Decay
    Provide awareness of CI decay caused by CI anti-patterns
    

    View Slide

16. The Problem of CI Decay
    !16
    

    View Slide

17. Selection and Validation
    !17
    Duvall et al.,
    2010
    

    View Slide

18. !18
    3 authors
    Duvall et al.,
    2010
    Internal Selection
    Anti-pattern
    candidates
    Pre-selection
    Selection and Validation
    

    View Slide

19. Selection and Validation
    !19
    124
    participants
    Feedback on
    Detection
    Strategies
    3 authors
    Duvall et al.,
    2010
    Internal Selection Survey
    Anti-pattern
    candidates
    Relevant
    Anti-patterns
    CI decay and
    anti-patterns
    Pre-selection Validation
    

    View Slide

20. !20
    Late Merging Skip Failed Tests Slow Builds
    Broken Release
    Branch
    Relevant CI Anti-patterns
    

    View Slide

21. CI Decay and Lack of Awareness
    The existence of anti-patterns provoke CI Decay
    • 77% agree that CI benefits diminish in case anti-patterns exist
    Developers are not aware of anti-patterns
    • 77% deviate unintentionally from CI principles
    !21
    

    View Slide

22. !22
    Detection of Anti-patterns in CI
    

    View Slide

23. CI-Odor: Detection of Anti-patterns in CI
    !23
    Commit Poll
    Revisions Build Logs
    

    View Slide

24. CI-Odor: Detection of Anti-patterns in CI
    !24
    Commit Poll
    Revisions Build Logs
    # Build Failures
    # Skipped Tests
    Compute
    CI Metrics
    …
    

    View Slide

25. CI-Odor: Detection of Anti-patterns in CI
    !25
    Commit Poll
    Revisions Build Logs
    # Build Failures
    # Skipped Tests
    …
    Compute
    CI Metrics
    Detectors
    Generate
    CI Reports
    

    View Slide

26. Late Merging
    !26
    featureX
    featureY
    master
    Time
    

    View Slide

27. Late Merging
    !27
    featureX
    featureY
    master
    Time
    

    View Slide

28. Late Merging
    !28
    featureX
    featureY
    master
    Time
    Last Sync
    With Master
    Last
    Commit
    Unsynced Activity
    

    View Slide

29. Late Merging
    !29
    featureX
    featureY
    master
    Time
    Last Sync
    With Master
    Last
    Commit
    Unsynced Activity
    In your project, branches are typically synced with master
    every 2.8 days.
    However, branch featureY was last synced with master on Nov 23rd at
    10:25 and branch master has commits that are 19 days newer that that.
    High
    

    View Slide

30. Late Merging
    !30
    featureX
    featureY
    master
    Time
    Last Commit on
    Other Branches
    Last
    Commit
    Branch Deviation
    

    View Slide

31. Late Merging
    !31
    featureX
    featureY
    master
    Time
    Last Commit on
    Other Branches
    Last
    Commit
    You latest commits were performed on branch featureX. While
    you typically merge branches within 1.8 days in your project, branch
    featureY was last changed 3.7 days ago and has not been merged into
    featureX yet.
    Medium
    Branch Deviation
    

    View Slide

32. Late Merging
    !32
    featureX
    featureY
    master
    Time
    First Commit
    Last
    Commit
    Total Activity
    

    View Slide

33. Late Merging
    !33
    featureX
    featureY
    master
    Time
    First Commit
    Last
    Commit
    Total Activity
    Your feature branches are typically open for 2.3 weeks.
    However, you have been working on featureX for 7.3 weeks now.
    High
    

    View Slide

34. Slow Builds
    !34
    Average Build Duration
    per Week
    Recent Slow Builds
    

    View Slide

35. Skip Failed Tests
    !35
    36
    Time
    37
    Test #3
    Test #1
    Test #2 Test #3
    Test #1
    

    View Slide

36. Skip Failed Tests
    !36
    Affected Builds
    (and Test Classes)
    # Incidents per
    Month
    

    View Slide

37. !37
    # Incidents per Week
    Average Resolution
    Time
    Broken Release Branch
    

    View Slide

38. Usefulness of CI-Odor
    !38
    

    View Slide

39. Evaluation
    !39
    Generate
    13 Original
    Developers
    Qualitative Analysis (Survey)
    CI Odor
    CI-Reports
    29 Other
    Developers
    Quantitative Analysis
    Data Collection
    36 Projects
    Compute
    

    View Slide

40. Qualitative Insights on CI-Odor
    CI Reports are useful and CI anti-patterns are properly identified
    Developers expect a positive effect of using CI Reports on their CI discipline
    • 67% expect an improvement of their CI practice
    Developers would like to use CI Reports in their teams
    • 55% are willing to integrate CI-Odor
    !40
    

    View Slide

41. Qualitative Insights on CI-Odor
    CI Reports should suggest how to fix anti-patterns
    • In case of Slow Builds developers do not know how to react to the
    warnings
    Need for configurability of CI-Odor
    !41
    

    View Slide

42. !42
    Late Merging Skip Failed Tests Slow Builds
    Broken Release
    Branch
    Quantitative Analysis of CI Decay
    97% of projects
    67% of branches
    42% of projects
    0.65% of builds
    55% of projects
    27% of builds
    100% of projects
    12% of builds
    

    View Slide

43. !43
    Increase awareness to avoid CI decay
    Facilitate the learning of CI principles
    Support more project-specific policies
    

    View Slide

44. 44
    Automated Reporting of Anti-patterns and Decay
    in Continuous Integration.
    Carmine Vassallo, Sebastian Proksch, Harald C. Gall,
    and Massimiliano Di Penta.
    @ccvassallo
    vassallo@ifi.uzh.ch
    Selection and Validation
    !X
    124
    participants
    Feedback on
    Detection
    Strategies
    3 authors
    Duvall et al.,
    2010
    Internal Selection Survey
    Anti-pattern
    candidates
    Relevant
    Anti-patterns
    CI decay and
    anti-patterns
    Pre-selection Validation
    Evaluation
    !X
    Generate
    13 Original
    Developers
    Qualitative Analysis (Survey)
    CI Odor
    CI-Reports
    29 Other
    Developers
    Quantitative Analysis
    Data Collection
    36 Projects
    Compute
    CI-Odor: Detection of Anti-patterns in CI
    !X
    Commit Poll
    Revisions Build Logs
    # Build Failures
    # Skipped Tests
    …
    Compute
    CI Metrics
    Detectors
    Generate
    CI Reports
    

    View Slide

45. CI-Odor: Detection of Anti-patterns in CI
    !X
    Commit Poll
    Revisions Build Logs
    # Build Failures
    # Skipped Tests
    …
    Compute
    CI Metrics
    Detectors
    Generate
    CI Reports
    45
    Automated Reporting of Anti-patterns and Decay
    in Continuous Integration.
    Carmine Vassallo, Sebastian Proksch, Harald C. Gall,
    and Massimiliano Di Penta.
    @ccvassallo
    vassallo@ifi.uzh.ch
    https://bit.ly/2Me62P0
    Selection and Validation
    !X
    124
    participants
    Feedback on
    Detection
    Strategies
    3 authors
    Duvall et al.,
    2010
    Internal Selection Survey
    Anti-pattern
    candidates
    Relevant
    Anti-patterns
    CI decay and
    anti-patterns
    Pre-selection Validation
    Evaluation
    !X
    Generate
    13 Original
    Developers
    Qualitative Analysis (Survey)
    CI Odor
    CI-Reports
    29 Other
    Developers
    Quantitative Analysis
    Data Collection
    36 Projects
    Compute
    

    View Slide