Getting Continuous Testing Done Right with CD-Linter

An effective and efficient application of Continuous Integration (CI) and Delivery (CD) requires software projects to follow certain principles and good practices such as Continuous Testing. Configuring such a CI/CD pipeline is challenging and error-prone. Therefore, automated linters have been proposed to detect errors in the pipeline. While existing linters identify syntactic errors, detect security vulnerabilities or misuse of the features provided by build servers, they do not support developers that want to prevent common misconfigurations of a CD pipeline that potentially violate CD principles ("CD smells"). In this talk, I present CD-Linter, a semantic linter that can automatically identify four different smells in pipeline configuration files, and show how it can help to foster Continuous Testing. We have evaluated our linter through a large-scale and long-term study on GitLab that consists of (i) monitoring 145 issues (opened in as many open-source projects) over a period of 6 months, (ii) manually validating the detection precision and recall on a representative sample of issues, and (iii) assessing the magnitude of the observed smells on 5,312 open-source projects. Our results show that CD smells are accepted and fixed by most of the developers and our linter achieves a precision of 87% and a recall of 94%. Those smells can be frequently observed in the wild, as 31% of projects with long configurations are affected by at least one smell.

Transcript

1. Getting Continuous Testing Done Right with CD-Linter Carmine Vassallo University of Zurich @ccvassallo DevOps Institute, Continuous Testing SKILup Day, November 19, 2020 Image from https://unsplash.com/photos/vBvfXIqC4E4
2. @ccvassallo Who Am I My name is Carmine Vassallo Research intern in the Continuous Delivery team at ING Nederland (2015) PhD Graduate from the University of Zurich (2020), where I am currently a postdoctoral researcher My research goal is to facilitate the adoption of DevOps practices 2 I’m on the Job Market! http://tiny.uzh.ch/WV
3. @ccvassallo 3 @ccvassallo Continuous Testing is a foundation of Continuous Delivery (Humble et Farley, 2010)
4. @ccvassallo 4 Compilation Testing Quality Assurance Continuous Delivery (CD) Repository Commit (often) Build Server Poll Release Candidate Build stages: - compilation - testing - qa variables: POSTGRES_USR: user POSTGRES_PWD: password compile_production_code: stage: compile script: “mvn compile” when: manual allow_failure: false compile_test_code: stage: compilation script: “mvn test” retry: 3 … .gitlab-ci.yml Build pipeline Icons from https://vitalitychicago.com/blog/top-reasons-agile-didnt-work-for-us-1-we-couldnt-co-locate-teams/, https://www.flaticon.com/authors/roundicons, https://www.pinclipart.com/pindetail/hbTb_clipart-info-server-png-transparent-png/
5. @ccvassallo 5 Compilation Testing Quality Assurance Continuous Delivery (CD) Repository Commit (often) Build Server Poll Release Candidate Build stages: - compilation - testing - qa variables: POSTGRES_USR: user POSTGRES_PWD: password compile_production_code: stage: compile script: “mvn compile” when: manual allow_failure: false compile_test_code: stage: compilation script: “mvn test” retry: 3 … .gitlab-ci.yml Build pipeline Developers struggle configuring build pipelines (Hilton et al., 2017)
6. @ccvassallo Linters for CD Configurations 6 stages: - compilation - testing - qa variables: POSTGRES_USR: user POSTGRES_PWD: password compile_production_code: stage: compile script: “mvn compile” when: manual allow_failure: false compile_test_code: stage: compilation script: “mvn test” retry: 3 … CI Lint (GitLab) Syntax is incorrect: chosen stage does not exist. Hansel (Gallaba et al., 2018) CD feature is misused: command unrelated to the stage. SLIC (Rahman et al., 2019) Security smell: hard-coded secrets. .gitlab-ci.yml ?
7. @ccvassallo Linters for CD Configurations 7 stages: - compilation - testing - qa variables: POSTGRES_USR: user POSTGRES_PWD: password compile_production_code: stage: compile script: “mvn compile” when: manual allow_failure: false compile_test_code: stage: compilation script: “mvn test” retry: 3 … CI Lint (GitLab) Syntax is incorrect: chosen stage does not exist. Hansel (Gallaba et al., 2018) SLIC (Rahman et al., 2019) CD feature is misused: command unrelated to the stage. Security smell: hard-coded secrets. .gitlab-ci.yml ? Developers typically lack awareness of CD principle (e.g., Continuous Testing) violations that threaten expected benefits (Vassallo et al., 2019)
8. @ccvassallo CD-Linter: Detecting violations of CD principles 8 Fake Success Retry Failure Manual Execution Fuzzy Version Carmine Vassallo, Sebastian Proksch, Anna Jancso, Harald C. Gall, Massimiliano Di Penta. Configuration Smells in Continuous Delivery Pipelines: A Linter and A Six-Month Study on GitLab. In ESEC/FSE, 2020.
9. @ccvassallo Fake Success Fail the build in presence of defects Prevent job failures from failing the build 9 … unit_test: stage: testing script: “mvn test” allow_failure: false … … CD Smell: ‘unit_test’ job is not allowed to fail. CD-Linter .gitlab-ci.yml
10. @ccvassallo Retry Failure The build process has to be deterministic Hiding flakiness by rerunning a job multiple times after failures. 10 … unit_test: stage: testing script: “mvn test” retry: 3 … … CD Smell: ‘unit_test’ job is retried after failures. CD-Linter .gitlab-ci.yml
11. @ccvassallo Manual Execution The pipeline has to be fully automated Some jobs are triggered manually 11 … unit_test: stage: testing script: “mvn test” when: manual … … CD Smell: ‘unit_test’ job is executed manually. CD-Linter .gitlab-ci.yml
12. @ccvassallo … pandas scipy==1.* scikit-learn=0.23.2 beautifulsoup4=4.9.3 … Fuzzy Version The build needs to be reproducible Do not specify the exact version of dependencies 12 CD Smells: ‘pandas’ does not have a version specified; ‘scipy’ has only the major release number. CD-Linter requirements.txt
13. @ccvassallo Evaluation of CD-Linter RQ1: Are the CD Smells Detected by CD-Linter Relevant to Developers? RQ2: How Accurate Is CD-Linter? RQ3: How Frequent Are the Investigated CD Smells in Practice? 13 ?
14. @ccvassallo Empirical Study 14 64 Developers (Resp. rate: 74%) RQ1: Relevance of CD Smells CD-Linter 145 (86) Issues Data Collection 5,312 Projects 6-month monitoring of states, comments, and fixes RQ2: Accuracy of CD-Linter 868 Config. files 2 validators (“k” agreement: 0.76) RQ3: Frequency of CD smells Icons from: https://www.flaticon.com/authors/freepik
15. @ccvassallo Empirical Study 15 64 Developers (Resp. rate: 74%) RQ1: Relevance of CD Smells CD-Linter 145 (86) Issues Data Collection 5,312 Projects 6-month monitoring of states, comments, and fixes RQ2: Accuracy of CD-Linter 868 Config. files 2 validators (“k” agreement: 0.76) RQ3: Frequency of CD smells Icons from: https://www.flaticon.com/authors/freepik
16. @ccvassallo RQ 1: GitLab issues reporting CD smells 16 stages: - build - package … package:snap: image: ubuntu:18.04 stage: package script: - snapcraft - echo $SNAPCRAFT_LOGIN_FILE | base64 --decode --ignore-garbage > snapcraft.login - snapcraft login --with snapcraft.login - snapcraft push *.snap --release beta allow_failure: true … https://gitlab.com/bitseater/meteo/blob/master/.gitlab-ci.yml#L107 https://gitlab.com/bitseater/meteo/-/issues/125 Fake Success Problem Fix
17. @ccvassallo RQ 1: Reactions to issues 17
18. @ccvassallo RQ 1: Reasons for rejecting issues Fake Success • Warned jobs are not essential or not fully implemented yet • The CD smell is contained in a template Retry Failure • Warned jobs are executed on out-of-control machines 18 Manual Execution • Lack of trust in automated issue reporting • Warned jobs are not fully integrated yet Fuzzy Version • Tools should be automatically updated to the latest version
19. @ccvassallo RQ 1: Reasons for rejecting issues Fake Success • Warned jobs are not essential or not fully implemented yet • The CD smell is contained in a template Retry Failure • Warned jobs are executed on out-of-control machines 19 Manual Execution • Lack of trust in automated issue reporting • Warned jobs are not fully integrated yet Fuzzy Version • Tools should be automatically updated to the latest version
20. @ccvassallo Empirical Study 20 64 Developers (Resp. rate: 74%) RQ1: Relevance of CD Smells CD-Linter 145 (86) Issues Data Collection 5,312 Projects 6-month monitoring of states, comments, and fixes RQ2: Accuracy of CD-Linter 868 Config. files 2 validators (“k” agreement: 0.76) RQ3: Frequency of CD smells
21. @ccvassallo RQ 2: Accuracy of CD-Linter Precision: 87% False positives: • Jobs (with unconventional names) executed in a release stage (Manual Execution) • Tool dependencies without versions (Fuzzy Version) 21 Recall: 94% False negatives: • Dependencies specified in a .pip file (Fuzzy Version) • Jobs with release-related names (Manual Execution)
22. @ccvassallo Empirical Study 22 64 Developers (Resp. rate: 74%) RQ1: Relevance of CD Smells CD-Linter 145 (86) Issues Data Collection 5,312 Projects 6-month monitoring of states, comments, and fixes RQ2: Accuracy of CD-Linter 868 Config. files 2 validators (“k” agreement: 0.76) RQ3: Frequency of CD smells
23. @ccvassallo 17% of projects RQ 3: Frequency of CD smells The majority of detected smells (70%) affect projects with long configuration files • 31% of them are affected by at least one CD smell 23 Fake Success Retry Failure Manual Execution Fuzzy Version 6% of projects 4% of projects 40% of projects
24. @ccvassallo Implications 24 CD-Linter as a mentor when configuring CD pipelines Linting rules have to be approved by developers Long and complex CD configurations are often smelly
25. Getting Continuous Testing Done Right with CD-Linter Carmine Vassallo @ccvassallo vassallocarmine@gmail.com I’m on the Job Market! http://tiny.uzh.ch/WV CD-Linter: Detecting violations of CD principles X Fake Success Retry Failure Manual Execution Fuzzy Version Empirical Study X 64 Developers (Resp. rate: 74%) RQ1: Relevance of CD Smells CD-Linter 145 (86) Issues Data Collection 5,312 Projects 6-month monitoring of states, comments, and fixes RQ2: Accuracy of CD-Linter 868 Config. files 2 validators (“k” agreement: 0.76) RQ3: Frequency of CD smells @ccvassallo Linters for CD Configurations X stages: - compilation - testing - qa variables: POSTGRES_USR: user POSTGRES_PWD: password compile_production_code: stage: compile script: “mvn compile” when: manual allow_failure: false compile_test_code: stage: compilation script: “mvn test” retry: 3 … CI Lint (GitLab) Syntax is incorrect: chosen stage does not exist. Hansel (Gallaba et al., 2018) SLIC (Rahman et al., 2019) CD feature is misused: command unrelated to the stage. Security smell: hard-coded secrets. .gitlab-ci.yml ? Developers typically lack awareness of CD principle (e.g., Continuous Testing) violations that threaten expected benefits (Vassallo et al., 2019)