Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Getting Continuous Testing Done Right with CD-Linter

Getting Continuous Testing Done Right with CD-Linter

An effective and efficient application of Continuous Integration (CI) and Delivery (CD) requires software projects to follow certain principles and good practices such as Continuous Testing. Configuring such a CI/CD pipeline is challenging and error-prone. Therefore, automated linters have been proposed to detect errors in the pipeline. While existing linters identify syntactic errors, detect security vulnerabilities or misuse of the features provided by build servers, they do not support developers that want to prevent common misconfigurations of a CD pipeline that potentially violate CD principles ("CD smells"). In this talk, I present CD-Linter, a semantic linter that can automatically identify four different smells in pipeline configuration files, and show how it can help to foster Continuous Testing. We have evaluated our linter through a large-scale and long-term study on GitLab that consists of (i) monitoring 145 issues (opened in as many open-source projects) over a period of 6 months, (ii) manually validating the detection precision and recall on a representative sample of issues, and (iii) assessing the magnitude of the observed smells on 5,312 open-source projects. Our results show that CD smells are accepted and fixed by most of the developers and our linter achieves a precision of 87% and a recall of 94%. Those smells can be frequently observed in the wild, as 31% of projects with long configurations are affected by at least one smell.

84581630350da72785f7d2448549ca06?s=128

Carmine Vassallo

November 02, 2020
Tweet

Transcript

  1. Getting Continuous Testing Done Right with CD-Linter Carmine Vassallo University

    of Zurich @ccvassallo DevOps Institute, Continuous Testing SKILup Day, November 19, 2020 Image from https://unsplash.com/photos/vBvfXIqC4E4
  2. @ccvassallo Who Am I My name is Carmine Vassallo Research

    intern in the Continuous Delivery team at ING Nederland (2015) PhD Graduate from the University of Zurich (2020), where I am currently a postdoctoral researcher My research goal is to facilitate the adoption of DevOps practices 2 I’m on the Job Market! http://tiny.uzh.ch/WV
  3. @ccvassallo 3 @ccvassallo Continuous Testing is a foundation of Continuous

    Delivery (Humble et Farley, 2010)
  4. @ccvassallo 4 Compilation Testing Quality Assurance Continuous Delivery (CD) Repository

    Commit (often) Build Server Poll Release Candidate Build stages: - compilation - testing - qa variables: POSTGRES_USR: user POSTGRES_PWD: password compile_production_code: stage: compile script: “mvn compile” when: manual allow_failure: false compile_test_code: stage: compilation script: “mvn test” retry: 3 … .gitlab-ci.yml Build pipeline Icons from https://vitalitychicago.com/blog/top-reasons-agile-didnt-work-for-us-1-we-couldnt-co-locate-teams/, https://www.flaticon.com/authors/roundicons, https://www.pinclipart.com/pindetail/hbTb_clipart-info-server-png-transparent-png/
  5. @ccvassallo 5 Compilation Testing Quality Assurance Continuous Delivery (CD) Repository

    Commit (often) Build Server Poll Release Candidate Build stages: - compilation - testing - qa variables: POSTGRES_USR: user POSTGRES_PWD: password compile_production_code: stage: compile script: “mvn compile” when: manual allow_failure: false compile_test_code: stage: compilation script: “mvn test” retry: 3 … .gitlab-ci.yml Build pipeline Developers struggle configuring build pipelines (Hilton et al., 2017)
  6. @ccvassallo Linters for CD Configurations 6 stages: - compilation -

    testing - qa variables: POSTGRES_USR: user POSTGRES_PWD: password compile_production_code: stage: compile script: “mvn compile” when: manual allow_failure: false compile_test_code: stage: compilation script: “mvn test” retry: 3 … CI Lint (GitLab) Syntax is incorrect: chosen stage does not exist. Hansel (Gallaba et al., 2018) CD feature is misused: command unrelated to the stage. SLIC (Rahman et al., 2019) Security smell: hard-coded secrets. .gitlab-ci.yml ?
  7. @ccvassallo Linters for CD Configurations 7 stages: - compilation -

    testing - qa variables: POSTGRES_USR: user POSTGRES_PWD: password compile_production_code: stage: compile script: “mvn compile” when: manual allow_failure: false compile_test_code: stage: compilation script: “mvn test” retry: 3 … CI Lint (GitLab) Syntax is incorrect: chosen stage does not exist. Hansel (Gallaba et al., 2018) SLIC (Rahman et al., 2019) CD feature is misused: command unrelated to the stage. Security smell: hard-coded secrets. .gitlab-ci.yml ? Developers typically lack awareness of CD principle (e.g., Continuous Testing) violations that threaten expected benefits (Vassallo et al., 2019)
  8. @ccvassallo CD-Linter: Detecting violations of CD principles 8 Fake Success

    Retry Failure Manual Execution Fuzzy Version Carmine Vassallo, Sebastian Proksch, Anna Jancso, Harald C. Gall, Massimiliano Di Penta. Configuration Smells in Continuous Delivery Pipelines: A Linter and A Six-Month Study on GitLab. In ESEC/FSE, 2020.
  9. @ccvassallo Fake Success Fail the build in presence of defects

    Prevent job failures from failing the build 9 … unit_test: stage: testing script: “mvn test” allow_failure: false … … CD Smell: ‘unit_test’ job is not allowed to fail. CD-Linter .gitlab-ci.yml
  10. @ccvassallo Retry Failure The build process has to be deterministic

    Hiding flakiness by rerunning a job multiple times after failures. 10 … unit_test: stage: testing script: “mvn test” retry: 3 … … CD Smell: ‘unit_test’ job is retried after failures. CD-Linter .gitlab-ci.yml
  11. @ccvassallo Manual Execution The pipeline has to be fully automated

    Some jobs are triggered manually 11 … unit_test: stage: testing script: “mvn test” when: manual … … CD Smell: ‘unit_test’ job is executed manually. CD-Linter .gitlab-ci.yml
  12. @ccvassallo … pandas scipy==1.* scikit-learn=0.23.2 beautifulsoup4=4.9.3 … Fuzzy Version The

    build needs to be reproducible Do not specify the exact version of dependencies 12 CD Smells: ‘pandas’ does not have a version specified; ‘scipy’ has only the major release number. CD-Linter requirements.txt
  13. @ccvassallo Evaluation of CD-Linter RQ1: Are the CD Smells Detected

    by CD-Linter Relevant to Developers? RQ2: How Accurate Is CD-Linter? RQ3: How Frequent Are the Investigated CD Smells in Practice? 13 ?
  14. @ccvassallo Empirical Study 14 64 Developers (Resp. rate: 74%) RQ1:

    Relevance of CD Smells CD-Linter 145 (86) Issues Data Collection 5,312 Projects 6-month monitoring of states, comments, and fixes RQ2: Accuracy of CD-Linter 868 Config. files 2 validators (“k” agreement: 0.76) RQ3: Frequency of CD smells Icons from: https://www.flaticon.com/authors/freepik
  15. @ccvassallo Empirical Study 15 64 Developers (Resp. rate: 74%) RQ1:

    Relevance of CD Smells CD-Linter 145 (86) Issues Data Collection 5,312 Projects 6-month monitoring of states, comments, and fixes RQ2: Accuracy of CD-Linter 868 Config. files 2 validators (“k” agreement: 0.76) RQ3: Frequency of CD smells Icons from: https://www.flaticon.com/authors/freepik
  16. @ccvassallo RQ 1: GitLab issues reporting CD smells 16 stages:

    - build - package … package:snap: image: ubuntu:18.04 stage: package script: - snapcraft - echo $SNAPCRAFT_LOGIN_FILE | base64 --decode --ignore-garbage > snapcraft.login - snapcraft login --with snapcraft.login - snapcraft push *.snap --release beta allow_failure: true … https://gitlab.com/bitseater/meteo/blob/master/.gitlab-ci.yml#L107 https://gitlab.com/bitseater/meteo/-/issues/125 Fake Success Problem Fix
  17. @ccvassallo RQ 1: Reactions to issues 17

  18. @ccvassallo RQ 1: Reasons for rejecting issues Fake Success •

    Warned jobs are not essential or not fully implemented yet • The CD smell is contained in a template Retry Failure • Warned jobs are executed on out-of-control machines 18 Manual Execution • Lack of trust in automated issue reporting • Warned jobs are not fully integrated yet Fuzzy Version • Tools should be automatically updated to the latest version
  19. @ccvassallo RQ 1: Reasons for rejecting issues Fake Success •

    Warned jobs are not essential or not fully implemented yet • The CD smell is contained in a template Retry Failure • Warned jobs are executed on out-of-control machines 19 Manual Execution • Lack of trust in automated issue reporting • Warned jobs are not fully integrated yet Fuzzy Version • Tools should be automatically updated to the latest version
  20. @ccvassallo Empirical Study 20 64 Developers (Resp. rate: 74%) RQ1:

    Relevance of CD Smells CD-Linter 145 (86) Issues Data Collection 5,312 Projects 6-month monitoring of states, comments, and fixes RQ2: Accuracy of CD-Linter 868 Config. files 2 validators (“k” agreement: 0.76) RQ3: Frequency of CD smells
  21. @ccvassallo RQ 2: Accuracy of CD-Linter Precision: 87% False positives:

    • Jobs (with unconventional names) executed in a release stage (Manual Execution) • Tool dependencies without versions (Fuzzy Version) 21 Recall: 94% False negatives: • Dependencies specified in a .pip file (Fuzzy Version) • Jobs with release-related names (Manual Execution)
  22. @ccvassallo Empirical Study 22 64 Developers (Resp. rate: 74%) RQ1:

    Relevance of CD Smells CD-Linter 145 (86) Issues Data Collection 5,312 Projects 6-month monitoring of states, comments, and fixes RQ2: Accuracy of CD-Linter 868 Config. files 2 validators (“k” agreement: 0.76) RQ3: Frequency of CD smells
  23. @ccvassallo 17% of projects RQ 3: Frequency of CD smells

    The majority of detected smells (70%) affect projects with long configuration files • 31% of them are affected by at least one CD smell 23 Fake Success Retry Failure Manual Execution Fuzzy Version 6% of projects 4% of projects 40% of projects
  24. @ccvassallo Implications 24 CD-Linter as a mentor when configuring CD

    pipelines Linting rules have to be approved by developers Long and complex CD configurations are often smelly
  25. Getting Continuous Testing Done Right with CD-Linter Carmine Vassallo @ccvassallo

    vassallocarmine@gmail.com I’m on the Job Market! http://tiny.uzh.ch/WV CD-Linter: Detecting violations of CD principles X Fake Success Retry Failure Manual Execution Fuzzy Version Empirical Study X 64 Developers (Resp. rate: 74%) RQ1: Relevance of CD Smells CD-Linter 145 (86) Issues Data Collection 5,312 Projects 6-month monitoring of states, comments, and fixes RQ2: Accuracy of CD-Linter 868 Config. files 2 validators (“k” agreement: 0.76) RQ3: Frequency of CD smells @ccvassallo Linters for CD Configurations X stages: - compilation - testing - qa variables: POSTGRES_USR: user POSTGRES_PWD: password compile_production_code: stage: compile script: “mvn compile” when: manual allow_failure: false compile_test_code: stage: compilation script: “mvn test” retry: 3 … CI Lint (GitLab) Syntax is incorrect: chosen stage does not exist. Hansel (Gallaba et al., 2018) SLIC (Rahman et al., 2019) CD feature is misused: command unrelated to the stage. Security smell: hard-coded secrets. .gitlab-ci.yml ? Developers typically lack awareness of CD principle (e.g., Continuous Testing) violations that threaten expected benefits (Vassallo et al., 2019)