$30 off During Our Annual Pro Sale. View Details »

CI 101

CI 101

Streamline your Python development process with Continuous Integration. From choosing the right CI tool to implementing it in your project, this presentation will cover all the basics and best practices of CI and leave you with a working setup that you can reuse in the future.

switowski

June 29, 2023
Tweet

More Decks by switowski

Other Decks in Programming

Transcript

  1. CI

  2. What is Continuous Integration? It's a process of merging developer's

    code into the main repository. Are tests passing?
  3. What is Continuous Integration? It's a process of merging developer's

    code into the main repository. What about the code style? Are tests passing?
  4. What is Continuous Integration? It's a process of merging developer's

    code into the main repository. What about the code style? Static analysis (flake8/pylint) Are tests passing?
  5. What is Continuous Integration? It's a process of merging developer's

    code into the main repository. What about the code style? Static analysis (flake8/pylint) Are tests passing? Merge conflicts?
  6. Main role of the CI Automate things • Tests •

    Static analysis • Build/publish/release
  7. Main role of the CI Automate things • Tests •

    Static analysis • Build/publish/release Ensure consistency • Different IDE/OS • File path separators • LF vs. CRLF • And yes, tabs vs. spaces
  8. Our first pipeline # .gitlab-ci.yml image: python:3.11-slim-bullseye test: script: -

    pip install pytest - pytest . black: image: registry.gitlab.com/pipeline-components/black:latest script: - black --check --diff .
  9. Our first pipeline # .gitlab-ci.yml image: python:3.11-slim-bullseye test: script: -

    pip install pytest - pytest . black: image: registry.gitlab.com/pipeline-components/black:latest script: - black --check --diff . flake8: allow_failure: true image: registry.gitlab.com/pipeline-components/flake8:0.11.2 script: - flake8 --verbose .
  10. Our first pipeline # .gitlab-ci.yml image: python:3.11-slim-bullseye test: script: -

    pip install pytest - pytest . black: image: registry.gitlab.com/pipeline-components/black:latest script: - black --check --diff . flake8: allow_failure: true image: registry.gitlab.com/pipeline-components/flake8:0.11.2 script: - flake8 --verbose .
  11. Our first pipeline # .gitlab-ci.yml image: python:3.11-slim-bullseye test: script: -

    pip install pytest - pytest . black: image: registry.gitlab.com/pipeline-components/black:latest script: - black --check --diff . flake8: allow_failure: true image: registry.gitlab.com/pipeline-components/flake8:0.11.2 script: - flake8 --verbose .
  12. Our first pipeline # .gitlab-ci.yml image: python:3.11-slim-bullseye test: script: -

    pip install pytest - pytest . black: image: registry.gitlab.com/pipeline-components/black:latest script: - black --check --diff . flake8: allow_failure: true image: registry.gitlab.com/pipeline-components/flake8:0.11.2 script: - flake8 --verbose .
  13. Our first pipeline # .gitlab-ci.yml image: python:3.11-slim-bullseye test: script: -

    pip install pytest - pytest . black: image: registry.gitlab.com/pipeline-components/black:latest script: - black --check --diff . flake8: allow_failure: true image: registry.gitlab.com/pipeline-components/flake8:0.11.2 script: - flake8 --verbose .
  14. Our first pipeline (alternative version) # .gitlab-ci.yml image: python:3.11-slim-bullseye test:

    script: - pip install pytest - pytest . black: script: - pip install black - black --check --diff . flake8: allow_failure: true script: - pip install flake8 - flake8 --verbose .
  15. Our first pipeline # .gitlab-ci.yml image: python:3.11-slim-bullseye test: script: -

    pip install pytest - pytest . black: image: registry.gitlab.com/pipeline-components/black:latest script: - black --check --diff . flake8: allow_failure: true image: registry.gitlab.com/pipeline-components/flake8:0.11.2 script: - flake8 --verbose .
  16. Our first pipeline # .gitlab-ci.yml image: python:3.11-slim-bullseye test: script: -

    pip install pytest - pytest . black: image: registry.gitlab.com/pipeline-components/black:latest script: - black --check --diff . flake8: allow_failure: true image: registry.gitlab.com/pipeline-components/flake8:0.11.2 script: - flake8 --verbose .
  17. Other tools for the CI • Type checks (e.g. mypy)

    • Security checks (e.g. bandit) • Unused code detectors (e.g. vulture) • Tools combining other tools (e.g. prospector) • Commercial static analysis tools
  18. Other tools for the CI • Type checks (e.g. mypy)

    • Security checks (e.g. bandit) • Unused code detectors (e.g. vulture) • Tools combining other tools (e.g. prospector) • Commercial static analysis tools + plugins
  19. Test coverage # Source: https://docs.gitlab.com/ee/ci/testing/ test_coverage_visualization.html#python-example test: script: - pip

    install pytest pytest-cov - pytest --cov --cov-report term --cov-report xml:coverage.xml coverage: '/(?i)total.*? (100(?:\.0+)?\%|[1-9]?\d(?:\.\d+)?\%)$/' artifacts: reports: coverage_report: coverage_format: cobertura path: coverage.xml
  20. Running linters on your computer • Enable a bunch of

    plugins in you IDE • Use pre-commit
  21. pre-commit default_language_version: python: python3.11 repos: - repo: https://github.com/pre-commit/pre-commit-hooks rev: 23.3.0

    hooks: - id: check-merge-conflict - id: check-toml - id: check-yaml - id: end-of-file-fixer - id: trailing-whitespace - repo: https://github.com/psf/black rev: 'refs/tags/23.3.0:refs/tags/23.3.0' hooks: - id: black alias: autoformat - repo: https://github.com/pycqa/flake8 rev: 6.0.0 hooks: - id: flake8
  22. pre-commit • Makes pipelines faster (and cheaper) by moving checks

    to your computer • Everyone on your project needs to set it up correctly
  23. Run linters and formatters in: Your IDE on save For

    a real-time feedback as you write code Combining it all together
  24. Your IDE on save For a real-time feedback as you

    write code pre-commit hooks Shareable configuration Faster feedback than a CI pipeline Combining it all together Run linters and formatters in:
  25. Your IDE on save For a real-time feedback as you

    write code pre-commit hooks Shareable configuration Faster feedback than a CI pipeline CI Automated way to check everyone's code Combining it all together Run linters and formatters in:
  26. Best practices •Run jobs in parallel •Cache things between jobs

    
 store docker images in container registry
  27. Best practices •Run jobs in parallel •Cache things between jobs

    
 store docker images in container registry •Fail fast 
 pytest -x
  28. Best practices •Run jobs in parallel •Cache things between jobs

    
 store docker images in container registry •Fail fast 
 pytest -x •Keep MR pipelines fast
  29. Best practices •Run jobs in parallel •Cache things between jobs

    
 store docker images in container registry •Fail fast 
 pytest -x •Keep MR pipelines fast •Keep your CI setup close to your production environment
  30. • 1st slide: https:/ /www.midjourney.com/app/search/?jobId=e4145c75-beb5-4d1f-8ab3-df613868bc2b • Caveman with a computer:

    https:/ /www.midjourney.com/app/search/?jobId=2ae3ba28-0ad4-4fc7-924d-c035dbac5767 • Computers in middle ages: https:/ /www.midjourney.com/app/jobs/9818d80c-91de-49c4-b406-ab422f802259/ • War: https:/ /www.midjourney.com/app/search/?jobId=73615195-2716-47c6-9ea2-06451496ad72 • Tombstone: https:/ /www.midjourney.com/app/search/?jobId=e6231e8d-ed31-4cf7-b7e5-ac146dd77c63 • Chalice in cave: https:/ /www.midjourney.com/app/jobs/27e13667-8625-4df3-a6bb-2a3cc4958109/ • Pressing a button: https:/ /www.midjourney.com/app/search/?jobId=32357809-c329-4451-9206-a78c6b1f3eee • Robot using a computer: https:/ /www.midjourney.com/app/jobs/4b3df90d-207d-4d70-8a8d-ad0197652fb3/ • Gitlab logo: https:/ /www.midjourney.com/app/search/?jobId=a5491c2d-199e-43dd-b65f-9ebefc031f65 • Octocat: https:/ /www.midjourney.com/app/search/?jobId=77c1c017-1e1e-4d19-bc49-347c780bdf80 • Balance: https:/ /www.midjourney.com/app/search/?jobId=03f4ea11-dd9b-4404-bfc0-88a3ad2a6d81 • Robots in a factory: https:/ /www.midjourney.com/app/search/?jobId=e9d9f29b-4546-42f7-b415-f33b4acddf03 Attributions Most images come from midjourney.com Drawings were done with excalidraw.com