CI 101

Streamline your Python development process with Continuous Integration. From choosing the right CI tool to implementing it in your project, this presentation will cover all the basics and best practices of CI and leave you with a working setup that you can reuse in the future.

Transcript

1. PyConPL 2023 Sebastian Witowski Python CI 101 A Beginner's Guide to Continuous Integration
2. PyConPL 2023 Sebastian Witowski Python CI 101 A Beginner's Guide to Continuous Integration
3. A bit of history
5. Internet == Collaboration
6. Tabs! Spaces!
7. The End
8. CI
9. That's why we need Continuous Integration
10. What is Continuous Integration? It's a process of merging developer's code into the main repository.
11. What is Continuous Integration? It's a process of merging developer's code into the main repository.
12. What is Continuous Integration? It's a process of merging developer's code into the main repository. Are tests passing?
13. What is Continuous Integration? It's a process of merging developer's code into the main repository. What about the code style? Are tests passing?
14. What is Continuous Integration? It's a process of merging developer's code into the main repository. What about the code style? Static analysis (flake8/pylint) Are tests passing?
15. What is Continuous Integration? It's a process of merging developer's code into the main repository. What about the code style? Static analysis (flake8/pylint) Are tests passing? Merge conflicts?
16. Automation • Release packages • Build Docker images • Publish documentation
17. Main role of the CI
18. Main role of the CI Automate things • Tests • Static analysis • Build/publish/release
19. Main role of the CI Automate things • Tests • Static analysis • Build/publish/release Ensure consistency • Different IDE/OS • File path separators • LF vs. CRLF • And yes, tabs vs. spaces
20. Bad review
21. Good review Bad review
22. Choosing a CI platform
27. Choosing a CI platform
28. The anatomy of a CI
29. The anatomy of a CI
30. The anatomy of a CI
31. The anatomy of a CI
32. https:/ /gitlab.com/switowski/python-ci-101
34. Our first pipeline
35. Our first pipeline # .gitlab-ci.yml image: python:3.11-slim-bullseye
36. Our first pipeline # .gitlab-ci.yml image: python:3.11-slim-bullseye test: script: - pip install pytest - pytest .
37. Our first pipeline # .gitlab-ci.yml image: python:3.11-slim-bullseye test: script: - pip install pytest - pytest . black: image: registry.gitlab.com/pipeline-components/black:latest script: - black --check --diff .
38. Our first pipeline # .gitlab-ci.yml image: python:3.11-slim-bullseye test: script: - pip install pytest - pytest . black: image: registry.gitlab.com/pipeline-components/black:latest script: - black --check --diff . flake8: allow_failure: true image: registry.gitlab.com/pipeline-components/flake8:0.11.2 script: - flake8 --verbose .
39. Our first pipeline # .gitlab-ci.yml image: python:3.11-slim-bullseye test: script: - pip install pytest - pytest . black: image: registry.gitlab.com/pipeline-components/black:latest script: - black --check --diff . flake8: allow_failure: true image: registry.gitlab.com/pipeline-components/flake8:0.11.2 script: - flake8 --verbose .
40. Our first pipeline # .gitlab-ci.yml image: python:3.11-slim-bullseye test: script: - pip install pytest - pytest . black: image: registry.gitlab.com/pipeline-components/black:latest script: - black --check --diff . flake8: allow_failure: true image: registry.gitlab.com/pipeline-components/flake8:0.11.2 script: - flake8 --verbose .
41. Our first pipeline # .gitlab-ci.yml image: python:3.11-slim-bullseye test: script: - pip install pytest - pytest . black: image: registry.gitlab.com/pipeline-components/black:latest script: - black --check --diff . flake8: allow_failure: true image: registry.gitlab.com/pipeline-components/flake8:0.11.2 script: - flake8 --verbose .
42. Our first pipeline
43. Our first pipeline # .gitlab-ci.yml image: python:3.11-slim-bullseye test: script: - pip install pytest - pytest . black: image: registry.gitlab.com/pipeline-components/black:latest script: - black --check --diff . flake8: allow_failure: true image: registry.gitlab.com/pipeline-components/flake8:0.11.2 script: - flake8 --verbose .
44. Our first pipeline (alternative version) # .gitlab-ci.yml image: python:3.11-slim-bullseye test: script: - pip install pytest - pytest . black: script: - pip install black - black --check --diff . flake8: allow_failure: true script: - pip install flake8 - flake8 --verbose .
45. Our first pipeline # .gitlab-ci.yml image: python:3.11-slim-bullseye test: script: - pip install pytest - pytest . black: image: registry.gitlab.com/pipeline-components/black:latest script: - black --check --diff . flake8: allow_failure: true image: registry.gitlab.com/pipeline-components/flake8:0.11.2 script: - flake8 --verbose .
46. Our first pipeline # .gitlab-ci.yml image: python:3.11-slim-bullseye test: script: - pip install pytest - pytest . black: image: registry.gitlab.com/pipeline-components/black:latest script: - black --check --diff . flake8: allow_failure: true image: registry.gitlab.com/pipeline-components/flake8:0.11.2 script: - flake8 --verbose .
47. Let's fix the pipeline
52. Config files # setup.cfg [flake8] max-line-length = 120 # pyproject.toml [tool.black] line-length = 120
53. Fixed pipeline
54. Other tools for the CI
55. Other tools for the CI • Type checks (e.g. mypy) • Security checks (e.g. bandit) • Unused code detectors (e.g. vulture) • Tools combining other tools (e.g. prospector) • Commercial static analysis tools
56. Other tools for the CI • Type checks (e.g. mypy) • Security checks (e.g. bandit) • Unused code detectors (e.g. vulture) • Tools combining other tools (e.g. prospector) • Commercial static analysis tools + plugins
57. Display test coverage
58. Test coverage test: script: - pip install pytest - pytest .
59. Test coverage # Source: https://docs.gitlab.com/ee/ci/testing/ test_coverage_visualization.html#python-example test: script: - pip install pytest pytest-cov - pytest --cov --cov-report term --cov-report xml:coverage.xml coverage: '/(?i)total.*? (100(?:\.0+)?\%|[1-9]?\d(?:\.\d+)?\%)$/' artifacts: reports: coverage_report: coverage_format: cobertura path: coverage.xml
60. Test coverage
61. Finding Balance
62. Running linters on your computer
63. Running linters on your computer • Enable a bunch of plugins in you IDE • Use pre-commit
65. pre-commit default_language_version: python: python3.11 repos: - repo: https://github.com/pre-commit/pre-commit-hooks rev: 23.3.0 hooks: - id: check-merge-conflict - id: check-toml - id: check-yaml - id: end-of-file-fixer - id: trailing-whitespace - repo: https://github.com/psf/black rev: 'refs/tags/23.3.0:refs/tags/23.3.0' hooks: - id: black alias: autoformat - repo: https://github.com/pycqa/flake8 rev: 6.0.0 hooks: - id: flake8
67. pre-commit • Makes pipelines faster (and cheaper) by moving checks to your computer
68. pre-commit • Makes pipelines faster (and cheaper) by moving checks to your computer • Everyone on your project needs to set it up correctly
69. Run linters and formatters in: Your IDE on save For a real-time feedback as you write code Combining it all together
70. Your IDE on save For a real-time feedback as you write code pre-commit hooks Shareable configuration Faster feedback than a CI pipeline Combining it all together Run linters and formatters in:
71. Your IDE on save For a real-time feedback as you write code pre-commit hooks Shareable configuration Faster feedback than a CI pipeline CI Automated way to check everyone's code Combining it all together Run linters and formatters in:
72. Best practices
73. Best practices •Run jobs in parallel
74. Best practices •Run jobs in parallel •Cache things between jobs 
 store docker images in container registry
75. Best practices •Run jobs in parallel •Cache things between jobs 
 store docker images in container registry •Fail fast 
 pytest -x
76. Best practices •Run jobs in parallel •Cache things between jobs 
 store docker images in container registry •Fail fast 
 pytest -x •Keep MR pipelines fast
77. Best practices •Run jobs in parallel •Cache things between jobs 
 store docker images in container registry •Fail fast 
 pytest -x •Keep MR pipelines fast •Keep your CI setup close to your production environment
78. Thank you! switowski.com @SebaWitowski
79. • 1st slide: https:/ /www.midjourney.com/app/search/?jobId=e4145c75-beb5-4d1f-8ab3-df613868bc2b • Caveman with a computer: https:/ /www.midjourney.com/app/search/?jobId=2ae3ba28-0ad4-4fc7-924d-c035dbac5767 • Computers in middle ages: https:/ /www.midjourney.com/app/jobs/9818d80c-91de-49c4-b406-ab422f802259/ • War: https:/ /www.midjourney.com/app/search/?jobId=73615195-2716-47c6-9ea2-06451496ad72 • Tombstone: https:/ /www.midjourney.com/app/search/?jobId=e6231e8d-ed31-4cf7-b7e5-ac146dd77c63 • Chalice in cave: https:/ /www.midjourney.com/app/jobs/27e13667-8625-4df3-a6bb-2a3cc4958109/ • Pressing a button: https:/ /www.midjourney.com/app/search/?jobId=32357809-c329-4451-9206-a78c6b1f3eee • Robot using a computer: https:/ /www.midjourney.com/app/jobs/4b3df90d-207d-4d70-8a8d-ad0197652fb3/ • Gitlab logo: https:/ /www.midjourney.com/app/search/?jobId=a5491c2d-199e-43dd-b65f-9ebefc031f65 • Octocat: https:/ /www.midjourney.com/app/search/?jobId=77c1c017-1e1e-4d19-bc49-347c780bdf80 • Balance: https:/ /www.midjourney.com/app/search/?jobId=03f4ea11-dd9b-4404-bfc0-88a3ad2a6d81 • Robots in a factory: https:/ /www.midjourney.com/app/search/?jobId=e9d9f29b-4546-42f7-b415-f33b4acddf03 Attributions Most images come from midjourney.com Drawings were done with excalidraw.com
80. Questions? switowski.com @SebaWitowski https:/ /gitlab.com/switowski/python-ci-101 Slides: https:/ /speakerdeck.com/switowski