CI 101

Streamline your Python development process with Continuous Integration. From choosing the right CI tool to implementing it in your project, this presentation will cover all the basics and best practices of CI and leave you with a working setup that you can reuse in the future.

Transcript

1. PyConPL 2023
   Sebastian Witowski
   Python CI 101
   A Beginner's Guide to
   Continuous Integration
   

   View Slide

2. PyConPL 2023
   Sebastian Witowski
   Python CI 101
   A Beginner's Guide to
   Continuous Integration
   

   View Slide

3. A bit of history
   

   View Slide

4. View Slide

5. Internet
   
   
   ==
   
   
   Collaboration
   

   View Slide

6. Tabs!
   Spaces!
   

   View Slide

7. The End
   

   View Slide

8. CI
   

   View Slide

9. That's why we need
   Continuous
   Integration
   

   View Slide

10. What is Continuous Integration?
    It's a process of merging developer's code into the
    main repository.
    

    View Slide

11. What is Continuous Integration?
    It's a process of merging developer's code into the
    main repository.
    

    View Slide

12. What is Continuous Integration?
    It's a process of merging developer's code into the
    main repository.
    Are tests passing?
    

    View Slide

13. What is Continuous Integration?
    It's a process of merging developer's code into the
    main repository.
    What about the code style?
    Are tests passing?
    

    View Slide

14. What is Continuous Integration?
    It's a process of merging developer's code into the
    main repository.
    What about the code style?
    Static analysis (flake8/pylint)
    Are tests passing?
    

    View Slide

15. What is Continuous Integration?
    It's a process of merging developer's code into the
    main repository.
    What about the code style?
    Static analysis (flake8/pylint)
    Are tests passing?
    Merge conflicts?
    

    View Slide

16. Automation
    • Release packages
    
    
    • Build Docker images
    
    
    • Publish documentation
    

    View Slide

17. Main role of the CI
    

    View Slide

18. Main role of the CI
    Automate things
    • Tests
    
    
    • Static analysis
    
    
    • Build/publish/release
    

    View Slide

19. Main role of the CI
    Automate things
    • Tests
    
    
    • Static analysis
    
    
    • Build/publish/release
    Ensure consistency
    • Different IDE/OS
    
    
    • File path separators
    
    
    • LF vs. CRLF
    
    
    • And yes, tabs vs. spaces
    

    View Slide

20. Bad review
    

    View Slide

21. Good review
    Bad review
    

    View Slide

22. Choosing a CI platform
    

    View Slide

23. View Slide

24. View Slide

25. View Slide

26. View Slide

27. Choosing a CI platform
    

    View Slide

28. The anatomy of a CI
    

    View Slide

29. The anatomy of a CI
    

    View Slide

30. The anatomy of a CI
    

    View Slide

31. The anatomy of a CI
    

    View Slide

32. https:/
    /gitlab.com/switowski/python-ci-101
    

    View Slide

33. View Slide

34. Our first pipeline
    

    View Slide

35. Our first pipeline
    # .gitlab-ci.yml
    
    
    image: python:3.11-slim-bullseye
    
    
    

    View Slide

36. Our first pipeline
    # .gitlab-ci.yml
    
    
    image: python:3.11-slim-bullseye
    
    
    test:
    
    
    script:
    
    
    - pip install pytest
    
    
    - pytest .
    
    
    

    View Slide

37. Our first pipeline
    # .gitlab-ci.yml
    
    
    image: python:3.11-slim-bullseye
    
    
    test:
    
    
    script:
    
    
    - pip install pytest
    
    
    - pytest .
    
    
    black:
    
    
    image: registry.gitlab.com/pipeline-components/black:latest
    
    
    script:
    
    
    - black --check --diff .
    
    
    

    View Slide

38. Our first pipeline
    # .gitlab-ci.yml
    
    
    image: python:3.11-slim-bullseye
    
    
    test:
    
    
    script:
    
    
    - pip install pytest
    
    
    - pytest .
    
    
    black:
    
    
    image: registry.gitlab.com/pipeline-components/black:latest
    
    
    script:
    
    
    - black --check --diff .
    
    
    flake8:
    
    
    allow_failure: true
    
    
    image: registry.gitlab.com/pipeline-components/flake8:0.11.2
    
    
    script:
    
    
    - flake8 --verbose .
    
    
    

    View Slide

39. Our first pipeline
    # .gitlab-ci.yml
    
    
    image: python:3.11-slim-bullseye
    
    
    test:
    
    
    script:
    
    
    - pip install pytest
    
    
    - pytest .
    
    
    black:
    
    
    image: registry.gitlab.com/pipeline-components/black:latest
    
    
    script:
    
    
    - black --check --diff .
    
    
    flake8:
    
    
    allow_failure: true
    
    
    image: registry.gitlab.com/pipeline-components/flake8:0.11.2
    
    
    script:
    
    
    - flake8 --verbose .
    
    
    

    View Slide

40. Our first pipeline
    # .gitlab-ci.yml
    
    
    image: python:3.11-slim-bullseye
    
    
    test:
    
    
    script:
    
    
    - pip install pytest
    
    
    - pytest .
    
    
    black:
    
    
    image: registry.gitlab.com/pipeline-components/black:latest
    
    
    script:
    
    
    - black --check --diff .
    
    
    flake8:
    
    
    allow_failure: true
    
    
    image: registry.gitlab.com/pipeline-components/flake8:0.11.2
    
    
    script:
    
    
    - flake8 --verbose .
    
    
    

    View Slide

41. Our first pipeline
    # .gitlab-ci.yml
    
    
    image: python:3.11-slim-bullseye
    
    
    test:
    
    
    script:
    
    
    - pip install pytest
    
    
    - pytest .
    
    
    black:
    
    
    image: registry.gitlab.com/pipeline-components/black:latest
    
    
    script:
    
    
    - black --check --diff .
    
    
    flake8:
    
    
    allow_failure: true
    
    
    image: registry.gitlab.com/pipeline-components/flake8:0.11.2
    
    
    script:
    
    
    - flake8 --verbose .
    
    
    

    View Slide

42. Our first pipeline
    

    View Slide

43. Our first pipeline
    # .gitlab-ci.yml
    
    
    image: python:3.11-slim-bullseye
    
    
    test:
    
    
    script:
    
    
    - pip install pytest
    
    
    - pytest .
    
    
    black:
    
    
    image: registry.gitlab.com/pipeline-components/black:latest
    
    
    script:
    
    
    - black --check --diff .
    
    
    flake8:
    
    
    allow_failure: true
    
    
    image: registry.gitlab.com/pipeline-components/flake8:0.11.2
    
    
    script:
    
    
    - flake8 --verbose .
    
    
    

    View Slide

44. Our first pipeline (alternative version)
    # .gitlab-ci.yml
    
    
    image: python:3.11-slim-bullseye
    
    
    test:
    
    
    script:
    
    
    - pip install pytest
    
    
    - pytest .
    
    
    black:
    
    
    script:
    
    
    - pip install black
    
    
    - black --check --diff .
    
    
    flake8:
    
    
    allow_failure: true
    
    
    script:
    
    
    - pip install flake8
    
    
    - flake8 --verbose .
    
    
    

    View Slide

45. Our first pipeline
    # .gitlab-ci.yml
    
    
    image: python:3.11-slim-bullseye
    
    
    test:
    
    
    script:
    
    
    - pip install pytest
    
    
    - pytest .
    
    
    black:
    
    
    image: registry.gitlab.com/pipeline-components/black:latest
    
    
    script:
    
    
    - black --check --diff .
    
    
    flake8:
    
    
    allow_failure: true
    
    
    image: registry.gitlab.com/pipeline-components/flake8:0.11.2
    
    
    script:
    
    
    - flake8 --verbose .
    
    
    

    View Slide

46. Our first pipeline
    # .gitlab-ci.yml
    
    
    image: python:3.11-slim-bullseye
    
    
    test:
    
    
    script:
    
    
    - pip install pytest
    
    
    - pytest .
    
    
    black:
    
    
    image: registry.gitlab.com/pipeline-components/black:latest
    
    
    script:
    
    
    - black --check --diff .
    
    
    flake8:
    
    
    allow_failure: true
    
    
    image: registry.gitlab.com/pipeline-components/flake8:0.11.2
    
    
    script:
    
    
    - flake8 --verbose .
    
    
    

    View Slide

47. Let's fix the pipeline
    

    View Slide

48. View Slide

49. View Slide

50. View Slide

51. View Slide

52. Config files
    # setup.cfg
    
    
    [flake8]
    
    
    max-line-length = 120
    
    
    # pyproject.toml
    
    
    [tool.black]
    
    
    line-length = 120
    
    
    

    View Slide

53. Fixed pipeline
    

    View Slide

54. Other tools for the CI
    

    View Slide

55. Other tools for the CI
    • Type checks (e.g. mypy)
    
    
    • Security checks (e.g. bandit)
    
    
    • Unused code detectors (e.g. vulture)
    
    
    • Tools combining other tools (e.g. prospector)
    
    
    • Commercial static analysis tools
    

    View Slide

56. Other tools for the CI
    • Type checks (e.g. mypy)
    
    
    • Security checks (e.g. bandit)
    
    
    • Unused code detectors (e.g. vulture)
    
    
    • Tools combining other tools (e.g. prospector)
    
    
    • Commercial static analysis tools
    + plugins
    

    View Slide

57. Display test coverage
    

    View Slide

58. Test coverage
    test:
    
    
    script:
    
    
    - pip install pytest
    
    
    - pytest .
    
    
    

    View Slide

59. Test coverage
    # Source: https://docs.gitlab.com/ee/ci/testing/
    test_coverage_visualization.html#python-example
    
    
    test:
    
    
    script:
    
    
    - pip install pytest pytest-cov
    
    
    - pytest --cov --cov-report term --cov-report xml:coverage.xml
    
    
    coverage: '/(?i)total.*? (100(?:\.0+)?\%|[1-9]?\d(?:\.\d+)?\%)$/'
    
    
    artifacts:
    
    
    reports:
    
    
    coverage_report:
    
    
    coverage_format: cobertura
    
    
    path: coverage.xml
    
    
    

    View Slide

60. Test coverage
    

    View Slide

61. Finding
    Balance
    

    View Slide

62. Running linters on your computer
    

    View Slide

63. Running linters on your computer
    • Enable a bunch of plugins in you IDE
    
    
    • Use pre-commit
    

    View Slide

64. View Slide

65. pre-commit
    default_language_version:
    
    
    python: python3.11
    
    
    repos:
    
    
    - repo: https://github.com/pre-commit/pre-commit-hooks
    
    
    rev: 23.3.0
    
    
    hooks:
    
    
    - id: check-merge-conflict
    
    
    - id: check-toml
    
    
    - id: check-yaml
    
    
    - id: end-of-file-fixer
    
    
    - id: trailing-whitespace
    
    
    - repo: https://github.com/psf/black
    
    
    rev: 'refs/tags/23.3.0:refs/tags/23.3.0'
    
    
    hooks:
    
    
    - id: black
    
    
    alias: autoformat
    
    
    - repo: https://github.com/pycqa/flake8
    
    
    rev: 6.0.0
    
    
    hooks:
    
    
    - id: flake8
    
    
    

    View Slide

66. View Slide

67. pre-commit
    • Makes pipelines faster (and cheaper) by moving checks
    to your computer
    

    View Slide

68. pre-commit
    • Makes pipelines faster (and cheaper) by moving checks
    to your computer
    
    
    • Everyone on your project needs to set it up correctly
    

    View Slide

69. Run linters and
    formatters in:
    Your IDE on save
    For a real-time feedback as you write code
    Combining it all together
    

    View Slide

70. Your IDE on save
    For a real-time feedback as you write code
    pre-commit hooks
    Shareable configuration
    
    
    Faster feedback than a CI pipeline
    Combining it all together
    Run linters and
    formatters in:
    

    View Slide

71. Your IDE on save
    For a real-time feedback as you write code
    pre-commit hooks
    Shareable configuration
    
    
    Faster feedback than a CI pipeline
    CI
    Automated way to check everyone's code
    Combining it all together
    Run linters and
    formatters in:
    

    View Slide

72. Best practices
    

    View Slide

73. Best practices
    •Run jobs in parallel
    

    View Slide

74. Best practices
    •Run jobs in parallel
    
    
    •Cache things between jobs
    

    store docker images in container registry
    
    
    

    View Slide

75. Best practices
    •Run jobs in parallel
    
    
    •Cache things between jobs
    

    store docker images in container registry
    
    
    •Fail fast
    

    pytest -x
    

    View Slide

76. Best practices
    •Run jobs in parallel
    
    
    •Cache things between jobs
    

    store docker images in container registry
    
    
    •Fail fast
    

    pytest -x
    
    
    •Keep MR pipelines fast
    

    View Slide

77. Best practices
    •Run jobs in parallel
    
    
    •Cache things between jobs
    

    store docker images in container registry
    
    
    •Fail fast
    

    pytest -x
    
    
    •Keep MR pipelines fast
    
    
    •Keep your CI setup close to your production
    environment
    

    View Slide

78. Thank you!
    switowski.com
    
    
    @SebaWitowski
    

    View Slide

79. • 1st slide: https:/
    /www.midjourney.com/app/search/?jobId=e4145c75-beb5-4d1f-8ab3-df613868bc2b
    
    
    • Caveman with a computer: https:/
    /www.midjourney.com/app/search/?jobId=2ae3ba28-0ad4-4fc7-924d-c035dbac5767
    
    
    • Computers in middle ages: https:/
    /www.midjourney.com/app/jobs/9818d80c-91de-49c4-b406-ab422f802259/
    
    
    • War: https:/
    /www.midjourney.com/app/search/?jobId=73615195-2716-47c6-9ea2-06451496ad72
    
    
    • Tombstone: https:/
    /www.midjourney.com/app/search/?jobId=e6231e8d-ed31-4cf7-b7e5-ac146dd77c63
    
    
    • Chalice in cave: https:/
    /www.midjourney.com/app/jobs/27e13667-8625-4df3-a6bb-2a3cc4958109/
    
    
    • Pressing a button: https:/
    /www.midjourney.com/app/search/?jobId=32357809-c329-4451-9206-a78c6b1f3eee
    
    
    • Robot using a computer: https:/
    /www.midjourney.com/app/jobs/4b3df90d-207d-4d70-8a8d-ad0197652fb3/
    
    
    • Gitlab logo: https:/
    /www.midjourney.com/app/search/?jobId=a5491c2d-199e-43dd-b65f-9ebefc031f65
    
    
    • Octocat: https:/
    /www.midjourney.com/app/search/?jobId=77c1c017-1e1e-4d19-bc49-347c780bdf80
    
    
    • Balance: https:/
    /www.midjourney.com/app/search/?jobId=03f4ea11-dd9b-4404-bfc0-88a3ad2a6d81
    
    
    • Robots in a factory: https:/
    /www.midjourney.com/app/search/?jobId=e9d9f29b-4546-42f7-b415-f33b4acddf03
    Attributions
    Most images come from midjourney.com
    
    
    Drawings were done with excalidraw.com
    

    View Slide

80. Questions?
    switowski.com
    
    
    @SebaWitowski
    https:/
    /gitlab.com/switowski/python-ci-101
    Slides: https:/
    /speakerdeck.com/switowski
    

    View Slide