Python Table Manners @ Taichung.py

想要在你的團隊建立標準的 Python 專案開發流程嗎？那就來聽這場分享吧！我將會介紹一些增加程式碼品質的小工具們。透過這些工具，你可以省去很多無聊且繁瑣的任務，把心力留給更重要的軟體設計。在最後，我會分享我如何把這些工具整合起來，讓你可以輕鬆地把這些工具帶到你的下一個專案。

Transcript

1. @clleew Python Table Manners Everything you need in software development life cycle
2. @clleew Slide
3. @clleew Outline • Clean up the table - Dependency Management • Put the correct tablewares - Testing • Use the tablewares elegantly - Coding Style • Mnemonic phrase - Task management • Say please! - Check before git operation • Speak formally - Cultivate a git commit convention • Safety matters - Security Issues • Where's the Cookie? - Project template
4. @clleew __name__ = 李唯 / Wei Lee __position__ = [ Software Engineer @ Rakuten Slice, 2021 Chairperson @ PyCon Taiwan, Maintainer of commitizen-tools, ] __twitter__ = @clleew __github__ = Lee-W __blog__ = http://lee-w.github.io $ cat speaker.py
5. @clleew $ python speaker.py File "speaker.py", line 1 __name__ = 李唯 / Wei Lee ^ SyntaxError: invalid syntax $ python speaker.py
6. @clleew Let's join PyCon TW 2021
7. @clleew Dependency Management Clean up the table
8. @clleew How we used to start a Python project
9. @clleew Sometimes we just forgot to… • ... activate / deactivate virtual environment • ... add package into requirements.txt
10. @clleew Pipenv Python Dev Workflow for Humans • manage virtual environment + package → update at once → No more manual sync up! • generate hashes from packages → ensure getting the same packages next time
11. @clleew Initial Virtual Environment
12. @clleew Empty Pipfile [[source]] name = "pypi" url = "https://pypi.org/simple" verify_ssl = true [dev-packages] [packages] [requires] python_version = "3.7"
13. @clleew Install Package
14. @clleew Pipfile with “requests” [[source]] name = "pypi" url = "https://pypi.org/simple" verify_ssl = true [dev-packages] [packages] requests = "*" [requires] python_version = "3.7" [packages] requests = "*"
15. @clleew "requests": { "hashes": [ "sha256:11e007a8a2aa0323f5a921e9e6a2d7e4e67d9877e85773fba9ba6419025cbeb4", "sha256:9cf5292fcd0f598c671cfc1e0d7d1a7f13bb8085e9a590f48c010551dc6c4b31" ], "index": "pypi", "version": "==2.22.0" }, { ...... "default": { ...... "requests": { "hashes": [ "sha256:11e007a8a2aa0323f5a921e9e6a2d7e4e67d9877e85773fba9ba6419025cbeb4", "sha256:9cf5292fcd0f598c671cfc1e0d7d1a7f13bb8085e9a590f48c010551dc6c4b31" ], "index": "pypi", "version": "==2.22.0" }, "urllib3": { "hashes": [ "sha256:3de946ffbed6e6746608990594d08faac602528ac7015ac28d33cee6a45b7398", "sha256:9a107b99a5393caf59c7aa3c1249c16e6879447533d0887f4336dde834c7be86" ], "version": "==1.25.6" } }, ...... } Pipfile.lock with “requests” You’re guaranteed to install the exact same package next time.
16. @clleew
17. @clleew Install package only in dev
18. @clleew Pipfile with “pytest" [[source]] name = "pypi" url = "https://pypi.org/simple" verify_ssl = true [dev-packages] pytest = "*" [packages] requests = "*" [requires] python_version = "3.7" [dev-packages] pytest = "*"
19. @clleew Run in virtual environment
20. @clleew Some people might say that pipenv ... • ... does not update frequently. • ... locks slowly. • ... does not syncs up with install_requires in setup.py Or maybe you can manage dependencies in setup.py, and use “pipenv install -e .” (learned from dboy )
21. @clleew Poetry
22. @clleew Poetry commands
23. @clleew Testing Put the correct tablewares
24. @clleew unittest Unit testing framework in Python Standard Library
25. @clleew pytest
26. @clleew Why pytest • More Pythonic • Compatible with old unittest tests • No more assert.+ (e.g., assertEqual, assertTrue, etc.)
 → just assert • Better test discovery • Advance features: mark, parameterize, etc. • Plenty of plugins
27. @clleew Run pytest
28. @clleew always run python inside a virtual environment
29. @clleew unittest style tests import unittest from atta.partner import sponsor class TestSponsor(unittest.TestCase): def setUp(self): sponsors = sponsor.get_all_sponsors('./data/packages.yaml', './data/sponsors.yaml') self.sponsors = sponsors ...... def test_sponsor_number(self): self.assertEqual(len(self.sponsors), 1) ....... Prepare all the data needed in test cases
30. @clleew pytest style tests import pytest from report_generator.partner import sponsor class TestSponsor: @pytest.fixture(scope="class") def sponsors(self): return sponsor.get_all_sponsors("test/data/packages.yaml", “test/data/sponsors.yaml") ...... def test_sponsor_number(self, sponsors): assert len(sponsors) == 1 ...... Prepare the data needed in separate fixtures
31. @clleew pytest style tests import pytest from report_generator.partner import sponsor @pytest.fixture(scope="function") def sponsors(): return sponsor.get_all_sponsors("test/data/packages.yaml", "test/data/sponsors.yaml") def test_sponsor_number(sponsors): assert len(sponsors) == 1
32. @clleew Configuration - pyproject.toml [tool.pytest.ini_options] minversion = "6.0" testpaths = "tests" addopts = "--strict-markers" norecursedirs = [ ".*", "build", "dist", "CVS", "_darcs", "{arch}", "*.egg", "venv", "env", "virtualenv" ]
33. @clleew Pytest - powerful plugins • pytest-mock: Replace objects that are hard to test with fake objects • pytest-cov: Generate test coverage report • pytest-xdist: Distributed testing • pytest-regressions • and etc.
34. @clleew Coding Style Use the tablewares elegantly
35. @clleew flake8 • Enforcing style consistency across Python project • Check possible errors before running your program • Eliminate bad style
36. @clleew import os os = "My Operating system" additional space
 (bad coding style) redefinition of imported library
 (possible error) Example - bad_code.py
37. @clleew Run flake8
38. @clleew Configuration - setup.cfg [flake8] ignore = # F632: use ==/!= to compare str, bytes, and int literals F632, # W503: Line break occurred before a binary operator W503, # E501: Line too long E501, # E203: Whitespace before ':' (for black) E203 exclude = .git, __pycache__, build, dist max-line-length = 88
39. @clleew Pylint
40. @clleew import os os = "My Operating system" additional space
 (bad coding style) redefinition of imported library
 (possible error) Example - bad_code.py
41. @clleew Run pylint
42. @clleew Run pylint with "-r" argument A bunch of reports that 
 you can compare 
 with your previous run
43. Configuration - pyproject.toml [tool.pylint.messages_control] disable = [ "bad-continuation", "missing-function-docstring", "missing-module-docstring", "invalid-name" ] [tool.pylint.format] max-line-length = 88
44. @clleew mypy • Static type checker • Compile-time type checking
 → avoid possible runtime errors • Type annotation enhances readability
 → machine-checked documentation
45. from typing import List def func(vals: List[str]): for val in vals: assert isinstance(val, str) func([1, 2, 3]) Example - error_type.py List[str] [1, 2, 3]
46. Run mypy all the files with .py extension ignore errors that libs that is not type annotated
47. @clleew Configuration - setup.cfg [mypy] files = **/*.py ignore_missing_imports = true follow_imports = silent warn_redundant_casts = True warn_unused_ignores = True warn_unused_configs = True [mypy] files = **/*.py ignore_missing_imports = true
48. Run mypy with configuration
49. @clleew One step forward Fix the style automatically
50. Black
51. Run Black
52. How Black reformat
53. How Black reformat
54. Why Black? The black code style is not configurable. No more arguing about which style is better
55. –The Zen of Python, by Tim Peters “There should be one-- and preferably only one --obvious way to do it.”
56. @clleew Configuration - pyproject.toml [tool.black] line-length = 88 include = '\.pyi?$' exclude = ''' /( \.eggs | \.git | \.hg | \.mypy_cache | \.tox | \.venv | _build | buck-out | build | dist )/ '''
57. Isort
58. Example wrong_import_order.py import django import os import some_custom_package import flask import datetime ...
59. According to PEP8 • Imports should be grouped in the following order: 1. Standard library imports. 2. Related third party imports. 3. Local application/library specific imports. • You should put a blank line between each group of imports.
60. Run Isort
61. Run Isort import datetime import os import django import flask import some_custom_package ...
62. @clleew Configuration - pyproject.toml [tool.isort] profile = "black"
63. Style formatting and linting
65. @clleew Task Management Mnemonic phrase
66. Invoke • Task execution tool that can manage your commands • It’s like a Makefile but in Python.
67. Invoke in practice (rg-cli)
68. Before invoke in rg-cli
69. After invoke in rg-cli
70. List all commands
71. How ? - tasks.py from invoke import task VENV_PREFIX = "pipenv run" ...... @task def install(cmd): """Install script in pipenv environment""" cmd.run(f"{VENV_PREFIX} python setup.py install")
72. Modulize - Namespace
73. Autocomplete
74. Why not use Makefile? 1.We’re Python developers! 2.Some tasks might not be easy to handle through shell script. 3.Combining Python and shell script
75. @clleew Check before git operation Say please!
76. @clleew pre-commit • Why? • Sometimes we still forget to run the check commands. • Prevent committing bad code into codebase • How? • pre-commit runs these commands before we do git operations (e.g., git push, git commit)
77. @clleew Config (customized hooks) .pre-commit-config.yaml repos: - repo: local hooks: - id: style-reformat name: style-reformat stages: [commit] language: system pass_filenames: false entry: inv style.reformat types: [python] - id: style-check name: style-check stages: [push] language: system pass_filenames: false entry: inv style types: [python] ...... stages: [push] entry: inv style.reformat - id: style-reformat stages: [commit] - id: style-check
78. @clleew Config (existing hooks) .pre-commit-config.yaml ...... - repo: https://github.com/pre-commit/pre-commit-hooks rev: v3.2.0 hooks: - id: end-of-file-fixer - id: trailing-whitespace args: [--markdown-linebreak-ext=md]
79. @clleew Setup pre-commit
80. @clleew Run pre-commit
81. @clleew Run pre-commit
82. @clleew Cultivate a git commit convention Speak formally
84. Hard to find to the right version
85. commitizen-tools
86. Commit - cz commit
87. Commit - cz commit
88. How does the commit look like? feat(Taichung.py): This is a great meetup! Hope you enjoy my talk! https://taichung-py.kktix.cc/events/ meetup-202011-clleew
89. Standardized git commits
90. Other Commitizen features • pre-commit hook prevents you from not using commitizen • customizable commit rules • auto bump project version (SemVar) • auto generate changelog (Keep a Changelog)
91. @clleew Security Issues Safety matters
94. Safety Dependency Checking
95. Check vulnerability in current environment
96. But if you use pipenv …
97. How does Safety work? • Search the vulnerability in CVS (Common Vulnerabilities and Exposures) • Free DB: update monthly • Paid DB: update realtime
98. Bandit Static Analysis
99. Check Common Security Issue in Python Code
100. Not all the warnings should be fixed • Add exclude into your bandit configuration • Add # nosec after the end of the code to skip the warning
101. @clleew
102. @clleew Project template Where's the Cookie?
103. @clleew We want these manners in all our Python projects ❗ But... we don't want to configure it everytime ❗
104. @clleew Create a project template once and initialize projects through it
105. @clleew Lee-W / cookiecutter-python-template
106. @clleew
107. @clleew
108. @clleew How to make a template? cookiecutter.json
109. @clleew Template structure Cookiecutter configuration Template (Use Jinja)
110. @clleew An example in template {{cookiecutter.project_slug}}/tasks/env.py from invoke import task from tasks.common import VENV_PREFIX @task def init(ctx): """Install production dependencies""" {% if cookiecutter.dependency_management_tool == 'pipenv' -%} ctx.run("pipenv install --deploy") {%- elif cookiecutter.dependency_management_tool == 'poetry' -%} ctx.run("poetry install --no-dev") {%- endif %} {% if cookiecutter.dependency_management_tool == 'pipenv' -%} ctx.run("pipenv install --deploy") {%- elif cookiecutter.dependency_management_tool == 'poetry' -%} ctx.run("poetry install --no-dev") {%- endif %}
111. @clleew import os def remove_pipfile(): os.remove("Pipfile") def main(): if "{{ cookiecutter.dependency_management_tool }}" != "pipenv": remove_pipfile() if __name__ == "__main__": main() Run before/after project is generated hooks/post_gen_project.py
112. @clleew Lee-W / cookiecutter-python-template
113. @clleew
114. Other Tools • Testing • Hypothesis • nox • Documentation • Mkdocs • Sphinx
115. @clleew Related Talks • Dependency Management • Tzu-ping Chung - 這樣的開發環境沒問題嗎？ (PyCon TW 2018) • Kenneth Reitz - Pipenv: The Future of Python Dependency Management (PyCon US 2018) • Patrick Muehlbauer - Python Dependency Management (PyCon DE 2018) • Test • Chun-Yu Tseng - 快快樂樂成為 Coding Ninja (by pytest) (PyCon APAC 2015) • Florian Bruhin – Pytest: Rapid Simple Testing (Swiss Python Summit 16)
116. @clleew Related Talks • Style Check • Kyle Knapp - Automating Code Quality (PyCon US 2018) • Łukasz Langa - Life Is Better Painted Black, or: How to Stop Worrying and Embrace Auto-Formatting (PyCon US 2019) • Raymond Hettinger - Beyond PEP 8 -- Best practices for beautiful intelligible code (PyCon 2015) • Static Typing • Dustin Ingram - Static Typing in Python (PyCon US 2020) • Task Managements • Thea Flowers - Break the Cycle: Three excellent Python tools to automate repetitive tasks (PyCon US 2019) • Security • Terri Oda - Python Security Tools (PyCon US 2019) • Tennessee Leeuwenburg - Watch out for Safety Bandits! (PyCon AU 2018)
117. @clleew References • pipenv • poetry • flake8 • pylint • mypy • black • isort • invoke • pre-commit • commitizen-tools • safety • bandit • cookiecutter
118. @clleew Let's join PyCon TW 2021
119. @clleew