Share
Embed
Transcript
160 slides
-
@clleew Python Table Manners: Cut the Cookie Gracefully A Guideline
Toward Cleaner Code -
@clleew Outline • Clean up the table - Dependency Management
• Put the correct tablewares - Testing • Use the tablewares elegantly - Coding Style • Mnemonic phrase - Task management • Say please! - Check before git operation • Speak formally - Cultivate a git commit convention • Safety matters - Security Issues • Where's the Cookie? - Project template -
@clleew __name__ = 李唯 / Wei Lee __position__ = [
Software Engineer @ Rakuten Slice, 2021 Chairperson @ PyCon Taiwan, Maintainer of commitizen-tools, ] __twitter__ = @clleew __github__ = Lee-W __blog__ = http://lee-w.github.io $ cat speaker.py -
@clleew $ python speaker.py File "speaker.py", line 1 __name__ =
李唯 / Wei Lee ^ SyntaxError: invalid syntax $ python speaker.py -
@clleew Sometimes we just forgot to… • ... activate /
deactivate virtual environment • ... add package into requirements.txt -
@clleew Pipenv Python Dev Workflow for Humans • manage virtual
environment + package → update at once → No more manual sync up! • generate hashes from packages → ensure getting the same packages next time -
@clleew Empty Pipfile [[source]] name = "pypi" url = "https://pypi.org/simple"
verify_ssl = true [dev-packages] [packages] [requires] python_version = "3.7" -
@clleew Pipfile with “requests” [[source]] name = "pypi" url =
"https://pypi.org/simple" verify_ssl = true [dev-packages] [packages] requests = "*" [requires] python_version = "3.7" [packages] requests = "*" -
@clleew "requests": { "hashes": [ "sha256:11e007a8a2aa0323f5a921e9e6a2d7e4e67d9877e85773fba9ba6419025cbeb4", "sha256:9cf5292fcd0f598c671cfc1e0d7d1a7f13bb8085e9a590f48c010551dc6c4b31" ], "index": "pypi",
"version": "==2.22.0" }, { ...... "default": { ...... "requests": { "hashes": [ "sha256:11e007a8a2aa0323f5a921e9e6a2d7e4e67d9877e85773fba9ba6419025cbeb4", "sha256:9cf5292fcd0f598c671cfc1e0d7d1a7f13bb8085e9a590f48c010551dc6c4b31" ], "index": "pypi", "version": "==2.22.0" }, "urllib3": { "hashes": [ "sha256:3de946ffbed6e6746608990594d08faac602528ac7015ac28d33cee6a45b7398", "sha256:9a107b99a5393caf59c7aa3c1249c16e6879447533d0887f4336dde834c7be86" ], "version": "==1.25.6" } }, ...... } Pipfile.lock with “requests” -
@clleew "requests": { "hashes": [ "sha256:11e007a8a2aa0323f5a921e9e6a2d7e4e67d9877e85773fba9ba6419025cbeb4", "sha256:9cf5292fcd0f598c671cfc1e0d7d1a7f13bb8085e9a590f48c010551dc6c4b31" ], "index": "pypi",
"version": "==2.22.0" }, { ...... "default": { ...... "requests": { "hashes": [ "sha256:11e007a8a2aa0323f5a921e9e6a2d7e4e67d9877e85773fba9ba6419025cbeb4", "sha256:9cf5292fcd0f598c671cfc1e0d7d1a7f13bb8085e9a590f48c010551dc6c4b31" ], "index": "pypi", "version": "==2.22.0" }, "urllib3": { "hashes": [ "sha256:3de946ffbed6e6746608990594d08faac602528ac7015ac28d33cee6a45b7398", "sha256:9a107b99a5393caf59c7aa3c1249c16e6879447533d0887f4336dde834c7be86" ], "version": "==1.25.6" } }, ...... } Pipfile.lock with “requests” You’re guaranteed to install the exact same package next time. -
@clleew Pipfile with “pytest" [[source]] name = "pypi" url =
"https://pypi.org/simple" verify_ssl = true [dev-packages] pytest = "*" [packages] requests = "*" [requires] python_version = "3.7" [dev-packages] pytest = "*" -
@clleew Some people might say that pipenv ... • ...
does not update frequently. (which is not true recently) • ... locks slowly. • ... does not syncs up with install_requires in setup.py -
@clleew Why pytest • More Pythonic • Compatible with old
unittest tests • No more assert.+ (e.g., assertEqual, assertTrue, etc.) → just assert • Better test discovery • Advance features: mark, parameterize, etc. • Plenty of plugins -
@clleew unittest style tests import unittest from atta.partner import sponsor
class TestSponsor(unittest.TestCase): def setUp(self): sponsors = sponsor.get_all_sponsors('./data/packages.yaml', './data/sponsors.yaml') self.sponsors = sponsors ...... def test_sponsor_number(self): self.assertEqual(len(self.sponsors), 1) ....... -
@clleew unittest style tests import unittest from atta.partner import sponsor
class TestSponsor(unittest.TestCase): def setUp(self): sponsors = sponsor.get_all_sponsors('./data/packages.yaml', './data/sponsors.yaml') self.sponsors = sponsors ...... def test_sponsor_number(self): self.assertEqual(len(self.sponsors), 1) ....... Prepare all the data needed in test cases -
@clleew unittest style tests import unittest from atta.partner import sponsor
class TestSponsor(unittest.TestCase): def setUp(self): sponsors = sponsor.get_all_sponsors('./data/packages.yaml', './data/sponsors.yaml') self.sponsors = sponsors ...... def test_sponsor_number(self): self.assertEqual(len(self.sponsors), 1) ....... -
@clleew unittest style tests import unittest from atta.partner import sponsor
class TestSponsor(unittest.TestCase): def setUp(self): sponsors = sponsor.get_all_sponsors('./data/packages.yaml', './data/sponsors.yaml') self.sponsors = sponsors ...... def test_sponsor_number(self): self.assertEqual(len(self.sponsors), 1) ....... -
@clleew pytest style tests import pytest from report_generator.partner import sponsor
class TestSponsor: @pytest.fixture(scope="class") def sponsors(self): return sponsor.get_all_sponsors("test/data/packages.yaml", “test/data/sponsors.yaml") ...... def test_sponsor_number(self, sponsors): assert len(sponsors) == 1 ...... -
@clleew pytest style tests import pytest from report_generator.partner import sponsor
class TestSponsor: @pytest.fixture(scope="class") def sponsors(self): return sponsor.get_all_sponsors("test/data/packages.yaml", “test/data/sponsors.yaml") ...... def test_sponsor_number(self, sponsors): assert len(sponsors) == 1 ...... Prepare the data needed in separate fixtures -
@clleew pytest style tests import pytest from report_generator.partner import sponsor
class TestSponsor: @pytest.fixture(scope="class") def sponsors(self): return sponsor.get_all_sponsors("test/data/packages.yaml", “test/data/sponsors.yaml") ...... def test_sponsor_number(self, sponsors): assert len(sponsors) == 1 ...... -
@clleew pytest style tests import pytest from report_generator.partner import sponsor
@pytest.fixture(scope="function") def sponsors(): return sponsor.get_all_sponsors("test/data/packages.yaml", "test/data/sponsors.yaml") def test_sponsor_number(sponsors): assert len(sponsors) == 1 -
@clleew pytest style tests import pytest from report_generator.partner import sponsor
@pytest.fixture(scope="function") def sponsors(): return sponsor.get_all_sponsors("test/data/packages.yaml", "test/data/sponsors.yaml") def test_sponsor_number(sponsors): assert len(sponsors) == 1 -
@clleew Configuration - pyproject.toml [tool.pytest.ini_options] minversion = "6.0" testpaths =
"tests" addopts = "--strict-markers" norecursedirs = [ ".*", "build", "dist", "CVS", "_darcs", "{arch}", "*.egg", "venv", "env", "virtualenv" ] -
@clleew Pytest - powerful plugins • pytest-mock: Replace objects that
are hard to test with fake objects • pytest-cov: Generate test coverage report • pytest-xdist: Distributed testing • and etc. -
@clleew flake8 • Enforcing style consistency across Python project •
Check possible errors before running your program • Eliminate bad style -
@clleew import os os = "My Operating system" redefinition of
imported library (possible error) Example - bad_code.py -
@clleew import os os = "My Operating system" additional space
(bad coding style) redefinition of imported library (possible error) Example - bad_code.py -
@clleew Configuration - setup.cfg [flake8] ignore = # F632: use
==/!= to compare str, bytes, and int literals F632, # W503: Line break occurred before a binary operator W503, # E501: Line too long E501, # E203: Whitespace before ':' (for black) E203 exclude = .git, __pycache__, build, dist max-line-length = 88 -
@clleew import os os = "My Operating system" additional space
(bad coding style) redefinition of imported library (possible error) Example - bad_code.py -
@clleew Run pylint with "-r" argument A bunch of reports
that you can compare with your previous run -
Configuration - pyproject.toml [tool.pylint.messages_control] disable = [ "bad-continuation", "missing-function-docstring", "missing-module-docstring",
"invalid-name" ] [tool.pylint.format] max-line-length = 88 -
@clleew mypy • Static type checker • Compile-time type checking
→ avoid possible runtime errors • Type annotation enhances readability → machine-checked documentation -
from typing import List def func(vals: List[str]): for val in
vals: assert isinstance(val, str) func([1, 2, 3]) Example - error_type.py List[str] [1, 2, 3] -
from typing import List def func(vals: List[str]): for val in
vals: assert isinstance(val, str) func([1, 2, 3]) Example - error_type.py List[str] [1, 2, 3] -
@clleew Configuration - setup.cfg [mypy] files = **/*.py ignore_missing_imports =
true follow_imports = silent warn_redundant_casts = True warn_unused_ignores = True warn_unused_configs = True [mypy] files = **/*.py ignore_missing_imports = true -
@clleew Configuration - setup.cfg [mypy] files = **/*.py ignore_missing_imports =
true follow_imports = silent warn_redundant_casts = True warn_unused_ignores = True warn_unused_configs = True [mypy] files = **/*.py ignore_missing_imports = true -
Why Black? The black code style is not configurable. No
more arguing about which style is better -
–The Zen of Python, by Tim Peters “There should be
one-- and preferably only one --obvious way to do it.” -
@clleew Configuration - pyproject.toml [tool.black] line-length = 88 include =
'\.pyi?$' exclude = ''' /( \.eggs | \.git | \.hg | \.mypy_cache | \.tox | \.venv | _build | buck-out | build | dist )/ ''' -
According to PEP8 • Imports should be grouped in the
following order: 1. Standard library imports. 2. Related third party imports. 3. Local application/library specific imports. • You should put a blank line between each group of imports. -
Run Isort import datetime import os import django import flask
import some_custom_package ... -
Invoke • Task execution tool that can manage your commands
• It’s like a Makefile but in Python. -
How ? - tasks.py from invoke import task VENV_PREFIX =
"pipenv run" ...... @task def install(cmd): """Install script in pipenv environment""" cmd.run(f"{VENV_PREFIX} python setup.py install") -
How ? - tasks.py from invoke import task VENV_PREFIX =
"pipenv run" ...... @task def install(cmd): """Install script in pipenv environment""" cmd.run(f"{VENV_PREFIX} python setup.py install") -
How ? - tasks.py from invoke import task VENV_PREFIX =
"pipenv run" ...... @task def install(cmd): """Install script in pipenv environment""" cmd.run(f"{VENV_PREFIX} python setup.py install") -
Why not use Makefile? 1.We’re Python developers! 2.Some tasks might
not be easy to handle through shell script. 3.Combining Python and shell script -
@clleew pre-commit • Why? • Sometimes we still forget to
run the check commands. • Prevent committing bad code into codebase • How? • pre-commit runs these commands before we do git operations (e.g., git push, git commit) -
@clleew Config (customized hooks) .pre-commit-config.yaml repos: - repo: local hooks:
- id: style-reformat name: style-reformat stages: [commit] language: system pass_filenames: false entry: inv style.reformat types: [python] - id: style-check name: style-check stages: [push] language: system pass_filenames: false entry: inv style types: [python] ...... -
@clleew Config (customized hooks) .pre-commit-config.yaml repos: - repo: local hooks:
- id: style-reformat name: style-reformat stages: [commit] language: system pass_filenames: false entry: inv style.reformat types: [python] - id: style-check name: style-check stages: [push] language: system pass_filenames: false entry: inv style types: [python] ...... entry: inv style.reformat - id: style-reformat stages: [commit] -
@clleew Config (customized hooks) .pre-commit-config.yaml repos: - repo: local hooks:
- id: style-reformat name: style-reformat stages: [commit] language: system pass_filenames: false entry: inv style.reformat types: [python] - id: style-check name: style-check stages: [push] language: system pass_filenames: false entry: inv style types: [python] ...... stages: [push] - id: style-check -
@clleew Config (existing hooks) .pre-commit-config.yaml ...... - repo: https://github.com/pre-commit/pre-commit-hooks rev:
v3.2.0 hooks: - id: end-of-file-fixer - id: trailing-whitespace args: [--markdown-linebreak-ext=md] -
How does the commit look like? feat(Hualien-py): This is a
great meetup! Hope you enjoy my talk! https://www.meetup.com/Hualien-Py/events/ 273609065/ -
Other Commitizen features • pre-commit hook prevents you from not
using commitizen • customizable commit rules • auto bump project version (SemVar) • auto generate changelog (Keep a Changelog) -
How does Safety work? • Search the vulnerability in CVS
(Common Vulnerabilities and Exposures) • Free DB: update monthly • Paid DB: update realtime -
Not all the warnings should be fixed • Add exclude
into your bandit configuration • Add # nosec after the end of the code to skip the warning -
@clleew We want these manners in all our Python projects
❗ But... we don't want to configure it everytime ❗ -
@clleew An example in template {{cookiecutter.project_slug}}/tasks/env.py from invoke import task
from tasks.common import VENV_PREFIX @task def init(ctx): """Install production dependencies""" {% if cookiecutter.dependency_management_tool == 'pipenv' -%} ctx.run("pipenv install --deploy") {%- elif cookiecutter.dependency_management_tool == 'poetry' -%} ctx.run("poetry install --no-dev") {%- endif %} {% if cookiecutter.dependency_management_tool == 'pipenv' -%} ctx.run("pipenv install --deploy") -
@clleew An example in template {{cookiecutter.project_slug}}/tasks/env.py from invoke import task
from tasks.common import VENV_PREFIX @task def init(ctx): """Install production dependencies""" {% if cookiecutter.dependency_management_tool == 'pipenv' -%} ctx.run("pipenv install --deploy") {%- elif cookiecutter.dependency_management_tool == 'poetry' -%} ctx.run("poetry install --no-dev") {%- endif %} {% if cookiecutter.dependency_management_tool == 'pipenv' -%} ctx.run("pipenv install --deploy") -
@clleew An example in template {{cookiecutter.project_slug}}/tasks/env.py from invoke import task
from tasks.common import VENV_PREFIX @task def init(ctx): """Install production dependencies""" {% if cookiecutter.dependency_management_tool == 'pipenv' -%} ctx.run("pipenv install --deploy") {%- elif cookiecutter.dependency_management_tool == 'poetry' -%} ctx.run("poetry install --no-dev") {%- endif %} {%- elif cookiecutter.dependency_management_tool == 'poetry' -%} ctx.run("poetry install --no-dev") {%- endif %} -
@clleew import os def remove_pipfile(): os.remove("Pipfile") def main(): if "{{
cookiecutter.dependency_management_tool }}" != "pipenv": remove_pipfile() if __name__ == "__main__": main() Run before/after project is generated hooks/post_gen_project.py -
Other Tools • Testing • Hypothesis • nox • Documentation
• Mkdocs • Sphinx -
@clleew Related Talks • Dependency Management • Tzu-ping Chung -
這樣的開發環境沒問題嗎? (PyCon TW 2018) • Kenneth Reitz - Pipenv: The Future of Python Dependency Management (PyCon US 2018) • Patrick Muehlbauer - Python Dependency Management (PyCon DE 2018) • Test • Chun-Yu Tseng - 快快樂樂成為 Coding Ninja (by pytest) (PyCon APAC 2015) • Florian Bruhin – Pytest: Rapid Simple Testing (Swiss Python Summit 16) -
@clleew Related Talks • Style Check • Kyle Knapp -
Automating Code Quality (PyCon US 2018) • Łukasz Langa - Life Is Better Painted Black, or: How to Stop Worrying and Embrace Auto-Formatting (PyCon US 2019) • Raymond Hettinger - Beyond PEP 8 -- Best practices for beautiful intelligible code (PyCon 2015) • Static Typing • Dustin Ingram - Static Typing in Python (PyCon US 2020) • Task Managements • Thea Flowers - Break the Cycle: Three excellent Python tools to automate repetitive tasks (PyCon US 2019) • Security • Terri Oda - Python Security Tools (PyCon US 2019) • Tennessee Leeuwenburg - Watch out for Safety Bandits! (PyCon AU 2018) -
@clleew References • pipenv • poetry • flake8 • pylint
• mypy • black • isort • invoke • pre-commit • commitizen-tools • safety • bandit • cookiecutter