Python Table Manners @ Taichung.py

Slide 1

Slide 1 text

@clleew Python Table Manners Everything you need in software development life cycle

Slide 2

Slide 2 text

@clleew Slide

Slide 3

Slide 3 text

@clleew Outline • Clean up the table - Dependency Management • Put the correct tablewares - Testing • Use the tablewares elegantly - Coding Style • Mnemonic phrase - Task management • Say please! - Check before git operation • Speak formally - Cultivate a git commit convention • Safety matters - Security Issues • Where's the Cookie? - Project template

Slide 4

Slide 4 text

@clleew __name__ = 李唯 / Wei Lee __position__ = [ Software Engineer @ Rakuten Slice, 2021 Chairperson @ PyCon Taiwan, Maintainer of commitizen-tools, ] __twitter__ = @clleew __github__ = Lee-W __blog__ = http://lee-w.github.io $ cat speaker.py

Slide 5

Slide 5 text

@clleew $ python speaker.py File "speaker.py", line 1 __name__ = 李唯 / Wei Lee ^ SyntaxError: invalid syntax $ python speaker.py

Slide 6

Slide 6 text

@clleew Let's join PyCon TW 2021

Slide 7

Slide 7 text

@clleew Dependency Management Clean up the table

Slide 8

Slide 8 text

@clleew How we used to start a Python project

Slide 9

Slide 9 text

@clleew Sometimes we just forgot to… • ... activate / deactivate virtual environment • ... add package into requirements.txt

Slide 10

Slide 10 text

@clleew Pipenv Python Dev Workﬂow for Humans • manage virtual environment + package → update at once → No more manual sync up! • generate hashes from packages → ensure getting the same packages next time

Slide 11

Slide 11 text

@clleew Initial Virtual Environment

Slide 12

Slide 12 text

@clleew Empty Pipfile [[source]] name = "pypi" url = "https://pypi.org/simple" verify_ssl = true [dev-packages] [packages] [requires] python_version = "3.7"

Slide 13

Slide 13 text

@clleew Install Package

Slide 14

Slide 14 text

@clleew Pipfile with “requests” [[source]] name = "pypi" url = "https://pypi.org/simple" verify_ssl = true [dev-packages] [packages] requests = "*" [requires] python_version = "3.7" [packages] requests = "*"

Slide 15

Slide 15 text

@clleew "requests": { "hashes": [ "sha256:11e007a8a2aa0323f5a921e9e6a2d7e4e67d9877e85773fba9ba6419025cbeb4", "sha256:9cf5292fcd0f598c671cfc1e0d7d1a7f13bb8085e9a590f48c010551dc6c4b31" ], "index": "pypi", "version": "==2.22.0" }, { ...... "default": { ...... "requests": { "hashes": [ "sha256:11e007a8a2aa0323f5a921e9e6a2d7e4e67d9877e85773fba9ba6419025cbeb4", "sha256:9cf5292fcd0f598c671cfc1e0d7d1a7f13bb8085e9a590f48c010551dc6c4b31" ], "index": "pypi", "version": "==2.22.0" }, "urllib3": { "hashes": [ "sha256:3de946ffbed6e6746608990594d08faac602528ac7015ac28d33cee6a45b7398", "sha256:9a107b99a5393caf59c7aa3c1249c16e6879447533d0887f4336dde834c7be86" ], "version": "==1.25.6" } }, ...... } Pipfile.lock with “requests” You’re guaranteed to install the exact same package next time.

Slide 16

Slide 16 text

@clleew

Slide 17

Slide 17 text

@clleew Install package only in dev

Slide 18

Slide 18 text

@clleew Pipﬁle with “pytest" [[source]] name = "pypi" url = "https://pypi.org/simple" verify_ssl = true [dev-packages] pytest = "*" [packages] requests = "*" [requires] python_version = "3.7" [dev-packages] pytest = "*"

Slide 19

Slide 19 text

@clleew Run in virtual environment

Slide 20

Slide 20 text

@clleew Some people might say that pipenv ... • ... does not update frequently. • ... locks slowly. • ... does not syncs up with install_requires in setup.py Or maybe you can manage dependencies in setup.py, and use “pipenv install -e .” (learned from dboy )

Slide 21

Slide 21 text

@clleew Poetry

Slide 22

Slide 22 text

@clleew Poetry commands

Slide 23

Slide 23 text

@clleew Testing Put the correct tablewares

Slide 24

Slide 24 text

@clleew unittest Unit testing framework in Python Standard Library

Slide 25

Slide 25 text

@clleew pytest

Slide 26

Slide 26 text

@clleew Why pytest • More Pythonic • Compatible with old unittest tests • No more assert.+ (e.g., assertEqual, assertTrue, etc.)  → just assert • Better test discovery • Advance features: mark, parameterize, etc. • Plenty of plugins

Slide 27

Slide 27 text

@clleew Run pytest

Slide 28

Slide 28 text

@clleew always run python inside a virtual environment

Slide 29

Slide 29 text

@clleew unittest style tests import unittest from atta.partner import sponsor class TestSponsor(unittest.TestCase): def setUp(self): sponsors = sponsor.get_all_sponsors('./data/packages.yaml', './data/sponsors.yaml') self.sponsors = sponsors ...... def test_sponsor_number(self): self.assertEqual(len(self.sponsors), 1) ....... Prepare all the data needed in test cases

Slide 30

Slide 30 text

@clleew pytest style tests import pytest from report_generator.partner import sponsor class TestSponsor: @pytest.fixture(scope="class") def sponsors(self): return sponsor.get_all_sponsors("test/data/packages.yaml", “test/data/sponsors.yaml") ...... def test_sponsor_number(self, sponsors): assert len(sponsors) == 1 ...... Prepare the data needed in separate ﬁxtures

Slide 31

Slide 31 text

@clleew pytest style tests import pytest from report_generator.partner import sponsor @pytest.fixture(scope="function") def sponsors(): return sponsor.get_all_sponsors("test/data/packages.yaml", "test/data/sponsors.yaml") def test_sponsor_number(sponsors): assert len(sponsors) == 1

Slide 32

Slide 32 text

@clleew Conﬁguration - pyproject.toml [tool.pytest.ini_options] minversion = "6.0" testpaths = "tests" addopts = "--strict-markers" norecursedirs = [ ".*", "build", "dist", "CVS", "_darcs", "{arch}", "*.egg", "venv", "env", "virtualenv" ]

Slide 33

Slide 33 text

@clleew Pytest - powerful plugins • pytest-mock: Replace objects that are hard to test with fake objects • pytest-cov: Generate test coverage report • pytest-xdist: Distributed testing • pytest-regressions • and etc.

Slide 34

Slide 34 text

@clleew Coding Style Use the tablewares elegantly

Slide 35

Slide 35 text

@clleew ﬂake8 • Enforcing style consistency across Python project • Check possible errors before running your program • Eliminate bad style

Slide 36

Slide 36 text

@clleew import os os = "My Operating system" additional space  (bad coding style) redeﬁnition of imported library  (possible error) Example - bad_code.py

Slide 37

Slide 37 text

@clleew Run ﬂake8

Slide 38

Slide 38 text

@clleew Conﬁguration - setup.cfg [flake8] ignore = # F632: use ==/!= to compare str, bytes, and int literals F632, # W503: Line break occurred before a binary operator W503, # E501: Line too long E501, # E203: Whitespace before ':' (for black) E203 exclude = .git, __pycache__, build, dist max-line-length = 88

Slide 39

Slide 39 text

@clleew Pylint

Slide 40

Slide 40 text

@clleew import os os = "My Operating system" additional space  (bad coding style) redeﬁnition of imported library  (possible error) Example - bad_code.py

Slide 41

Slide 41 text

@clleew Run pylint

Slide 42

Slide 42 text

@clleew Run pylint with "-r" argument A bunch of reports that   you can compare   with your previous run

Slide 43

Slide 43 text

Conﬁguration - pyproject.toml [tool.pylint.messages_control] disable = [ "bad-continuation", "missing-function-docstring", "missing-module-docstring", "invalid-name" ] [tool.pylint.format] max-line-length = 88

Slide 44

Slide 44 text

@clleew mypy • Static type checker • Compile-time type checking  → avoid possible runtime errors • Type annotation enhances readability  → machine-checked documentation

Slide 45

Slide 45 text

from typing import List def func(vals: List[str]): for val in vals: assert isinstance(val, str) func([1, 2, 3]) Example - error_type.py List[str] [1, 2, 3]

Slide 46

Slide 46 text

Run mypy all the ﬁles with .py extension ignore errors that libs that is not type annotated

Slide 47

Slide 47 text

@clleew Conﬁguration - setup.cfg [mypy] files = **/*.py ignore_missing_imports = true follow_imports = silent warn_redundant_casts = True warn_unused_ignores = True warn_unused_configs = True [mypy] files = **/*.py ignore_missing_imports = true

Slide 48

Slide 48 text

Run mypy with conﬁguration

Slide 49

Slide 49 text

@clleew One step forward Fix the style automatically

Slide 50

Slide 50 text

Black

Slide 51

Slide 51 text

Run Black

Slide 52

Slide 52 text

How Black reformat

Slide 53

Slide 53 text

How Black reformat

Slide 54

Slide 54 text

Why Black? The black code style is not conﬁgurable. No more arguing about which style is better

Slide 55

Slide 55 text

–The Zen of Python, by Tim Peters “There should be one-- and preferably only one --obvious way to do it.”

Slide 56

Slide 56 text

Slide 57

Slide 57 text

Isort

Slide 58

Slide 58 text

Example wrong_import_order.py import django import os import some_custom_package import flask import datetime ...

Slide 59

Slide 59 text

According to PEP8 • Imports should be grouped in the following order: 1. Standard library imports. 2. Related third party imports. 3. Local application/library speciﬁc imports. • You should put a blank line between each group of imports.

Slide 60

Slide 60 text

Run Isort

Slide 61

Slide 61 text

Run Isort import datetime import os import django import flask import some_custom_package ...

Slide 62

Slide 62 text

@clleew Conﬁguration - pyproject.toml [tool.isort] profile = "black"

Slide 63

Slide 63 text

Style formatting and linting

Slide 64

Slide 64 text

No content

Slide 65

Slide 65 text

@clleew Task Management Mnemonic phrase

Slide 66

Slide 66 text

Invoke • Task execution tool that can manage your commands • It’s like a Makeﬁle but in Python.

Slide 67

Slide 67 text

Invoke in practice (rg-cli)

Slide 68

Slide 68 text

Before invoke in rg-cli

Slide 69

Slide 69 text

After invoke in rg-cli

Slide 70

Slide 70 text

List all commands

Slide 71

Slide 71 text

How ? - tasks.py from invoke import task VENV_PREFIX = "pipenv run" ...... @task def install(cmd): """Install script in pipenv environment""" cmd.run(f"{VENV_PREFIX} python setup.py install")

Slide 72

Slide 72 text

Modulize - Namespace

Slide 73

Slide 73 text

Autocomplete

Slide 74

Slide 74 text

Why not use Makeﬁle? 1.We’re Python developers! 2.Some tasks might not be easy to handle through shell script. 3.Combining Python and shell script

Slide 75

Slide 75 text

@clleew Check before git operation Say please!

Slide 76

Slide 76 text

@clleew pre-commit • Why? • Sometimes we still forget to run the check commands. • Prevent committing bad code into codebase • How? • pre-commit runs these commands before we do git operations (e.g., git push, git commit)

Slide 77

Slide 77 text

@clleew Conﬁg (customized hooks) .pre-commit-config.yaml repos: - repo: local hooks: - id: style-reformat name: style-reformat stages: [commit] language: system pass_filenames: false entry: inv style.reformat types: [python] - id: style-check name: style-check stages: [push] language: system pass_filenames: false entry: inv style types: [python] ...... stages: [push] entry: inv style.reformat - id: style-reformat stages: [commit] - id: style-check

Slide 78

Slide 78 text

@clleew Conﬁg (existing hooks) .pre-commit-config.yaml ...... - repo: https://github.com/pre-commit/pre-commit-hooks rev: v3.2.0 hooks: - id: end-of-file-fixer - id: trailing-whitespace args: [--markdown-linebreak-ext=md]

Slide 79

Slide 79 text

@clleew Setup pre-commit

Slide 80

Slide 80 text

@clleew Run pre-commit

Slide 81

Slide 81 text

@clleew Run pre-commit

Slide 82

Slide 82 text

@clleew Cultivate a git commit convention Speak formally

Slide 83

Slide 83 text

No content

Slide 84

Slide 84 text

Hard to ﬁnd to the right version

Slide 85

Slide 85 text

commitizen-tools

Slide 86

Slide 86 text

Commit - cz commit

Slide 87

Slide 87 text

Commit - cz commit

Slide 88

Slide 88 text

How does the commit look like? feat(Taichung.py): This is a great meetup! Hope you enjoy my talk! https://taichung-py.kktix.cc/events/ meetup-202011-clleew

Slide 89

Slide 89 text

Standardized git commits

Slide 90

Slide 90 text

Other Commitizen features • pre-commit hook prevents you from not using commitizen • customizable commit rules • auto bump project version (SemVar) • auto generate changelog (Keep a Changelog)

Slide 91

Slide 91 text

@clleew Security Issues Safety matters

Slide 92

Slide 92 text

No content

Slide 93

Slide 93 text

No content

Slide 94

Slide 94 text

Safety Dependency Checking

Slide 95

Slide 95 text

Check vulnerability in current environment

Slide 96

Slide 96 text

But if you use pipenv …

Slide 97

Slide 97 text

How does Safety work? • Search the vulnerability in CVS (Common Vulnerabilities and Exposures) • Free DB: update monthly • Paid DB: update realtime

Slide 98

Slide 98 text

Bandit Static Analysis

Slide 99

Slide 99 text

Check Common Security Issue in Python Code

Slide 100

Slide 100 text

Not all the warnings should be ﬁxed • Add exclude into your bandit conﬁguration • Add # nosec after the end of the code to skip the warning

Slide 101

Slide 101 text

@clleew

Slide 102

Slide 102 text

@clleew Project template Where's the Cookie?

Slide 103

Slide 103 text

@clleew We want these manners in all our Python projects ❗ But... we don't want to conﬁgure it everytime ❗

Slide 104

Slide 104 text

@clleew Create a project template once and initialize projects through it

Slide 105

Slide 105 text

@clleew Lee-W / cookiecutter-python-template

Slide 106

Slide 106 text

@clleew

Slide 107

Slide 107 text

@clleew

Slide 108

Slide 108 text

@clleew How to make a template? cookiecutter.json

Slide 109

Slide 109 text

@clleew Template structure Cookiecutter conﬁguration Template (Use Jinja)

Slide 110

Slide 110 text

@clleew An example in template {{cookiecutter.project_slug}}/tasks/env.py from invoke import task from tasks.common import VENV_PREFIX @task def init(ctx): """Install production dependencies""" {% if cookiecutter.dependency_management_tool == 'pipenv' -%} ctx.run("pipenv install --deploy") {%- elif cookiecutter.dependency_management_tool == 'poetry' -%} ctx.run("poetry install --no-dev") {%- endif %} {% if cookiecutter.dependency_management_tool == 'pipenv' -%} ctx.run("pipenv install --deploy") {%- elif cookiecutter.dependency_management_tool == 'poetry' -%} ctx.run("poetry install --no-dev") {%- endif %}

Slide 111

Slide 111 text

@clleew import os def remove_pipfile(): os.remove("Pipfile") def main(): if "{{ cookiecutter.dependency_management_tool }}" != "pipenv": remove_pipfile() if __name__ == "__main__": main() Run before/after project is generated hooks/post_gen_project.py

Slide 112

Slide 112 text

@clleew Lee-W / cookiecutter-python-template

Slide 113

Slide 113 text

@clleew

Slide 114

Slide 114 text

Other Tools • Testing • Hypothesis • nox • Documentation • Mkdocs • Sphinx

Slide 115

Slide 115 text

@clleew Related Talks • Dependency Management • Tzu-ping Chung - 這樣的開發環境沒問題嗎？ (PyCon TW 2018) • Kenneth Reitz - Pipenv: The Future of Python Dependency Management (PyCon US 2018) • Patrick Muehlbauer - Python Dependency Management (PyCon DE 2018) • Test • Chun-Yu Tseng - 快快樂樂成為 Coding Ninja (by pytest) (PyCon APAC 2015) • Florian Bruhin – Pytest: Rapid Simple Testing (Swiss Python Summit 16)

Slide 116

Slide 116 text

@clleew Related Talks • Style Check • Kyle Knapp - Automating Code Quality (PyCon US 2018) • Łukasz Langa - Life Is Better Painted Black, or: How to Stop Worrying and Embrace Auto-Formatting (PyCon US 2019) • Raymond Hettinger - Beyond PEP 8 -- Best practices for beautiful intelligible code (PyCon 2015) • Static Typing • Dustin Ingram - Static Typing in Python (PyCon US 2020) • Task Managements • Thea Flowers - Break the Cycle: Three excellent Python tools to automate repetitive tasks (PyCon US 2019) • Security • Terri Oda - Python Security Tools (PyCon US 2019) • Tennessee Leeuwenburg - Watch out for Safety Bandits! (PyCon AU 2018)

Slide 117

Slide 117 text

@clleew References • pipenv • poetry • ﬂake8 • pylint • mypy • black • isort • invoke • pre-commit • commitizen-tools • safety • bandit • cookiecutter