Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Python Table Manners- Cut the Cookie Gracefully @ Hualien py

3e049ed33195d00a8b745b16c63dce6e?s=47 Lee Wei
October 16, 2020

Python Table Manners- Cut the Cookie Gracefully @ Hualien py

3e049ed33195d00a8b745b16c63dce6e?s=128

Lee Wei

October 16, 2020
Tweet

Transcript

  1. @clleew Python Table Manners: Cut the Cookie Gracefully A Guideline

    Toward Cleaner Code
  2. @clleew Slide

  3. @clleew Outline • Clean up the table - Dependency Management

    • Put the correct tablewares - Testing • Use the tablewares elegantly - Coding Style • Mnemonic phrase - Task management • Say please! - Check before git operation • Speak formally - Cultivate a git commit convention • Safety matters - Security Issues • Where's the Cookie? - Project template
  4. @clleew __name__ = 李唯 / Wei Lee __position__ = [

    Software Engineer @ Rakuten Slice, 2021 Chairperson @ PyCon Taiwan, Maintainer of commitizen-tools, ] __twitter__ = @clleew __github__ = Lee-W __blog__ = http://lee-w.github.io $ cat speaker.py
  5. @clleew $ python speaker.py File "speaker.py", line 1 __name__ =

    李唯 / Wei Lee ^ SyntaxError: invalid syntax $ python speaker.py
  6. @clleew Let's volunteering in PyCon TW 2021

  7. @clleew Dependency Management Clean up the table

  8. @clleew How we used to start a Python project

  9. @clleew Sometimes we just forgot to… • ... activate /

    deactivate virtual environment • ... add package into requirements.txt
  10. @clleew Pipenv
 Python Dev Workflow for Humans • manage virtual

    environment + package
 → update at once
 → No more manual sync up! • generate hashes from packages
 → ensure getting the same packages next time
  11. @clleew Initial Virtual Environment

  12. @clleew Empty Pipfile [[source]] name = "pypi" url = "https://pypi.org/simple"

    verify_ssl = true [dev-packages] [packages] [requires] python_version = "3.7"
  13. @clleew Install Package

  14. @clleew Pipfile with “requests” [[source]] name = "pypi" url =

    "https://pypi.org/simple" verify_ssl = true [dev-packages] [packages] requests = "*" [requires] python_version = "3.7" [packages] requests = "*"
  15. @clleew "requests": { "hashes": [ "sha256:11e007a8a2aa0323f5a921e9e6a2d7e4e67d9877e85773fba9ba6419025cbeb4", "sha256:9cf5292fcd0f598c671cfc1e0d7d1a7f13bb8085e9a590f48c010551dc6c4b31" ], "index": "pypi",

    "version": "==2.22.0" }, { ...... "default": { ...... "requests": { "hashes": [ "sha256:11e007a8a2aa0323f5a921e9e6a2d7e4e67d9877e85773fba9ba6419025cbeb4", "sha256:9cf5292fcd0f598c671cfc1e0d7d1a7f13bb8085e9a590f48c010551dc6c4b31" ], "index": "pypi", "version": "==2.22.0" }, "urllib3": { "hashes": [ "sha256:3de946ffbed6e6746608990594d08faac602528ac7015ac28d33cee6a45b7398", "sha256:9a107b99a5393caf59c7aa3c1249c16e6879447533d0887f4336dde834c7be86" ], "version": "==1.25.6" } }, ...... } Pipfile.lock with “requests”
  16. @clleew "requests": { "hashes": [ "sha256:11e007a8a2aa0323f5a921e9e6a2d7e4e67d9877e85773fba9ba6419025cbeb4", "sha256:9cf5292fcd0f598c671cfc1e0d7d1a7f13bb8085e9a590f48c010551dc6c4b31" ], "index": "pypi",

    "version": "==2.22.0" }, { ...... "default": { ...... "requests": { "hashes": [ "sha256:11e007a8a2aa0323f5a921e9e6a2d7e4e67d9877e85773fba9ba6419025cbeb4", "sha256:9cf5292fcd0f598c671cfc1e0d7d1a7f13bb8085e9a590f48c010551dc6c4b31" ], "index": "pypi", "version": "==2.22.0" }, "urllib3": { "hashes": [ "sha256:3de946ffbed6e6746608990594d08faac602528ac7015ac28d33cee6a45b7398", "sha256:9a107b99a5393caf59c7aa3c1249c16e6879447533d0887f4336dde834c7be86" ], "version": "==1.25.6" } }, ...... } Pipfile.lock with “requests” You’re guaranteed to install the exact same package next time.
  17. @clleew

  18. @clleew Install package only in dev

  19. @clleew Pipfile with “pytest" [[source]] name = "pypi" url =

    "https://pypi.org/simple" verify_ssl = true [dev-packages] pytest = "*" [packages] requests = "*" [requires] python_version = "3.7" [dev-packages] pytest = "*"
  20. @clleew Run in virtual environment

  21. @clleew Some people might say that pipenv ... • ...

    does not update frequently. (which is not true recently) • ... locks slowly. • ... does not syncs up with install_requires in setup.py
  22. @clleew Poetry

  23. @clleew Poetry commands

  24. @clleew Testing Put the correct tablewares

  25. @clleew unittest 
 Unit testing framework in Python Standard Library

  26. @clleew pytest

  27. @clleew Why pytest • More Pythonic • Compatible with old

    unittest tests • No more assert.+ (e.g., assertEqual, assertTrue, etc.)
 → just assert • Better test discovery • Advance features: mark, parameterize, etc. • Plenty of plugins
  28. @clleew Run pytest

  29. @clleew Run pytest

  30. @clleew always run python inside a virtual environment

  31. @clleew unittest style tests import unittest from atta.partner import sponsor

    class TestSponsor(unittest.TestCase): def setUp(self): sponsors = sponsor.get_all_sponsors('./data/packages.yaml', './data/sponsors.yaml') self.sponsors = sponsors ...... def test_sponsor_number(self): self.assertEqual(len(self.sponsors), 1) .......
  32. @clleew unittest style tests import unittest from atta.partner import sponsor

    class TestSponsor(unittest.TestCase): def setUp(self): sponsors = sponsor.get_all_sponsors('./data/packages.yaml', './data/sponsors.yaml') self.sponsors = sponsors ...... def test_sponsor_number(self): self.assertEqual(len(self.sponsors), 1) ....... Prepare all the data needed in test cases
  33. @clleew unittest style tests import unittest from atta.partner import sponsor

    class TestSponsor(unittest.TestCase): def setUp(self): sponsors = sponsor.get_all_sponsors('./data/packages.yaml', './data/sponsors.yaml') self.sponsors = sponsors ...... def test_sponsor_number(self): self.assertEqual(len(self.sponsors), 1) .......
  34. @clleew unittest style tests import unittest from atta.partner import sponsor

    class TestSponsor(unittest.TestCase): def setUp(self): sponsors = sponsor.get_all_sponsors('./data/packages.yaml', './data/sponsors.yaml') self.sponsors = sponsors ...... def test_sponsor_number(self): self.assertEqual(len(self.sponsors), 1) .......
  35. @clleew pytest style tests import pytest from report_generator.partner import sponsor

    class TestSponsor: @pytest.fixture(scope="class") def sponsors(self): return sponsor.get_all_sponsors("test/data/packages.yaml", “test/data/sponsors.yaml") ...... def test_sponsor_number(self, sponsors): assert len(sponsors) == 1 ......
  36. @clleew pytest style tests import pytest from report_generator.partner import sponsor

    class TestSponsor: @pytest.fixture(scope="class") def sponsors(self): return sponsor.get_all_sponsors("test/data/packages.yaml", “test/data/sponsors.yaml") ...... def test_sponsor_number(self, sponsors): assert len(sponsors) == 1 ...... Prepare the data needed in separate fixtures
  37. @clleew pytest style tests import pytest from report_generator.partner import sponsor

    class TestSponsor: @pytest.fixture(scope="class") def sponsors(self): return sponsor.get_all_sponsors("test/data/packages.yaml", “test/data/sponsors.yaml") ...... def test_sponsor_number(self, sponsors): assert len(sponsors) == 1 ......
  38. @clleew pytest style tests import pytest from report_generator.partner import sponsor

    @pytest.fixture(scope="function") def sponsors(): return sponsor.get_all_sponsors("test/data/packages.yaml", "test/data/sponsors.yaml") def test_sponsor_number(sponsors): assert len(sponsors) == 1
  39. @clleew pytest style tests import pytest from report_generator.partner import sponsor

    @pytest.fixture(scope="function") def sponsors(): return sponsor.get_all_sponsors("test/data/packages.yaml", "test/data/sponsors.yaml") def test_sponsor_number(sponsors): assert len(sponsors) == 1
  40. @clleew Configuration - pyproject.toml [tool.pytest.ini_options] minversion = "6.0" testpaths =

    "tests" addopts = "--strict-markers" norecursedirs = [ ".*", "build", "dist", "CVS", "_darcs", "{arch}", "*.egg", "venv", "env", "virtualenv" ]
  41. @clleew Pytest - powerful plugins • pytest-mock: Replace objects that

    are hard to test with fake objects • pytest-cov: Generate test coverage report • pytest-xdist: Distributed testing • and etc.
  42. @clleew Coding Style Use the tablewares elegantly

  43. @clleew flake8 • Enforcing style consistency across Python project •

    Check possible errors before running your program • Eliminate bad style
  44. @clleew import os os = "My Operating system" Example -

    bad_code.py
  45. @clleew import os os = "My Operating system" redefinition of

    imported library
 (possible error) Example - bad_code.py
  46. @clleew import os os = "My Operating system" additional space


    (bad coding style) redefinition of imported library
 (possible error) Example - bad_code.py
  47. @clleew Run flake8

  48. @clleew Configuration - setup.cfg [flake8] ignore = # F632: use

    ==/!= to compare str, bytes, and int literals F632, # W503: Line break occurred before a binary operator W503, # E501: Line too long E501, # E203: Whitespace before ':' (for black) E203 exclude = .git, __pycache__, build, dist max-line-length = 88
  49. @clleew Pylint

  50. @clleew import os os = "My Operating system" additional space


    (bad coding style) redefinition of imported library
 (possible error) Example - bad_code.py
  51. @clleew Run pylint

  52. @clleew Run pylint with "-r" argument A bunch of reports

    that 
 you can compare 
 with your previous run
  53. Configuration - pyproject.toml [tool.pylint.messages_control] disable = [ "bad-continuation", "missing-function-docstring", "missing-module-docstring",

    "invalid-name" ] [tool.pylint.format] max-line-length = 88
  54. @clleew mypy • Static type checker • Compile-time type checking


    → avoid possible runtime errors • Type annotation enhances readability
 → machine-checked documentation
  55. from typing import List def func(vals: List[str]): for val in

    vals: assert isinstance(val, str) func([1, 2, 3]) Example - error_type.py List[str] [1, 2, 3]
  56. from typing import List def func(vals: List[str]): for val in

    vals: assert isinstance(val, str) func([1, 2, 3]) Example - error_type.py List[str] [1, 2, 3]
  57. Run mypy

  58. Run mypy all the files with .py extension

  59. Run mypy ignore errors that libs that is not type

    annotated
  60. @clleew Configuration - setup.cfg [mypy] files = **/*.py ignore_missing_imports =

    true follow_imports = silent warn_redundant_casts = True warn_unused_ignores = True warn_unused_configs = True [mypy] files = **/*.py ignore_missing_imports = true
  61. @clleew Configuration - setup.cfg [mypy] files = **/*.py ignore_missing_imports =

    true follow_imports = silent warn_redundant_casts = True warn_unused_ignores = True warn_unused_configs = True [mypy] files = **/*.py ignore_missing_imports = true
  62. Run mypy with configuration

  63. @clleew One step forward Fix the style automatically

  64. Black

  65. Run Black

  66. How Black reformat

  67. How Black reformat

  68. Why Black? The black code style is not configurable.

  69. Why Black? The black code style is not configurable. No

    more arguing about which style is better
  70. –The Zen of Python, by Tim Peters “There should be

    one-- and preferably only one --obvious way to do it.”
  71. @clleew Configuration - pyproject.toml [tool.black] line-length = 88 include =

    '\.pyi?$' exclude = ''' /( \.eggs | \.git | \.hg | \.mypy_cache | \.tox | \.venv | _build | buck-out | build | dist )/ '''
  72. Isort

  73. Example
 wrong_import_order.py import django import os import some_custom_package import flask

    import datetime ...
  74. According to PEP8 • Imports should be grouped in the

    following order: 1. Standard library imports. 2. Related third party imports. 3. Local application/library specific imports. • You should put a blank line between each group of imports.
  75. Run Isort

  76. Run Isort import datetime import os import django import flask

    import some_custom_package ...
  77. @clleew Configuration - pyproject.toml [tool.isort] profile = "black"

  78. Style formatting and linting

  79. None
  80. @clleew Task Management Mnemonic phrase

  81. Invoke • Task execution tool that can manage your commands

    • It’s like a Makefile but in Python.
  82. Invoke in practice (rg-cli)

  83. Before invoke in rg-cli

  84. Before invoke in rg-cli

  85. After invoke in rg-cli

  86. List all commands

  87. How ? - tasks.py from invoke import task VENV_PREFIX =

    "pipenv run" ...... @task def install(cmd): """Install script in pipenv environment""" cmd.run(f"{VENV_PREFIX} python setup.py install")
  88. How ? - tasks.py from invoke import task VENV_PREFIX =

    "pipenv run" ...... @task def install(cmd): """Install script in pipenv environment""" cmd.run(f"{VENV_PREFIX} python setup.py install")
  89. How ? - tasks.py from invoke import task VENV_PREFIX =

    "pipenv run" ...... @task def install(cmd): """Install script in pipenv environment""" cmd.run(f"{VENV_PREFIX} python setup.py install")
  90. Modulize - Namespace

  91. Autocomplete

  92. Autocomplete

  93. Autocomplete

  94. Why not use Makefile? 1.We’re Python developers! 2.Some tasks might

    not be easy to handle through shell script. 3.Combining Python and shell script

  95. @clleew Check before git operation Say please!

  96. @clleew pre-commit • Why? • Sometimes we still forget to

    run the check commands. • Prevent committing bad code into codebase • How? • pre-commit runs these commands before we do git operations (e.g., git push, git commit)
  97. @clleew Config (customized hooks)
 .pre-commit-config.yaml repos: - repo: local hooks:

    - id: style-reformat name: style-reformat stages: [commit] language: system pass_filenames: false entry: inv style.reformat types: [python] - id: style-check name: style-check stages: [push] language: system pass_filenames: false entry: inv style types: [python] ......
  98. @clleew Config (customized hooks)
 .pre-commit-config.yaml repos: - repo: local hooks:

    - id: style-reformat name: style-reformat stages: [commit] language: system pass_filenames: false entry: inv style.reformat types: [python] - id: style-check name: style-check stages: [push] language: system pass_filenames: false entry: inv style types: [python] ...... entry: inv style.reformat - id: style-reformat stages: [commit]
  99. @clleew Config (customized hooks)
 .pre-commit-config.yaml repos: - repo: local hooks:

    - id: style-reformat name: style-reformat stages: [commit] language: system pass_filenames: false entry: inv style.reformat types: [python] - id: style-check name: style-check stages: [push] language: system pass_filenames: false entry: inv style types: [python] ...... stages: [push] - id: style-check
  100. @clleew Config (existing hooks) .pre-commit-config.yaml ...... - repo: https://github.com/pre-commit/pre-commit-hooks rev:

    v3.2.0 hooks: - id: end-of-file-fixer - id: trailing-whitespace args: [--markdown-linebreak-ext=md]
  101. @clleew Setup pre-commit

  102. @clleew Run pre-commit

  103. @clleew Run pre-commit

  104. @clleew Cultivate a git commit convention Speak formally

  105. None
  106. Hard to find to the right version

  107. commitizen-tools

  108. Commit - cz commit

  109. Commit - cz commit

  110. Commit - cz commit

  111. Commit - cz commit

  112. Commit - cz commit

  113. Commit - cz commit

  114. Commit - cz commit

  115. Commit - cz commit

  116. How does the commit look like? feat(Hualien-py): This is a

    great meetup! Hope you enjoy my talk! https://www.meetup.com/Hualien-Py/events/ 273609065/
  117. Standardized git commits

  118. Other Commitizen features • pre-commit hook prevents you from not

    using commitizen • customizable commit rules • auto bump project version (SemVar) • auto generate changelog (Keep a Changelog)
  119. @clleew Security Issues Safety matters

  120. None
  121. None
  122. Safety Dependency Checking

  123. Check vulnerability in current environment

  124. Check vulnerability in current environment

  125. But if you use pipenv …

  126. But if you use pipenv …

  127. How does Safety work? • Search the vulnerability in CVS

    
 (Common Vulnerabilities and Exposures) • Free DB: update monthly • Paid DB: update realtime
  128. Bandit Static Analysis

  129. Check Common Security Issue in Python Code

  130. Check Common Security Issue in Python Code

  131. Check Common Security Issue in Python Code

  132. Check Common Security Issue in Python Code

  133. Check Common Security Issue in Python Code

  134. Not all the warnings should be fixed • Add exclude

    into your bandit configuration • Add # nosec after the end of the code to skip the warning
  135. @clleew

  136. @clleew Project template Where's the Cookie?

  137. @clleew We want these manners in all our Python projects

  138. @clleew We want these manners in all our Python projects

    ❗ But... we don't want to configure it everytime ❗
  139. @clleew Create a project template once and initialize projects through

    it
  140. @clleew Lee-W / cookiecutter-python-template

  141. @clleew

  142. @clleew

  143. @clleew

  144. @clleew

  145. @clleew

  146. @clleew How to make a template?
 cookiecutter.json

  147. @clleew How to make a template?
 cookiecutter.json

  148. @clleew Template structure Cookiecutter configuration Template (Use Jinja)

  149. @clleew An example in template {{cookiecutter.project_slug}}/tasks/env.py from invoke import task

    from tasks.common import VENV_PREFIX @task def init(ctx): """Install production dependencies""" {% if cookiecutter.dependency_management_tool == 'pipenv' -%} ctx.run("pipenv install --deploy") {%- elif cookiecutter.dependency_management_tool == 'poetry' -%} ctx.run("poetry install --no-dev") {%- endif %} {% if cookiecutter.dependency_management_tool == 'pipenv' -%} ctx.run("pipenv install --deploy")
  150. @clleew An example in template {{cookiecutter.project_slug}}/tasks/env.py from invoke import task

    from tasks.common import VENV_PREFIX @task def init(ctx): """Install production dependencies""" {% if cookiecutter.dependency_management_tool == 'pipenv' -%} ctx.run("pipenv install --deploy") {%- elif cookiecutter.dependency_management_tool == 'poetry' -%} ctx.run("poetry install --no-dev") {%- endif %} {% if cookiecutter.dependency_management_tool == 'pipenv' -%} ctx.run("pipenv install --deploy")
  151. @clleew An example in template {{cookiecutter.project_slug}}/tasks/env.py from invoke import task

    from tasks.common import VENV_PREFIX @task def init(ctx): """Install production dependencies""" {% if cookiecutter.dependency_management_tool == 'pipenv' -%} ctx.run("pipenv install --deploy") {%- elif cookiecutter.dependency_management_tool == 'poetry' -%} ctx.run("poetry install --no-dev") {%- endif %} {%- elif cookiecutter.dependency_management_tool == 'poetry' -%} ctx.run("poetry install --no-dev") {%- endif %}
  152. @clleew import os def remove_pipfile(): os.remove("Pipfile") def main(): if "{{

    cookiecutter.dependency_management_tool }}" != "pipenv": remove_pipfile() if __name__ == "__main__": main() Run before/after project is generated hooks/post_gen_project.py
  153. @clleew Lee-W / cookiecutter-python-template

  154. @clleew

  155. Other Tools • Testing • Hypothesis • nox • Documentation

    • Mkdocs • Sphinx
  156. @clleew Related Talks • Dependency Management • Tzu-ping Chung -

    這樣的開發環境沒問題嗎? (PyCon TW 2018) • Kenneth Reitz - Pipenv: The Future of Python Dependency Management (PyCon US 2018) • Patrick Muehlbauer - Python Dependency Management (PyCon DE 2018) • Test • Chun-Yu Tseng - 快快樂樂成為 Coding Ninja (by pytest) (PyCon APAC 2015) • Florian Bruhin – Pytest: Rapid Simple Testing (Swiss Python Summit 16)
  157. @clleew Related Talks • Style Check • Kyle Knapp -

    Automating Code Quality (PyCon US 2018) • Łukasz Langa - Life Is Better Painted Black, or: How to Stop Worrying and Embrace Auto-Formatting (PyCon US 2019) • Raymond Hettinger - Beyond PEP 8 -- Best practices for beautiful intelligible code (PyCon 2015) • Static Typing • Dustin Ingram - Static Typing in Python (PyCon US 2020) • Task Managements • Thea Flowers - Break the Cycle: Three excellent Python tools to automate repetitive tasks (PyCon US 2019) • Security • Terri Oda - Python Security Tools (PyCon US 2019) • Tennessee Leeuwenburg - Watch out for Safety Bandits! (PyCon AU 2018)
  158. @clleew References • pipenv • poetry • flake8 • pylint

    • mypy • black • isort • invoke • pre-commit • commitizen-tools • safety • bandit • cookiecutter
  159. @clleew

  160. @clleew Let's volunteering in PyCon TW 2021