Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Python Table Manners @ Taichung.py

Lee Wei
November 02, 2020

Python Table Manners @ Taichung.py

想要在你的團隊建立標準的 Python 專案開發流程嗎?那就來聽這場分享吧!我將會介紹一些增加程式碼品質的小工具們。透過這些工具,你可以省去很多無聊且繁瑣的任務,把心力留給更重要的軟體設計。在最後,我會分享我如何把這些工具整合起來,讓你可以輕鬆地把這些工具帶到你的下一個專案。

Lee Wei

November 02, 2020
Tweet

More Decks by Lee Wei

Other Decks in Programming

Transcript

  1. @clleew
    Python Table Manners

    Everything you need
    in software development life cycle

    View full-size slide

  2. @clleew
    Slide

    View full-size slide

  3. @clleew
    Outline
    • Clean up the table - Dependency Management
    • Put the correct tablewares - Testing
    • Use the tablewares elegantly - Coding Style
    • Mnemonic phrase - Task management
    • Say please! - Check before git operation
    • Speak formally - Cultivate a git commit convention
    • Safety matters - Security Issues
    • Where's the Cookie? - Project template

    View full-size slide

  4. @clleew
    __name__ = 李唯 / Wei Lee
    __position__ = [
    Software Engineer @ Rakuten Slice,
    2021 Chairperson @ PyCon Taiwan,
    Maintainer of commitizen-tools,
    ]
    __twitter__ = @clleew
    __github__ = Lee-W
    __blog__ = http://lee-w.github.io
    $ cat speaker.py

    View full-size slide

  5. @clleew
    $ python speaker.py
    File "speaker.py", line 1
    __name__ = 李唯 / Wei Lee
    ^
    SyntaxError: invalid syntax
    $ python speaker.py

    View full-size slide

  6. @clleew
    Let's join PyCon TW 2021

    View full-size slide

  7. @clleew
    Dependency Management
    Clean up the table

    View full-size slide

  8. @clleew
    How we used to start
    a Python project

    View full-size slide

  9. @clleew
    Sometimes we just forgot to…
    • ... activate / deactivate virtual environment
    • ... add package into requirements.txt

    View full-size slide

  10. @clleew
    Pipenv
    Python Dev Workflow for Humans
    • manage virtual environment + package
    → update at once
    → No more manual sync up!
    • generate hashes from packages
    → ensure getting the same packages next time

    View full-size slide

  11. @clleew
    Initial Virtual Environment

    View full-size slide

  12. @clleew
    Empty Pipfile
    [[source]]
    name = "pypi"
    url = "https://pypi.org/simple"
    verify_ssl = true
    [dev-packages]
    [packages]
    [requires]
    python_version = "3.7"

    View full-size slide

  13. @clleew
    Install Package

    View full-size slide

  14. @clleew
    Pipfile with “requests”
    [[source]]
    name = "pypi"
    url = "https://pypi.org/simple"
    verify_ssl = true
    [dev-packages]
    [packages]
    requests = "*"
    [requires]
    python_version = "3.7"
    [packages]
    requests = "*"

    View full-size slide

  15. @clleew
    "requests": {
    "hashes": [
    "sha256:11e007a8a2aa0323f5a921e9e6a2d7e4e67d9877e85773fba9ba6419025cbeb4",
    "sha256:9cf5292fcd0f598c671cfc1e0d7d1a7f13bb8085e9a590f48c010551dc6c4b31"
    ],
    "index": "pypi",
    "version": "==2.22.0"
    },
    {
    ......
    "default": {
    ......
    "requests": {
    "hashes": [
    "sha256:11e007a8a2aa0323f5a921e9e6a2d7e4e67d9877e85773fba9ba6419025cbeb4",
    "sha256:9cf5292fcd0f598c671cfc1e0d7d1a7f13bb8085e9a590f48c010551dc6c4b31"
    ],
    "index": "pypi",
    "version": "==2.22.0"
    },
    "urllib3": {
    "hashes": [
    "sha256:3de946ffbed6e6746608990594d08faac602528ac7015ac28d33cee6a45b7398",
    "sha256:9a107b99a5393caf59c7aa3c1249c16e6879447533d0887f4336dde834c7be86"
    ],
    "version": "==1.25.6"
    }
    },
    ......
    }
    Pipfile.lock with “requests”
    You’re guaranteed to install the
    exact same package next time.

    View full-size slide

  16. @clleew
    Install package only in dev

    View full-size slide

  17. @clleew
    Pipfile with “pytest"
    [[source]]
    name = "pypi"
    url = "https://pypi.org/simple"
    verify_ssl = true
    [dev-packages]
    pytest = "*"
    [packages]
    requests = "*"
    [requires]
    python_version = "3.7"
    [dev-packages]
    pytest = "*"

    View full-size slide

  18. @clleew
    Run in virtual environment

    View full-size slide

  19. @clleew
    Some people might say
    that pipenv ...
    • ... does not update frequently.

    • ... locks slowly.

    • ... does not syncs up with install_requires in setup.py
    Or maybe you can manage dependencies in setup.py,
    and use “pipenv install -e .” (learned from dboy )

    View full-size slide

  20. @clleew
    Poetry

    View full-size slide

  21. @clleew
    Poetry commands

    View full-size slide

  22. @clleew
    Testing
    Put the correct tablewares

    View full-size slide

  23. @clleew
    unittest
    Unit testing framework in Python Standard Library

    View full-size slide

  24. @clleew
    pytest

    View full-size slide

  25. @clleew
    Why pytest
    • More Pythonic

    • Compatible with old unittest tests

    • No more assert.+ (e.g., assertEqual, assertTrue, etc.)

    → just assert

    • Better test discovery

    • Advance features: mark, parameterize, etc.

    • Plenty of plugins

    View full-size slide

  26. @clleew
    Run pytest

    View full-size slide

  27. @clleew
    always run python inside a virtual environment

    View full-size slide

  28. @clleew
    unittest style tests
    import unittest
    from atta.partner import sponsor
    class TestSponsor(unittest.TestCase):
    def setUp(self):
    sponsors = sponsor.get_all_sponsors('./data/packages.yaml',
    './data/sponsors.yaml')
    self.sponsors = sponsors
    ......
    def test_sponsor_number(self):
    self.assertEqual(len(self.sponsors), 1)
    .......
    Prepare all the data needed in test cases

    View full-size slide

  29. @clleew
    pytest style tests
    import pytest
    from report_generator.partner import sponsor
    class TestSponsor:
    @pytest.fixture(scope="class")
    def sponsors(self):
    return sponsor.get_all_sponsors("test/data/packages.yaml",
    “test/data/sponsors.yaml")
    ......
    def test_sponsor_number(self, sponsors):
    assert len(sponsors) == 1
    ......
    Prepare the data needed in separate fixtures

    View full-size slide

  30. @clleew
    pytest style tests
    import pytest
    from report_generator.partner import sponsor
    @pytest.fixture(scope="function")
    def sponsors():
    return sponsor.get_all_sponsors("test/data/packages.yaml",
    "test/data/sponsors.yaml")
    def test_sponsor_number(sponsors):
    assert len(sponsors) == 1

    View full-size slide

  31. @clleew
    Configuration - pyproject.toml
    [tool.pytest.ini_options]
    minversion = "6.0"
    testpaths = "tests"
    addopts = "--strict-markers"
    norecursedirs = [
    ".*",
    "build",
    "dist",
    "CVS",
    "_darcs",
    "{arch}",
    "*.egg",
    "venv",
    "env",
    "virtualenv"
    ]

    View full-size slide

  32. @clleew
    Pytest - powerful plugins
    • pytest-mock: Replace objects that are hard to test with
    fake objects

    • pytest-cov: Generate test coverage report

    • pytest-xdist: Distributed testing

    • pytest-regressions

    • and etc.

    View full-size slide

  33. @clleew
    Coding Style
    Use the tablewares elegantly

    View full-size slide

  34. @clleew
    flake8
    • Enforcing style consistency across Python project

    • Check possible errors before running your program

    • Eliminate bad style

    View full-size slide

  35. @clleew
    import os
    os = "My Operating system"
    additional space

    (bad coding style)
    redefinition of imported library

    (possible error)
    Example - bad_code.py

    View full-size slide

  36. @clleew
    Run flake8

    View full-size slide

  37. @clleew
    Configuration - setup.cfg
    [flake8]
    ignore =
    # F632: use ==/!= to compare str, bytes, and int literals
    F632,
    # W503: Line break occurred before a binary operator
    W503,
    # E501: Line too long
    E501,
    # E203: Whitespace before ':' (for black)
    E203
    exclude =
    .git,
    __pycache__,
    build,
    dist
    max-line-length = 88

    View full-size slide

  38. @clleew
    Pylint

    View full-size slide

  39. @clleew
    import os
    os = "My Operating system"
    additional space

    (bad coding style)
    redefinition of imported library

    (possible error)
    Example - bad_code.py

    View full-size slide

  40. @clleew
    Run pylint

    View full-size slide

  41. @clleew
    Run pylint with "-r" argument
    A bunch of reports that 

    you can compare 

    with your previous run

    View full-size slide

  42. Configuration - pyproject.toml
    [tool.pylint.messages_control]
    disable = [
    "bad-continuation",
    "missing-function-docstring",
    "missing-module-docstring",
    "invalid-name"
    ]
    [tool.pylint.format]
    max-line-length = 88

    View full-size slide

  43. @clleew
    mypy
    • Static type checker

    • Compile-time type checking

    → avoid possible runtime errors

    • Type annotation enhances readability

    → machine-checked documentation

    View full-size slide

  44. from typing import List
    def func(vals: List[str]):
    for val in vals:
    assert isinstance(val, str)
    func([1, 2, 3])
    Example - error_type.py
    List[str]
    [1, 2, 3]

    View full-size slide

  45. Run mypy
    all the files with .py extension
    ignore errors that libs that is not type annotated

    View full-size slide

  46. @clleew
    Configuration - setup.cfg
    [mypy]
    files = **/*.py
    ignore_missing_imports = true
    follow_imports = silent
    warn_redundant_casts = True
    warn_unused_ignores = True
    warn_unused_configs = True
    [mypy]
    files = **/*.py
    ignore_missing_imports = true

    View full-size slide

  47. Run mypy with configuration

    View full-size slide

  48. @clleew
    One step forward
    Fix the style automatically

    View full-size slide

  49. How Black reformat

    View full-size slide

  50. How Black reformat

    View full-size slide

  51. Why Black?
    The black code style is not configurable.
    No more arguing about which style is better

    View full-size slide

  52. –The Zen of Python, by Tim Peters
    “There should be one-- and preferably only
    one --obvious way to do it.”

    View full-size slide

  53. @clleew
    Configuration - pyproject.toml
    [tool.black]
    line-length = 88
    include = '\.pyi?$'
    exclude = '''
    /(
    \.eggs
    | \.git
    | \.hg
    | \.mypy_cache
    | \.tox
    | \.venv
    | _build
    | buck-out
    | build
    | dist
    )/
    '''

    View full-size slide

  54. Example
    wrong_import_order.py
    import django
    import os
    import some_custom_package
    import flask
    import datetime
    ...

    View full-size slide

  55. According to PEP8
    • Imports should be grouped in the following order:
    1. Standard library imports.
    2. Related third party imports.
    3. Local application/library specific imports.
    • You should put a blank line between each group
    of imports.

    View full-size slide

  56. Run Isort
    import datetime
    import os
    import django
    import flask
    import some_custom_package
    ...

    View full-size slide

  57. @clleew
    Configuration - pyproject.toml
    [tool.isort]
    profile = "black"

    View full-size slide

  58. Style formatting and linting

    View full-size slide

  59. @clleew
    Task Management
    Mnemonic phrase

    View full-size slide

  60. Invoke
    • Task execution tool that can manage your
    commands
    • It’s like a Makefile but in Python.

    View full-size slide

  61. Invoke in practice (rg-cli)

    View full-size slide

  62. Before invoke in rg-cli

    View full-size slide

  63. After invoke in rg-cli

    View full-size slide

  64. List all commands

    View full-size slide

  65. How ? - tasks.py
    from invoke import task
    VENV_PREFIX = "pipenv run"
    ......
    @task
    def install(cmd):
    """Install script in pipenv environment"""
    cmd.run(f"{VENV_PREFIX} python setup.py install")

    View full-size slide

  66. Modulize - Namespace

    View full-size slide

  67. Autocomplete

    View full-size slide

  68. Why not use Makefile?
    1.We’re Python developers!
    2.Some tasks might not be easy to handle through
    shell script.
    3.Combining Python and shell script

    View full-size slide

  69. @clleew
    Check before git operation
    Say please!

    View full-size slide

  70. @clleew
    pre-commit
    • Why?

    • Sometimes we still forget to run the check commands.

    • Prevent committing bad code into codebase

    • How?

    • pre-commit runs these commands before we do git
    operations (e.g., git push, git commit)

    View full-size slide

  71. @clleew
    Config (customized hooks)
    .pre-commit-config.yaml
    repos:
    - repo: local
    hooks:
    - id: style-reformat
    name: style-reformat
    stages: [commit]
    language: system
    pass_filenames: false
    entry: inv style.reformat
    types: [python]
    - id: style-check
    name: style-check
    stages: [push]
    language: system
    pass_filenames: false
    entry: inv style
    types: [python]
    ......
    stages: [push]
    entry: inv style.reformat
    - id: style-reformat
    stages: [commit]
    - id: style-check

    View full-size slide

  72. @clleew
    Config (existing hooks)
    .pre-commit-config.yaml
    ......
    - repo: https://github.com/pre-commit/pre-commit-hooks
    rev: v3.2.0
    hooks:
    - id: end-of-file-fixer
    - id: trailing-whitespace
    args: [--markdown-linebreak-ext=md]

    View full-size slide

  73. @clleew
    Setup pre-commit

    View full-size slide

  74. @clleew
    Run pre-commit

    View full-size slide

  75. @clleew
    Run pre-commit

    View full-size slide

  76. @clleew
    Cultivate
    a git commit convention
    Speak formally

    View full-size slide

  77. Hard to find to the right version

    View full-size slide

  78. commitizen-tools

    View full-size slide

  79. Commit - cz commit

    View full-size slide

  80. Commit - cz commit

    View full-size slide

  81. How does the commit look like?
    feat(Taichung.py): This is a great meetup!
    Hope you enjoy my talk!
    https://taichung-py.kktix.cc/events/
    meetup-202011-clleew

    View full-size slide

  82. Standardized git commits

    View full-size slide

  83. Other Commitizen features
    • pre-commit hook prevents you from not using
    commitizen
    • customizable commit rules
    • auto bump project version (SemVar)
    • auto generate changelog (Keep a Changelog)

    View full-size slide

  84. @clleew
    Security Issues
    Safety matters

    View full-size slide

  85. Safety
    Dependency Checking

    View full-size slide

  86. Check vulnerability
    in current environment

    View full-size slide

  87. But if you use pipenv …

    View full-size slide

  88. How does Safety work?
    • Search the vulnerability in CVS
    (Common Vulnerabilities and Exposures)
    • Free DB: update monthly
    • Paid DB: update realtime

    View full-size slide

  89. Bandit
    Static Analysis

    View full-size slide

  90. Check Common Security
    Issue in Python Code

    View full-size slide

  91. Not all the warnings
    should be fixed
    • Add exclude into your bandit configuration
    • Add # nosec after the end of the code to skip
    the warning

    View full-size slide

  92. @clleew
    Project template
    Where's the Cookie?

    View full-size slide

  93. @clleew
    We want these manners in all our Python projects ❗
    But... we don't want to configure it everytime ❗

    View full-size slide

  94. @clleew
    Create a project template once
    and initialize projects through it

    View full-size slide

  95. @clleew
    Lee-W / cookiecutter-python-template

    View full-size slide

  96. @clleew
    How to make a template?
    cookiecutter.json

    View full-size slide

  97. @clleew
    Template structure
    Cookiecutter configuration
    Template (Use Jinja)

    View full-size slide

  98. @clleew
    An example in template
    {{cookiecutter.project_slug}}/tasks/env.py
    from invoke import task
    from tasks.common import VENV_PREFIX
    @task
    def init(ctx):
    """Install production dependencies"""
    {% if cookiecutter.dependency_management_tool == 'pipenv' -%}
    ctx.run("pipenv install --deploy")
    {%- elif cookiecutter.dependency_management_tool == 'poetry' -%}
    ctx.run("poetry install --no-dev")
    {%- endif %}
    {% if cookiecutter.dependency_management_tool == 'pipenv' -%}
    ctx.run("pipenv install --deploy")
    {%- elif cookiecutter.dependency_management_tool == 'poetry' -%}
    ctx.run("poetry install --no-dev")
    {%- endif %}

    View full-size slide

  99. @clleew
    import os
    def remove_pipfile():
    os.remove("Pipfile")
    def main():
    if "{{ cookiecutter.dependency_management_tool }}" != "pipenv":
    remove_pipfile()
    if __name__ == "__main__":
    main()
    Run before/after project is generated
    hooks/post_gen_project.py

    View full-size slide

  100. @clleew
    Lee-W / cookiecutter-python-template

    View full-size slide

  101. Other Tools
    • Testing
    • Hypothesis
    • nox
    • Documentation
    • Mkdocs
    • Sphinx

    View full-size slide

  102. @clleew
    Related Talks
    • Dependency Management
    • Tzu-ping Chung - 這樣的開發環境沒問題嗎? (PyCon TW 2018)
    • Kenneth Reitz - Pipenv: The Future of Python Dependency
    Management (PyCon US 2018)
    • Patrick Muehlbauer - Python Dependency Management (PyCon DE
    2018)
    • Test
    • Chun-Yu Tseng - 快快樂樂成為 Coding Ninja (by pytest) (PyCon APAC
    2015)
    • Florian Bruhin – Pytest: Rapid Simple Testing (Swiss Python Summit 16)

    View full-size slide

  103. @clleew
    Related Talks
    • Style Check
    • Kyle Knapp - Automating Code Quality (PyCon US 2018)
    • Łukasz Langa - Life Is Better Painted Black, or: How to Stop Worrying and Embrace Auto-Formatting
    (PyCon US 2019)
    • Raymond Hettinger - Beyond PEP 8 -- Best practices for beautiful intelligible code (PyCon 2015)
    • Static Typing
    • Dustin Ingram - Static Typing in Python (PyCon US 2020)
    • Task Managements
    • Thea Flowers - Break the Cycle: Three excellent Python tools to automate repetitive tasks (PyCon US
    2019)
    • Security
    • Terri Oda - Python Security Tools (PyCon US 2019)
    • Tennessee Leeuwenburg - Watch out for Safety Bandits! (PyCon AU 2018)

    View full-size slide

  104. @clleew
    References
    • pipenv

    • poetry

    • flake8

    • pylint

    • mypy

    • black

    • isort
    • invoke

    • pre-commit

    • commitizen-tools

    • safety

    • bandit

    • cookiecutter

    View full-size slide

  105. @clleew
    Let's join PyCon TW 2021

    View full-size slide