Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Python Table Manners - A Clean Style @ PyCon CA 2019

Lee Wei
November 17, 2019

Python Table Manners - A Clean Style @ PyCon CA 2019

Lee Wei

November 17, 2019
Tweet

More Decks by Lee Wei

Other Decks in Technology

Transcript

  1. Python Table Manners
    A Clean Style

    View full-size slide

  2. Collaboration Note

    View full-size slide

  3. $whoami
    Wei Lee / 李唯
    Software Engineer @ Rakuten Slice
    Volunteer @ PyCon TW
    @clleew
    Lee-W
    http://lee-w.github.io

    View full-size slide

  4. PyCon Tour 2019
    (Unexpected)
    1.PyCon US !
    2.PyCon JP "
    3.PyCon TW #
    4.PyCon CA $

    View full-size slide

  5. PyCon Tour 2019
    (Unexpected)
    1.PyCon US ! ⇦ First PyCon Oversea
    2.PyCon JP "
    3.PyCon TW #
    4.PyCon CA $

    View full-size slide

  6. PyCon Tour 2019
    (Unexpected)
    1.PyCon US !
    2.PyCon JP " ⇦ First Lightning Talk
    3.PyCon TW #
    4.PyCon CA $

    View full-size slide

  7. PyCon Tour 2019
    (Unexpected)
    1.PyCon US !
    2.PyCon JP "
    3.PyCon TW # ⇦ First Volunteering
    4.PyCon CA $

    View full-size slide

  8. PyCon Tour 2019
    (Unexpected)
    1.PyCon US !
    2.PyCon JP "
    3.PyCon TW #
    4.PyCon CA $ ⇦ First Regular Talk in a PyCon!

    View full-size slide

  9. What I’m going to talk about
    • A brief introduction of tools solving the following problems
    • Dependency Management
    • Testing
    • Style Check
    • Task Management
    • git commit
    • Security
    • Continuous Integration

    View full-size slide

  10. Some examples come from
    rg-cli
    • pycontw/pycontw-postevent-report-generator
    • A PyCon TW attendees data analyser
    • Work in progress. Welcome to contribute

    View full-size slide

  11. I won’t cover…

    View full-size slide

  12. I won’t cover…

    View full-size slide

  13. Dependency Management

    View full-size slide

  14. How we used to start
    a Python project
    python -m venv ./venv
    source venv/bin/activate
    pip freeze >> requirements.txt

    View full-size slide

  15. Looks good. Right?

    View full-size slide

  16. Is this story familiar to you?
    Your code does not work.
    But, it worked on my machine.

    View full-size slide

  17. Is this story familiar to you?
    Your code does not work.
    But, it worked on my machine.
    It seems you miss some dependencies.

    View full-size slide

  18. Is this story familiar to you?
    Your code does not work.
    But, it worked on my machine.
    It seems you miss some dependencies.
    Ahh… I forgot to activate the virtual environment
    when I tested it…

    View full-size slide

  19. Sometimes we just forgot to…
    • Activate / deactivate virtual environment
    • Add package into requirements.txt

    View full-size slide

  20. Pipenv

    Python Dev Workflow for Humans

    View full-size slide

  21. Why Pipenv?
    • Virtual environments + Package management
    • No more manual update dependencies

    We use Pipfile and Pipfile.lock .

    View full-size slide

  22. Initial Virtual Environment
    # Initial pipenv
    # Create Pipfile and Pipfile.lock if not yet exist
    pipenv install

    View full-size slide

  23. Empty Pipfile
    [[source]]
    name = "pypi"
    url = "https://pypi.org/simple"
    verify_ssl = true
    [dev-packages]
    [packages]
    [requires]
    python_version = "3.7"

    View full-size slide

  24. Install Package
    # Install package
    pipenv install ==
    # Uninstall package
    pipenv uninstall

    View full-size slide

  25. Pipfile with “requests”
    [[source]]
    name = "pypi"
    url = "https://pypi.org/simple"
    verify_ssl = true
    [dev-packages]
    [packages]
    requests = “*"
    [requires]
    python_version = "3.7"

    View full-size slide

  26. Pipfile.lock with “requests”
    {
    ......
    "default": {
    ......
    "requests": {
    "hashes": [
    "sha256:11e007a8a2aa0323f5a921e9e6a2d7e4e67d9877e85773fba9ba6419025cbeb4",
    "sha256:9cf5292fcd0f598c671cfc1e0d7d1a7f13bb8085e9a590f48c010551dc6c4b31"
    ],
    "index": "pypi",
    "version": "==2.22.0"
    },
    "urllib3": {
    "hashes": [
    "sha256:3de946ffbed6e6746608990594d08faac602528ac7015ac28d33cee6a45b7398",
    "sha256:9a107b99a5393caf59c7aa3c1249c16e6879447533d0887f4336dde834c7be86"
    ],
    "version": "==1.25.6"
    }
    },
    ......
    }

    View full-size slide

  27. Pipfile.lock with “requests”
    {
    ......
    "default": {
    ......
    "requests": {
    "hashes": [
    "sha256:11e007a8a2aa0323f5a921e9e6a2d7e4e67d9877e85773fba9ba6419025cbeb4",
    "sha256:9cf5292fcd0f598c671cfc1e0d7d1a7f13bb8085e9a590f48c010551dc6c4b31"
    ],
    "index": "pypi",
    "version": "==2.22.0"
    },
    "urllib3": {
    "hashes": [
    "sha256:3de946ffbed6e6746608990594d08faac602528ac7015ac28d33cee6a45b7398",
    "sha256:9a107b99a5393caf59c7aa3c1249c16e6879447533d0887f4336dde834c7be86"
    ],
    "version": "==1.25.6"
    }
    },
    ......
    }
    You’re guaranteed to install the
    exact same package next time.

    View full-size slide

  28. Install package only in dev
    # Install dev only package
    pipenv install == --dev

    View full-size slide

  29. Pipfile with “pytest"
    [[source]]
    name = "pypi"
    url = "https://pypi.org/simple"
    verify_ssl = true
    [dev-packages]
    pytest = "*"
    [packages]
    requests = "*"
    [requires]
    python_version = "3.7"

    View full-size slide

  30. Run in virtual environment
    # Run command inside pipenv shell
    pipenv run
    # e.g.
    pipenv run python your_program.py

    View full-size slide

  31. I don’t recommend using
    • Reason 1: You might accidentally install
    something inside the environment through 

    pip install
    • Reason 2: You can do almost anything through
    pipenv run
    pipenv shell

    View full-size slide

  32. Testing
    Don't let your customer debug for you

    View full-size slide

  33. Add new features to a project
    without tests is like…

    View full-size slide

  34. unittest

    Unit testing framework in Python Standard Library

    View full-size slide

  35. Why pytest
    • More Pythonic
    • Compatible with old unittest tests
    • No boilerplate code required
    • No more assert.+ (e.g., assertEqual), just assert
    • Better test discovery
    • Advance features: mark, parameterize and etc.
    • Plenty of plugins

    View full-size slide

  36. Running unittest in rg-cli
    python -m unittest discover -s ./ -p 'test_*.py'
    !"" test

    #"" test_sponsor.py
    !"" test_title.py

    View full-size slide

  37. Well… It’s not all unittest’s fault.
    • test directory need to be a package

    (even for pytest)




    • After that we can use
    python -m unittest
    !"" test
    #"" __init__.py
    ...
    #"" test_sponsor.py
    !"" test_title.py

    View full-size slide

  38. Running pytest in rg-cli
    pytest

    View full-size slide

  39. Running pytest in rg-cli
    pytest

    View full-size slide

  40. Run test in pytest
    pipenv install pytest --dev
    pipenv run pytest
    After using pipenv (or any other virtual environment tool),
    you should always run in a virtual environment.
    Note that I won’t add the pipenv run prefix future pages.

    View full-size slide

  41. unittest style tests
    import unittest
    from atta.partner import sponsor
    class TestSponsor(unittest.TestCase):
    def setUp(self):
    sponsors = sponsor.get_all_sponsors('./data/packages.yaml',
    './data/sponsors.yaml')
    self.sponsors = sponsors
    ......
    def test_sponsor_number(self):
    self.assertEqual(len(self.sponsors), 1)
    .......

    View full-size slide

  42. unittest style tests
    import unittest
    from atta.partner import sponsor
    class TestSponsor(unittest.TestCase):
    def setUp(self):
    sponsors = sponsor.get_all_sponsors('./data/packages.yaml',
    './data/sponsors.yaml')
    self.sponsors = sponsors
    ......
    def test_sponsor_number(self):
    self.assertEqual(len(self.sponsors), 1)
    .......

    View full-size slide

  43. unittest style tests
    import unittest
    from atta.partner import sponsor
    class TestSponsor(unittest.TestCase):
    def setUp(self):
    sponsors = sponsor.get_all_sponsors('./data/packages.yaml',
    './data/sponsors.yaml')
    self.sponsors = sponsors
    ......
    def test_sponsor_number(self):
    self.assertEqual(len(self.sponsors), 1)
    .......

    View full-size slide

  44. unittest style tests
    import unittest
    from atta.partner import sponsor
    class TestSponsor(unittest.TestCase):
    def setUp(self):
    sponsors = sponsor.get_all_sponsors('./data/packages.yaml',
    './data/sponsors.yaml')
    self.sponsors = sponsors
    ......
    def test_sponsor_number(self):
    self.assertEqual(len(self.sponsors), 1)
    .......

    View full-size slide

  45. pytest style tests
    import pytest
    from report_generator.partner import sponsor
    class TestSponsor:
    @pytest.fixture(scope="class")
    def sponsors(self):
    return sponsor.get_all_sponsors("test/data/packages.yaml",
    “test/data/sponsors.yaml")
    ......
    def test_sponsor_number(self, sponsors):
    assert len(sponsors) == 1
    ......

    View full-size slide

  46. pytest style tests
    import pytest
    from report_generator.partner import sponsor
    class TestSponsor:
    @pytest.fixture(scope="class")
    def sponsors(self):
    return sponsor.get_all_sponsors("test/data/packages.yaml",
    “test/data/sponsors.yaml")
    ......
    def test_sponsor_number(self, sponsors):
    assert len(sponsors) == 1
    ......

    View full-size slide

  47. pytest style tests
    import pytest
    from report_generator.partner import sponsor
    class TestSponsor:
    @pytest.fixture(scope="class")
    def sponsors(self):
    return sponsor.get_all_sponsors("test/data/packages.yaml",
    “test/data/sponsors.yaml")
    ......
    def test_sponsor_number(self, sponsors):
    assert len(sponsors) == 1
    ......

    View full-size slide

  48. pytest style tests
    import pytest
    from report_generator.partner import sponsor
    @pytest.fixture(scope="class")
    def sponsors():
    return sponsor.get_all_sponsors("test/data/packages.yaml",
    "test/data/sponsors.yaml")
    def test_sponsor_number(sponsors):
    assert len(sponsors) == 1

    View full-size slide

  49. pytest style tests
    import pytest
    from report_generator.partner import sponsor
    @pytest.fixture(scope="class")
    def sponsors():
    return sponsor.get_all_sponsors("test/data/packages.yaml",
    "test/data/sponsors.yaml")
    def test_sponsor_number(sponsors):
    assert len(sponsors) == 1

    View full-size slide

  50. Pytest - powerful plugins
    • pytest-mock: Replace objects that are hard to
    test with fake objects
    • pytest-cov: Generate test coverage report
    • pytest-xdist: Distributed testing
    • and etc.

    View full-size slide

  51. Style Check
    Check syntax correctness and make your code Pythonic

    View full-size slide

  52. Style check tools
    • Check syntax and coding style
    • Flake8
    • Pylint
    • Static Typing
    • Mypy

    View full-size slide

  53. • Run Flake8
    • Code to Check


    • Warning
    Run Flake8
    import os
    os = "My Operating system"
    ./bad_code.py:4:1: F811 redefinition of unused 'os'
    from line 1
    ./bad_code.py:4:5: E222 multiple spaces after operator
    flake8

    View full-size slide

  54. Config Flake8 - .flake8
    [flake8]
    ignore =
    # F632: use ==/!= to compare str, bytes,
    and int literals
    F632,
    max-line-length = 119

    View full-size slide

  55. Config Pylint - .pylintrc
    • Run Pylint
    • Generate default config file

    • What I usually config in my .pylintrc
    pylint --generate-rcfile >> .pylintrc
    ...
    disable=print-statement,
    ...
    max-line-length=119
    pylint ......

    View full-size slide

  56. Flake8 v.s. Pylint
    • Flake8: Faster
    • Pylint: Stricter and more optional
    • How do I use them?
    • Enforce flake8 but run pylint for reference only

    View full-size slide

  57. Run mypy
    mypy -p -p ......
    --ignore-missing-imports

    View full-size slide

  58. Run mypy - code to check
    from typing import List
    def func(val: List[str]):
    print(val)
    func([1, 2, 3])

    View full-size slide

  59. Run mypy - Error Message
    wrong_type_hint.py:8: error: List item 0 has
    incompatible type "int"; expected “str”
    wrong_type_hint.py:8: error: List item 1 has
    incompatible type "int"; expected “str"
    wrong_type_hint.py:8: error: List item 2 has
    incompatible type "int"; expected "str"

    View full-size slide

  60. One step forward
    Fix the style automatically

    View full-size slide

  61. Run Black
    black ...

    View full-size slide

  62. How Black reformat

    View full-size slide

  63. How Black reformat

    View full-size slide

  64. Why Black?
    • It’s not configurable.

    View full-size slide

  65. Why Black?
    • It’s not configurable.
    No more arguing about which style is better

    View full-size slide

  66. –The Zen of Python, by Tim Peters
    “There should be one-- and preferably only
    one --obvious way to do it.”

    View full-size slide

  67. How I combine these tools
    pipenv run black -l 119
    pipenv run flake8
    pipenv run mypy -p --ignore-missing-imports
    pipenv run pylint

    View full-size slide

  68. Task Management
    Break the loop and repeat no more

    View full-size slide

  69. Invoke
    • Task execution tool that can manage your
    commands.
    • It’s like Makefile in Python.
    • No more type or memorize long commands!

    View full-size slide

  70. Before invoke in rg-cli

    View full-size slide

  71. Before invoke in rg-cli

    View full-size slide

  72. After invoke in rg-cli

    View full-size slide

  73. List all commands
    inv -l

    View full-size slide

  74. How ? - tasks.py
    from invoke import task
    PIPENV_PREFIX = "pipenv run"
    ......
    @task
    def install(cmd):
    """Install script in pipenv environment"""
    cmd.run(f"{PIPENV_PREFIX} python setup.py install")

    View full-size slide

  75. How ? - tasks.py
    from invoke import task
    PIPENV_PREFIX = "pipenv run"
    ......
    @task
    def install(cmd):
    """Install script in pipenv environment"""
    cmd.run(f"{PIPENV_PREFIX} python setup.py install")

    View full-size slide

  76. How ? - tasks.py
    from invoke import task
    PIPENV_PREFIX = "pipenv run"
    ......
    @task
    def install(cmd):
    """Install script in pipenv environment"""
    cmd.run(f"{PIPENV_PREFIX} python setup.py install")

    View full-size slide

  77. Modulize - Namespace
    #"" tasks
    $ #"" __init__.py
    $ #"" build.py
    $ #"" common.py
    $ #"" env.py
    $ #"" style.py
    $ !"" test.py

    View full-size slide

  78. Autocomplete
    inv --print-completion-script=zsh >> ~/.zshrc

    View full-size slide

  79. Autocomplete
    inv --print-completion-script=zsh >> ~/.zshrc

    View full-size slide

  80. Why not use Makefile?
    1.We’re Python developers!
    2.Some tasks might not be easy to handle through
    shell script.
    3.Python + Shell Script

    It’s the best of the both world!

    View full-size slide

  81. Commit message
    A clear way back to the past

    View full-size slide

  82. If you’re like him…

    View full-size slide

  83. You’ll hate yourself when
    you need to roll back your code.

    View full-size slide

  84. Commitizen in Python

    View full-size slide

  85. Commit - cz commit
    type of this change

    View full-size slide

  86. Commit - cz commit
    type of this change

    View full-size slide

  87. Commit - cz commit
    scope of this change

    View full-size slide

  88. Commit - cz commit

    a short description

    View full-size slide

  89. Regulated git commits

    View full-size slide

  90. Regulated git commits

    View full-size slide

  91. Other Commitizen features
    • Bump version automatically based on commits
    • Customizable commit rules

    View full-size slide

  92. Other Commitizen features
    • Bump version automatically based on commits
    • Customizable commit rules
    • Auto generate changelog (WIP)

    View full-size slide

  93. Let’s Sprint!

    View full-size slide

  94. Safety
    Dependency Checking

    View full-size slide

  95. Check vulnerability
    in current environment
    safety check

    View full-size slide

  96. Check vulnerability
    in current environment
    safety check

    View full-size slide

  97. But if you use pipenv …
    pipenv check

    View full-size slide

  98. How does Safety work?
    • Search the vulnerability in CVS 

    (Common Vulnerabilities and Exposures)
    • Free DB: update monthly
    • Paid DB: update realtime

    View full-size slide

  99. Bandit
    Static Analysis

    View full-size slide

  100. Check Common Security
    Issue in Python Code
    bandit -r

    View full-size slide

  101. Check Common Security
    Issue in Python Code
    bandit -r

    View full-size slide

  102. Check Common Security
    Issue in Python Code
    bandit -r

    View full-size slide

  103. Check Common Security
    Issue in Python Code
    bandit -r

    View full-size slide

  104. Check Common Security
    Issue in Python Code
    bandit -r

    View full-size slide

  105. Not all the warnings
    should be fixed
    • Use bandit as a reference.

    View full-size slide

  106. Skip the warning in bandit
    • Add # nosec after the line code
    • Config in .bandit

    View full-size slide

  107. Continuous Integration
    Assemble all the trivial steps

    View full-size slide

  108. Why Drone
    • Plenty of Plugins
    • CI configuration as code
    • Easy to config
    • Container-based
    • Run on local easily ($ drone exec)
    • Easy to setup

    View full-size slide

  109. Install drone-cli

    View full-size slide

  110. Drone in rg-cli

    View full-size slide

  111. Config Drone - .drone.yml
    ---
    kind: pipeline
    type: docker
    name: default
    steps:
    - name: setup
    image: python:3.7-slim
    pull: if-not-exists
    volumes:
    - name: system_dependencies
    path: /usr/local
    - name: virtual_env
    path: /root/.local/share/virtualenvs
    commands:
    - pip install pipenv invoke
    - inv env.init-dev


    ... 

    volumes:
    - name: system_dependencies
    temp: {}
    - name: virtual_env
    temp: {}

    View full-size slide

  112. Config Drone - .drone.yml
    ---
    kind: pipeline
    type: docker
    name: default
    steps:
    - name: setup
    image: python:3.7-slim
    pull: if-not-exists
    volumes:
    - name: system_dependencies
    path: /usr/local
    - name: virtual_env
    path: /root/.local/share/virtualenvs
    commands:
    - pip install pipenv invoke
    - inv env.init-dev


    ... 

    volumes:
    - name: system_dependencies
    temp: {}
    - name: virtual_env
    temp: {}

    View full-size slide

  113. Config Drone - .drone.yml
    ---
    kind: pipeline
    type: docker
    name: default
    steps:
    - name: setup
    image: python:3.7-slim
    pull: if-not-exists
    volumes:
    - name: system_dependencies
    path: /usr/local
    - name: virtual_env
    path: /root/.local/share/virtualenvs
    commands:
    - pip install pipenv invoke
    - inv env.init-dev


    ... 

    volumes:
    - name: system_dependencies
    temp: {}
    - name: virtual_env
    temp: {}

    View full-size slide

  114. Config Drone - .drone.yml
    ---
    kind: pipeline
    type: docker
    name: default
    steps:
    - name: setup
    image: python:3.7-slim
    pull: if-not-exists
    volumes:
    - name: system_dependencies
    path: /usr/local
    - name: virtual_env
    path: /root/.local/share/virtualenvs
    commands:
    - pip install pipenv invoke
    - inv env.init-dev


    ... 

    volumes:
    - name: system_dependencies
    temp: {}
    - name: virtual_env
    temp: {}

    View full-size slide

  115. Config Drone - .drone.yml
    - name: check package security
    image: python:3.7-slim
    pull: if-not-exists
    volumes:
    - name: system_dependencies
    path: /usr/local
    - name: virtual_env
    path: /root/.local/share/virtualenvs
    commands:
    - inv secure
    depends_on:
    - setup

    View full-size slide

  116. Config Drone - .drone.yml
    - name: check package security
    image: python:3.7-slim
    pull: if-not-exists
    volumes:
    - name: system_dependencies
    path: /usr/local
    - name: virtual_env
    path: /root/.local/share/virtualenvs
    commands:
    - inv secure
    depends_on:
    - setup

    View full-size slide

  117. Config Drone - .drone.yml
    - name: check package security
    image: python:3.7-slim
    pull: if-not-exists
    volumes:
    - name: system_dependencies
    path: /usr/local
    - name: virtual_env
    path: /root/.local/share/virtualenvs
    commands:
    - inv secure
    depends_on:
    - setup

    View full-size slide

  118. Run CI locally
    drone exec

    View full-size slide

  119. What you need to do next
    1.Push your code change to your remote git repo
    2.Let drone check automatically
    3.
    4.Celebrate

    View full-size slide

  120. Conclusion
    1. Initial project and install packages through pipenv
    2. Write unit tests to ensure functionality correctness
    and run pytest
    3. Format your code through black
    4. Check coding style through flake8, pylint, mypy
    5. Manage all the commands through invoke
    6. Regulate commit message through commitizen

    View full-size slide

  121. Conclusion (Cont.)
    7. Check vulnerability through Safety and bandit
    8. Setup Drone CI on your remote git repo and let
    it check for you automatically

    View full-size slide

  122. Tools
    • Dependency Management
    • Poetry: Manage packages and support
    package releasing (replace setup.py)
    • pip-tools: Compile requirements to pin
    package versions and without leaving pip
    and venv

    View full-size slide

  123. Tools
    • Testing
    • Hypothesis
    • nox
    • Project Creation
    • Cookiecutter
    • PyScaffold

    View full-size slide

  124. Related Talks
    • Dependency Management
    • Tzu-ping Chung - 這樣的開發環境沒問題嗎? (PyCon TW 2018)
    • Kenneth Reitz - Pipenv: The Future of Python Dependency
    Management (PyCon US 2018)
    • Style Check
    • Kyle Knapp - Automating Code Quality - Kyle Knapp (PyCon US
    2018)
    • Task Managements
    • Thea Flowers - Break the Cycle: Three excellent Python tools to
    automate repetitive tasks (PyCon US 2019)

    View full-size slide

  125. Related Talks
    • Test
    • Chun-Yu Tseng - 快快樂樂成為 Coding Ninja (by pytest) (PyCon APAC
    2015)
    • Florian Bruhin – Pytest: Rapid Simple Testing (Swiss Python Summit 16)
    • Security
    • Terri Oda - Python Security Tools (PyCon US 2019)
    • Tennessee Leeuwenburg - Watch out for Safety Bandits! (PyCon AU 2018)
    • Multiple Topic
    • Dustin Ingram - Modern development environments for Pythonistas (PyCon
    JP 2019)

    View full-size slide