Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Python Table Manners - A Clean Style @ PyCon CA 2019

Lee Wei
November 17, 2019

Python Table Manners - A Clean Style @ PyCon CA 2019

Lee Wei

November 17, 2019
Tweet

More Decks by Lee Wei

Other Decks in Technology

Transcript

  1. $whoami Wei Lee / 李唯 Software Engineer @ Rakuten Slice

    Volunteer @ PyCon TW @clleew Lee-W http://lee-w.github.io
  2. PyCon Tour 2019 (Unexpected) 1.PyCon US ! ⇦ First PyCon

    Oversea 2.PyCon JP " 3.PyCon TW # 4.PyCon CA $
  3. PyCon Tour 2019 (Unexpected) 1.PyCon US ! 2.PyCon JP "

    ⇦ First Lightning Talk 3.PyCon TW # 4.PyCon CA $
  4. PyCon Tour 2019 (Unexpected) 1.PyCon US ! 2.PyCon JP "

    3.PyCon TW # ⇦ First Volunteering 4.PyCon CA $
  5. PyCon Tour 2019 (Unexpected) 1.PyCon US ! 2.PyCon JP "

    3.PyCon TW # 4.PyCon CA $ ⇦ First Regular Talk in a PyCon!
  6. What I’m going to talk about • A brief introduction

    of tools solving the following problems • Dependency Management • Testing • Style Check • Task Management • git commit • Security • Continuous Integration
  7. Some examples come from rg-cli • pycontw/pycontw-postevent-report-generator • A PyCon

    TW attendees data analyser • Work in progress. Welcome to contribute
  8. How we used to start a Python project python -m

    venv ./venv source venv/bin/activate pip freeze >> requirements.txt
  9. Is this story familiar to you? Your code does not

    work. But, it worked on my machine.
  10. Is this story familiar to you? Your code does not

    work. But, it worked on my machine. It seems you miss some dependencies.
  11. Is this story familiar to you? Your code does not

    work. But, it worked on my machine. It seems you miss some dependencies. Ahh… I forgot to activate the virtual environment when I tested it…
  12. Sometimes we just forgot to… • Activate / deactivate virtual

    environment • Add package into requirements.txt
  13. Why Pipenv? • Virtual environments + Package management • No

    more manual update dependencies
 We use Pipfile and Pipfile.lock .
  14. Initial Virtual Environment # Initial pipenv # Create Pipfile and

    Pipfile.lock if not yet exist pipenv install
  15. Empty Pipfile [[source]] name = "pypi" url = "https://pypi.org/simple" verify_ssl

    = true [dev-packages] [packages] [requires] python_version = "3.7"
  16. Pipfile with “requests” [[source]] name = "pypi" url = "https://pypi.org/simple"

    verify_ssl = true [dev-packages] [packages] requests = “*" [requires] python_version = "3.7"
  17. Pipfile.lock with “requests” { ...... "default": { ...... "requests": {

    "hashes": [ "sha256:11e007a8a2aa0323f5a921e9e6a2d7e4e67d9877e85773fba9ba6419025cbeb4", "sha256:9cf5292fcd0f598c671cfc1e0d7d1a7f13bb8085e9a590f48c010551dc6c4b31" ], "index": "pypi", "version": "==2.22.0" }, "urllib3": { "hashes": [ "sha256:3de946ffbed6e6746608990594d08faac602528ac7015ac28d33cee6a45b7398", "sha256:9a107b99a5393caf59c7aa3c1249c16e6879447533d0887f4336dde834c7be86" ], "version": "==1.25.6" } }, ...... }
  18. Pipfile.lock with “requests” { ...... "default": { ...... "requests": {

    "hashes": [ "sha256:11e007a8a2aa0323f5a921e9e6a2d7e4e67d9877e85773fba9ba6419025cbeb4", "sha256:9cf5292fcd0f598c671cfc1e0d7d1a7f13bb8085e9a590f48c010551dc6c4b31" ], "index": "pypi", "version": "==2.22.0" }, "urllib3": { "hashes": [ "sha256:3de946ffbed6e6746608990594d08faac602528ac7015ac28d33cee6a45b7398", "sha256:9a107b99a5393caf59c7aa3c1249c16e6879447533d0887f4336dde834c7be86" ], "version": "==1.25.6" } }, ...... } You’re guaranteed to install the exact same package next time.
  19. Install package only in dev # Install dev only package

    pipenv install <package>==<version> --dev
  20. Pipfile with “pytest" [[source]] name = "pypi" url = "https://pypi.org/simple"

    verify_ssl = true [dev-packages] pytest = "*" [packages] requests = "*" [requires] python_version = "3.7"
  21. Run in virtual environment # Run command inside pipenv shell

    pipenv run <Command> # e.g. pipenv run python your_program.py
  22. I don’t recommend using • Reason 1: You might accidentally

    install something inside the environment through 
 pip install • Reason 2: You can do almost anything through pipenv run <command> pipenv shell
  23. Why pytest • More Pythonic • Compatible with old unittest

    tests • No boilerplate code required • No more assert.+ (e.g., assertEqual), just assert • Better test discovery • Advance features: mark, parameterize and etc. • Plenty of plugins
  24. Running unittest in rg-cli python -m unittest discover -s ./

    -p 'test_*.py' !"" test … #"" test_sponsor.py !"" test_title.py
  25. Well… It’s not all unittest’s fault. • test directory need

    to be a package
 (even for pytest)
 
 
 
 • After that we can use python -m unittest !"" test #"" __init__.py ... #"" test_sponsor.py !"" test_title.py
  26. Run test in pytest pipenv install pytest --dev pipenv run

    pytest After using pipenv (or any other virtual environment tool), you should always run in a virtual environment. Note that I won’t add the pipenv run prefix future pages.
  27. unittest style tests import unittest from atta.partner import sponsor class

    TestSponsor(unittest.TestCase): def setUp(self): sponsors = sponsor.get_all_sponsors('./data/packages.yaml', './data/sponsors.yaml') self.sponsors = sponsors ...... def test_sponsor_number(self): self.assertEqual(len(self.sponsors), 1) .......
  28. unittest style tests import unittest from atta.partner import sponsor class

    TestSponsor(unittest.TestCase): def setUp(self): sponsors = sponsor.get_all_sponsors('./data/packages.yaml', './data/sponsors.yaml') self.sponsors = sponsors ...... def test_sponsor_number(self): self.assertEqual(len(self.sponsors), 1) .......
  29. unittest style tests import unittest from atta.partner import sponsor class

    TestSponsor(unittest.TestCase): def setUp(self): sponsors = sponsor.get_all_sponsors('./data/packages.yaml', './data/sponsors.yaml') self.sponsors = sponsors ...... def test_sponsor_number(self): self.assertEqual(len(self.sponsors), 1) .......
  30. unittest style tests import unittest from atta.partner import sponsor class

    TestSponsor(unittest.TestCase): def setUp(self): sponsors = sponsor.get_all_sponsors('./data/packages.yaml', './data/sponsors.yaml') self.sponsors = sponsors ...... def test_sponsor_number(self): self.assertEqual(len(self.sponsors), 1) .......
  31. pytest style tests import pytest from report_generator.partner import sponsor class

    TestSponsor: @pytest.fixture(scope="class") def sponsors(self): return sponsor.get_all_sponsors("test/data/packages.yaml", “test/data/sponsors.yaml") ...... def test_sponsor_number(self, sponsors): assert len(sponsors) == 1 ......
  32. pytest style tests import pytest from report_generator.partner import sponsor class

    TestSponsor: @pytest.fixture(scope="class") def sponsors(self): return sponsor.get_all_sponsors("test/data/packages.yaml", “test/data/sponsors.yaml") ...... def test_sponsor_number(self, sponsors): assert len(sponsors) == 1 ......
  33. pytest style tests import pytest from report_generator.partner import sponsor class

    TestSponsor: @pytest.fixture(scope="class") def sponsors(self): return sponsor.get_all_sponsors("test/data/packages.yaml", “test/data/sponsors.yaml") ...... def test_sponsor_number(self, sponsors): assert len(sponsors) == 1 ......
  34. pytest style tests import pytest from report_generator.partner import sponsor @pytest.fixture(scope="class")

    def sponsors(): return sponsor.get_all_sponsors("test/data/packages.yaml", "test/data/sponsors.yaml") def test_sponsor_number(sponsors): assert len(sponsors) == 1
  35. pytest style tests import pytest from report_generator.partner import sponsor @pytest.fixture(scope="class")

    def sponsors(): return sponsor.get_all_sponsors("test/data/packages.yaml", "test/data/sponsors.yaml") def test_sponsor_number(sponsors): assert len(sponsors) == 1
  36. Pytest - powerful plugins • pytest-mock: Replace objects that are

    hard to test with fake objects • pytest-cov: Generate test coverage report • pytest-xdist: Distributed testing • and etc.
  37. Style check tools • Check syntax and coding style •

    Flake8 • Pylint • Static Typing • Mypy
  38. • Run Flake8 • Code to Check
 
 • Warning

    Run Flake8 import os os = "My Operating system" ./bad_code.py:4:1: F811 redefinition of unused 'os' from line 1 ./bad_code.py:4:5: E222 multiple spaces after operator flake8
  39. Config Flake8 - .flake8 [flake8] ignore = # F632: use

    ==/!= to compare str, bytes, and int literals F632, max-line-length = 119
  40. Config Pylint - .pylintrc • Run Pylint • Generate default

    config file
 • What I usually config in my .pylintrc pylint --generate-rcfile >> .pylintrc ... disable=print-statement, ... max-line-length=119 pylint <package> ......
  41. Flake8 v.s. Pylint • Flake8: Faster • Pylint: Stricter and

    more optional • How do I use them? • Enforce flake8 but run pylint for reference only
  42. Run mypy - code to check from typing import List

    def func(val: List[str]): print(val) func([1, 2, 3])
  43. Run mypy - Error Message wrong_type_hint.py:8: error: List item 0

    has incompatible type "int"; expected “str” wrong_type_hint.py:8: error: List item 1 has incompatible type "int"; expected “str" wrong_type_hint.py:8: error: List item 2 has incompatible type "int"; expected "str"
  44. –The Zen of Python, by Tim Peters “There should be

    one-- and preferably only one --obvious way to do it.”
  45. How I combine these tools pipenv run black -l 119

    <package> pipenv run flake8 pipenv run mypy -p <package> --ignore-missing-imports pipenv run pylint <package>
  46. Invoke • Task execution tool that can manage your commands.

    • It’s like Makefile in Python. • No more type or memorize long commands!
  47. How ? - tasks.py from invoke import task PIPENV_PREFIX =

    "pipenv run" ...... @task def install(cmd): """Install script in pipenv environment""" cmd.run(f"{PIPENV_PREFIX} python setup.py install")
  48. How ? - tasks.py from invoke import task PIPENV_PREFIX =

    "pipenv run" ...... @task def install(cmd): """Install script in pipenv environment""" cmd.run(f"{PIPENV_PREFIX} python setup.py install")
  49. How ? - tasks.py from invoke import task PIPENV_PREFIX =

    "pipenv run" ...... @task def install(cmd): """Install script in pipenv environment""" cmd.run(f"{PIPENV_PREFIX} python setup.py install")
  50. Modulize - Namespace #"" tasks $ #"" __init__.py $ #""

    build.py $ #"" common.py $ #"" env.py $ #"" style.py $ !"" test.py
  51. Why not use Makefile? 1.We’re Python developers! 2.Some tasks might

    not be easy to handle through shell script. 3.Python + Shell Script
 It’s the best of the both world!
  52. Other Commitizen features • Bump version automatically based on commits

    • Customizable commit rules • Auto generate changelog (WIP)
  53. How does Safety work? • Search the vulnerability in CVS

    
 (Common Vulnerabilities and Exposures) • Free DB: update monthly • Paid DB: update realtime
  54. Skip the warning in bandit • Add # nosec after

    the line code • Config in .bandit
  55. Why Drone • Plenty of Plugins • CI configuration as

    code • Easy to config • Container-based • Run on local easily ($ drone exec) • Easy to setup
  56. Config Drone - .drone.yml --- kind: pipeline type: docker name:

    default steps: - name: setup image: python:3.7-slim pull: if-not-exists volumes: - name: system_dependencies path: /usr/local - name: virtual_env path: /root/.local/share/virtualenvs commands: - pip install pipenv invoke - inv env.init-dev
 
 ... 
 volumes: - name: system_dependencies temp: {} - name: virtual_env temp: {}
  57. Config Drone - .drone.yml --- kind: pipeline type: docker name:

    default steps: - name: setup image: python:3.7-slim pull: if-not-exists volumes: - name: system_dependencies path: /usr/local - name: virtual_env path: /root/.local/share/virtualenvs commands: - pip install pipenv invoke - inv env.init-dev
 
 ... 
 volumes: - name: system_dependencies temp: {} - name: virtual_env temp: {}
  58. Config Drone - .drone.yml --- kind: pipeline type: docker name:

    default steps: - name: setup image: python:3.7-slim pull: if-not-exists volumes: - name: system_dependencies path: /usr/local - name: virtual_env path: /root/.local/share/virtualenvs commands: - pip install pipenv invoke - inv env.init-dev
 
 ... 
 volumes: - name: system_dependencies temp: {} - name: virtual_env temp: {}
  59. Config Drone - .drone.yml --- kind: pipeline type: docker name:

    default steps: - name: setup image: python:3.7-slim pull: if-not-exists volumes: - name: system_dependencies path: /usr/local - name: virtual_env path: /root/.local/share/virtualenvs commands: - pip install pipenv invoke - inv env.init-dev
 
 ... 
 volumes: - name: system_dependencies temp: {} - name: virtual_env temp: {}
  60. Config Drone - .drone.yml - name: check package security image:

    python:3.7-slim pull: if-not-exists volumes: - name: system_dependencies path: /usr/local - name: virtual_env path: /root/.local/share/virtualenvs commands: - inv secure depends_on: - setup
  61. Config Drone - .drone.yml - name: check package security image:

    python:3.7-slim pull: if-not-exists volumes: - name: system_dependencies path: /usr/local - name: virtual_env path: /root/.local/share/virtualenvs commands: - inv secure depends_on: - setup
  62. Config Drone - .drone.yml - name: check package security image:

    python:3.7-slim pull: if-not-exists volumes: - name: system_dependencies path: /usr/local - name: virtual_env path: /root/.local/share/virtualenvs commands: - inv secure depends_on: - setup
  63. What you need to do next 1.Push your code change

    to your remote git repo 2.Let drone check automatically 3. 4.Celebrate
  64. Conclusion 1. Initial project and install packages through pipenv 2.

    Write unit tests to ensure functionality correctness and run pytest 3. Format your code through black 4. Check coding style through flake8, pylint, mypy 5. Manage all the commands through invoke 6. Regulate commit message through commitizen
  65. Conclusion (Cont.) 7. Check vulnerability through Safety and bandit 8.

    Setup Drone CI on your remote git repo and let it check for you automatically
  66. Tools • Dependency Management • Poetry: Manage packages and support

    package releasing (replace setup.py) • pip-tools: Compile requirements to pin package versions and without leaving pip and venv
  67. Related Talks • Dependency Management • Tzu-ping Chung - 這樣的開發環境沒問題嗎?

    (PyCon TW 2018) • Kenneth Reitz - Pipenv: The Future of Python Dependency Management (PyCon US 2018) • Style Check • Kyle Knapp - Automating Code Quality - Kyle Knapp (PyCon US 2018) • Task Managements • Thea Flowers - Break the Cycle: Three excellent Python tools to automate repetitive tasks (PyCon US 2019)
  68. Related Talks • Test • Chun-Yu Tseng - 快快樂樂成為 Coding

    Ninja (by pytest) (PyCon APAC 2015) • Florian Bruhin – Pytest: Rapid Simple Testing (Swiss Python Summit 16) • Security • Terri Oda - Python Security Tools (PyCon US 2019) • Tennessee Leeuwenburg - Watch out for Safety Bandits! (PyCon AU 2018) • Multiple Topic • Dustin Ingram - Modern development environments for Pythonistas (PyCon JP 2019)