Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Python Table Manners - A Clean Style @ PyCon CA 2019

3e049ed33195d00a8b745b16c63dce6e?s=47 Lee Wei
November 17, 2019

Python Table Manners - A Clean Style @ PyCon CA 2019

3e049ed33195d00a8b745b16c63dce6e?s=128

Lee Wei

November 17, 2019
Tweet

Transcript

  1. Python Table Manners A Clean Style

  2. Slide

  3. Collaboration Note

  4. $whoami Wei Lee / 李唯 Software Engineer @ Rakuten Slice

    Volunteer @ PyCon TW @clleew Lee-W http://lee-w.github.io
  5. PyCon Tour 2019 (Unexpected) 1.PyCon US ! 2.PyCon JP "

    3.PyCon TW # 4.PyCon CA $
  6. PyCon Tour 2019 (Unexpected) 1.PyCon US ! ⇦ First PyCon

    Oversea 2.PyCon JP " 3.PyCon TW # 4.PyCon CA $
  7. PyCon Tour 2019 (Unexpected) 1.PyCon US ! 2.PyCon JP "

    ⇦ First Lightning Talk 3.PyCon TW # 4.PyCon CA $
  8. PyCon Tour 2019 (Unexpected) 1.PyCon US ! 2.PyCon JP "

    3.PyCon TW # ⇦ First Volunteering 4.PyCon CA $
  9. PyCon Tour 2019 (Unexpected) 1.PyCon US ! 2.PyCon JP "

    3.PyCon TW # 4.PyCon CA $ ⇦ First Regular Talk in a PyCon!
  10. What I’m going to talk about • A brief introduction

    of tools solving the following problems • Dependency Management • Testing • Style Check • Task Management • git commit • Security • Continuous Integration
  11. Some examples come from rg-cli • pycontw/pycontw-postevent-report-generator • A PyCon

    TW attendees data analyser • Work in progress. Welcome to contribute
  12. I won’t cover…

  13. I won’t cover…

  14. None
  15. Dependency Management

  16. How we used to start a Python project python -m

    venv ./venv source venv/bin/activate pip freeze >> requirements.txt
  17. Looks good. Right?

  18. Is this story familiar to you? Your code does not

    work. But, it worked on my machine.
  19. Is this story familiar to you? Your code does not

    work. But, it worked on my machine. It seems you miss some dependencies.
  20. Is this story familiar to you? Your code does not

    work. But, it worked on my machine. It seems you miss some dependencies. Ahh… I forgot to activate the virtual environment when I tested it…
  21. None
  22. Sometimes we just forgot to… • Activate / deactivate virtual

    environment • Add package into requirements.txt
  23. Till…

  24. Pipenv
 Python Dev Workflow for Humans

  25. Why Pipenv? • Virtual environments + Package management • No

    more manual update dependencies
 We use Pipfile and Pipfile.lock .
  26. Initial Virtual Environment # Initial pipenv # Create Pipfile and

    Pipfile.lock if not yet exist pipenv install
  27. Empty Pipfile [[source]] name = "pypi" url = "https://pypi.org/simple" verify_ssl

    = true [dev-packages] [packages] [requires] python_version = "3.7"
  28. Install Package # Install package pipenv install <package>==<version> # Uninstall

    package pipenv uninstall <package>
  29. Pipfile with “requests” [[source]] name = "pypi" url = "https://pypi.org/simple"

    verify_ssl = true [dev-packages] [packages] requests = “*" [requires] python_version = "3.7"
  30. Pipfile.lock with “requests” { ...... "default": { ...... "requests": {

    "hashes": [ "sha256:11e007a8a2aa0323f5a921e9e6a2d7e4e67d9877e85773fba9ba6419025cbeb4", "sha256:9cf5292fcd0f598c671cfc1e0d7d1a7f13bb8085e9a590f48c010551dc6c4b31" ], "index": "pypi", "version": "==2.22.0" }, "urllib3": { "hashes": [ "sha256:3de946ffbed6e6746608990594d08faac602528ac7015ac28d33cee6a45b7398", "sha256:9a107b99a5393caf59c7aa3c1249c16e6879447533d0887f4336dde834c7be86" ], "version": "==1.25.6" } }, ...... }
  31. Pipfile.lock with “requests” { ...... "default": { ...... "requests": {

    "hashes": [ "sha256:11e007a8a2aa0323f5a921e9e6a2d7e4e67d9877e85773fba9ba6419025cbeb4", "sha256:9cf5292fcd0f598c671cfc1e0d7d1a7f13bb8085e9a590f48c010551dc6c4b31" ], "index": "pypi", "version": "==2.22.0" }, "urllib3": { "hashes": [ "sha256:3de946ffbed6e6746608990594d08faac602528ac7015ac28d33cee6a45b7398", "sha256:9a107b99a5393caf59c7aa3c1249c16e6879447533d0887f4336dde834c7be86" ], "version": "==1.25.6" } }, ...... } You’re guaranteed to install the exact same package next time.
  32. None
  33. Install package only in dev # Install dev only package

    pipenv install <package>==<version> --dev
  34. Pipfile with “pytest" [[source]] name = "pypi" url = "https://pypi.org/simple"

    verify_ssl = true [dev-packages] pytest = "*" [packages] requests = "*" [requires] python_version = "3.7"
  35. Run in virtual environment # Run command inside pipenv shell

    pipenv run <Command> # e.g. pipenv run python your_program.py
  36. I don’t recommend using • Reason 1: You might accidentally

    install something inside the environment through 
 pip install • Reason 2: You can do almost anything through pipenv run <command> pipenv shell
  37. Testing Don't let your customer debug for you

  38. Add new features to a project without tests is like…

  39. None
  40. unittest 
 Unit testing framework in Python Standard Library

  41. Pytest

  42. Why pytest • More Pythonic • Compatible with old unittest

    tests • No boilerplate code required • No more assert.+ (e.g., assertEqual), just assert • Better test discovery • Advance features: mark, parameterize and etc. • Plenty of plugins
  43. None
  44. Running unittest in rg-cli python -m unittest discover -s ./

    -p 'test_*.py' !"" test … #"" test_sponsor.py !"" test_title.py
  45. Well… It’s not all unittest’s fault. • test directory need

    to be a package
 (even for pytest)
 
 
 
 • After that we can use python -m unittest !"" test #"" __init__.py ... #"" test_sponsor.py !"" test_title.py
  46. Running pytest in rg-cli pytest

  47. Running pytest in rg-cli pytest

  48. Run test in pytest pipenv install pytest --dev pipenv run

    pytest After using pipenv (or any other virtual environment tool), you should always run in a virtual environment. Note that I won’t add the pipenv run prefix future pages.
  49. unittest style tests import unittest from atta.partner import sponsor class

    TestSponsor(unittest.TestCase): def setUp(self): sponsors = sponsor.get_all_sponsors('./data/packages.yaml', './data/sponsors.yaml') self.sponsors = sponsors ...... def test_sponsor_number(self): self.assertEqual(len(self.sponsors), 1) .......
  50. unittest style tests import unittest from atta.partner import sponsor class

    TestSponsor(unittest.TestCase): def setUp(self): sponsors = sponsor.get_all_sponsors('./data/packages.yaml', './data/sponsors.yaml') self.sponsors = sponsors ...... def test_sponsor_number(self): self.assertEqual(len(self.sponsors), 1) .......
  51. unittest style tests import unittest from atta.partner import sponsor class

    TestSponsor(unittest.TestCase): def setUp(self): sponsors = sponsor.get_all_sponsors('./data/packages.yaml', './data/sponsors.yaml') self.sponsors = sponsors ...... def test_sponsor_number(self): self.assertEqual(len(self.sponsors), 1) .......
  52. unittest style tests import unittest from atta.partner import sponsor class

    TestSponsor(unittest.TestCase): def setUp(self): sponsors = sponsor.get_all_sponsors('./data/packages.yaml', './data/sponsors.yaml') self.sponsors = sponsors ...... def test_sponsor_number(self): self.assertEqual(len(self.sponsors), 1) .......
  53. pytest style tests import pytest from report_generator.partner import sponsor class

    TestSponsor: @pytest.fixture(scope="class") def sponsors(self): return sponsor.get_all_sponsors("test/data/packages.yaml", “test/data/sponsors.yaml") ...... def test_sponsor_number(self, sponsors): assert len(sponsors) == 1 ......
  54. pytest style tests import pytest from report_generator.partner import sponsor class

    TestSponsor: @pytest.fixture(scope="class") def sponsors(self): return sponsor.get_all_sponsors("test/data/packages.yaml", “test/data/sponsors.yaml") ...... def test_sponsor_number(self, sponsors): assert len(sponsors) == 1 ......
  55. pytest style tests import pytest from report_generator.partner import sponsor class

    TestSponsor: @pytest.fixture(scope="class") def sponsors(self): return sponsor.get_all_sponsors("test/data/packages.yaml", “test/data/sponsors.yaml") ...... def test_sponsor_number(self, sponsors): assert len(sponsors) == 1 ......
  56. pytest style tests import pytest from report_generator.partner import sponsor @pytest.fixture(scope="class")

    def sponsors(): return sponsor.get_all_sponsors("test/data/packages.yaml", "test/data/sponsors.yaml") def test_sponsor_number(sponsors): assert len(sponsors) == 1
  57. pytest style tests import pytest from report_generator.partner import sponsor @pytest.fixture(scope="class")

    def sponsors(): return sponsor.get_all_sponsors("test/data/packages.yaml", "test/data/sponsors.yaml") def test_sponsor_number(sponsors): assert len(sponsors) == 1
  58. Pytest - powerful plugins • pytest-mock: Replace objects that are

    hard to test with fake objects • pytest-cov: Generate test coverage report • pytest-xdist: Distributed testing • and etc.
  59. Style Check Check syntax correctness and make your code Pythonic

  60. None
  61. Style check tools • Check syntax and coding style •

    Flake8 • Pylint • Static Typing • Mypy
  62. Flake8

  63. • Run Flake8 • Code to Check
 
 • Warning

    Run Flake8 import os os = "My Operating system" ./bad_code.py:4:1: F811 redefinition of unused 'os' from line 1 ./bad_code.py:4:5: E222 multiple spaces after operator flake8
  64. Config Flake8 - .flake8 [flake8] ignore = # F632: use

    ==/!= to compare str, bytes, and int literals F632, max-line-length = 119
  65. Pylint

  66. Config Pylint - .pylintrc • Run Pylint • Generate default

    config file
 • What I usually config in my .pylintrc pylint --generate-rcfile >> .pylintrc ... disable=print-statement, ... max-line-length=119 pylint <package> ......
  67. Flake8 v.s. Pylint • Flake8: Faster • Pylint: Stricter and

    more optional • How do I use them? • Enforce flake8 but run pylint for reference only
  68. Mypy

  69. Run mypy mypy -p <package> -p <package> ...... --ignore-missing-imports

  70. Run mypy - code to check from typing import List

    def func(val: List[str]): print(val) func([1, 2, 3])
  71. Run mypy - Error Message wrong_type_hint.py:8: error: List item 0

    has incompatible type "int"; expected “str” wrong_type_hint.py:8: error: List item 1 has incompatible type "int"; expected “str" wrong_type_hint.py:8: error: List item 2 has incompatible type "int"; expected "str"
  72. One step forward Fix the style automatically

  73. Black

  74. Run Black black <package> ...

  75. How Black reformat

  76. How Black reformat

  77. Why Black? • It’s not configurable.

  78. Why Black? • It’s not configurable. No more arguing about

    which style is better
  79. –The Zen of Python, by Tim Peters “There should be

    one-- and preferably only one --obvious way to do it.”
  80. How I combine these tools pipenv run black -l 119

    <package> pipenv run flake8 pipenv run mypy -p <package> --ignore-missing-imports pipenv run pylint <package>
  81. None
  82. Task Management Break the loop and repeat no more

  83. Invoke • Task execution tool that can manage your commands.

    • It’s like Makefile in Python. • No more type or memorize long commands!
  84. Before invoke in rg-cli

  85. Before invoke in rg-cli

  86. After invoke in rg-cli

  87. List all commands inv -l

  88. How ? - tasks.py from invoke import task PIPENV_PREFIX =

    "pipenv run" ...... @task def install(cmd): """Install script in pipenv environment""" cmd.run(f"{PIPENV_PREFIX} python setup.py install")
  89. How ? - tasks.py from invoke import task PIPENV_PREFIX =

    "pipenv run" ...... @task def install(cmd): """Install script in pipenv environment""" cmd.run(f"{PIPENV_PREFIX} python setup.py install")
  90. How ? - tasks.py from invoke import task PIPENV_PREFIX =

    "pipenv run" ...... @task def install(cmd): """Install script in pipenv environment""" cmd.run(f"{PIPENV_PREFIX} python setup.py install")
  91. Modulize - Namespace #"" tasks $ #"" __init__.py $ #""

    build.py $ #"" common.py $ #"" env.py $ #"" style.py $ !"" test.py
  92. Autocomplete inv --print-completion-script=zsh >> ~/.zshrc

  93. Autocomplete inv --print-completion-script=zsh >> ~/.zshrc

  94. Why not use Makefile? 1.We’re Python developers! 2.Some tasks might

    not be easy to handle through shell script. 3.Python + Shell Script
 It’s the best of the both world!
  95. Commit message A clear way back to the past

  96. If you’re like him…

  97. You’ll hate yourself when you need to roll back your

    code.
  98. Commitizen

  99. Commitizen in Python

  100. Commit - cz commit type of this change

  101. Commit - cz commit type of this change

  102. Commit - cz commit scope of this change

  103. Commit - cz commit
 a short description

  104. Regulated git commits

  105. Regulated git commits

  106. Other Commitizen features • Bump version automatically based on commits

    • Customizable commit rules
  107. Other Commitizen features • Bump version automatically based on commits

    • Customizable commit rules • Auto generate changelog (WIP)
  108. Let’s Sprint!

  109. None
  110. Security

  111. None
  112. None
  113. Safety Dependency Checking

  114. Check vulnerability in current environment safety check

  115. Check vulnerability in current environment safety check

  116. But if you use pipenv … pipenv check

  117. How does Safety work? • Search the vulnerability in CVS

    
 (Common Vulnerabilities and Exposures) • Free DB: update monthly • Paid DB: update realtime
  118. Bandit Static Analysis

  119. Check Common Security Issue in Python Code bandit -r <package>

  120. Check Common Security Issue in Python Code bandit -r <package>

  121. Check Common Security Issue in Python Code bandit -r <package>

  122. Check Common Security Issue in Python Code bandit -r <package>

  123. Check Common Security Issue in Python Code bandit -r <package>

  124. Not all the warnings should be fixed • Use bandit

    as a reference.
  125. Skip the warning in bandit • Add # nosec after

    the line code • Config in .bandit
  126. Continuous Integration Assemble all the trivial steps

  127. drone

  128. drone

  129. Why Drone • Plenty of Plugins • CI configuration as

    code • Easy to config • Container-based • Run on local easily ($ drone exec) • Easy to setup
  130. Install drone-cli

  131. Drone in rg-cli

  132. Steps

  133. Config Drone - .drone.yml --- kind: pipeline type: docker name:

    default steps: - name: setup image: python:3.7-slim pull: if-not-exists volumes: - name: system_dependencies path: /usr/local - name: virtual_env path: /root/.local/share/virtualenvs commands: - pip install pipenv invoke - inv env.init-dev
 
 ... 
 volumes: - name: system_dependencies temp: {} - name: virtual_env temp: {}
  134. Config Drone - .drone.yml --- kind: pipeline type: docker name:

    default steps: - name: setup image: python:3.7-slim pull: if-not-exists volumes: - name: system_dependencies path: /usr/local - name: virtual_env path: /root/.local/share/virtualenvs commands: - pip install pipenv invoke - inv env.init-dev
 
 ... 
 volumes: - name: system_dependencies temp: {} - name: virtual_env temp: {}
  135. Config Drone - .drone.yml --- kind: pipeline type: docker name:

    default steps: - name: setup image: python:3.7-slim pull: if-not-exists volumes: - name: system_dependencies path: /usr/local - name: virtual_env path: /root/.local/share/virtualenvs commands: - pip install pipenv invoke - inv env.init-dev
 
 ... 
 volumes: - name: system_dependencies temp: {} - name: virtual_env temp: {}
  136. Config Drone - .drone.yml --- kind: pipeline type: docker name:

    default steps: - name: setup image: python:3.7-slim pull: if-not-exists volumes: - name: system_dependencies path: /usr/local - name: virtual_env path: /root/.local/share/virtualenvs commands: - pip install pipenv invoke - inv env.init-dev
 
 ... 
 volumes: - name: system_dependencies temp: {} - name: virtual_env temp: {}
  137. Config Drone - .drone.yml - name: check package security image:

    python:3.7-slim pull: if-not-exists volumes: - name: system_dependencies path: /usr/local - name: virtual_env path: /root/.local/share/virtualenvs commands: - inv secure depends_on: - setup
  138. Config Drone - .drone.yml - name: check package security image:

    python:3.7-slim pull: if-not-exists volumes: - name: system_dependencies path: /usr/local - name: virtual_env path: /root/.local/share/virtualenvs commands: - inv secure depends_on: - setup
  139. Config Drone - .drone.yml - name: check package security image:

    python:3.7-slim pull: if-not-exists volumes: - name: system_dependencies path: /usr/local - name: virtual_env path: /root/.local/share/virtualenvs commands: - inv secure depends_on: - setup
  140. Run CI locally drone exec

  141. What you need to do next 1.Push your code change

    to your remote git repo 2.Let drone check automatically 3. 4.Celebrate
  142. None
  143. Conclusion 1. Initial project and install packages through pipenv 2.

    Write unit tests to ensure functionality correctness and run pytest 3. Format your code through black 4. Check coding style through flake8, pylint, mypy 5. Manage all the commands through invoke 6. Regulate commit message through commitizen
  144. Conclusion (Cont.) 7. Check vulnerability through Safety and bandit 8.

    Setup Drone CI on your remote git repo and let it check for you automatically
  145. None
  146. Tools • Dependency Management • Poetry: Manage packages and support

    package releasing (replace setup.py) • pip-tools: Compile requirements to pin package versions and without leaving pip and venv
  147. Tools • Testing • Hypothesis • nox • Project Creation

    • Cookiecutter • PyScaffold
  148. Related Talks • Dependency Management • Tzu-ping Chung - 這樣的開發環境沒問題嗎?

    (PyCon TW 2018) • Kenneth Reitz - Pipenv: The Future of Python Dependency Management (PyCon US 2018) • Style Check • Kyle Knapp - Automating Code Quality - Kyle Knapp (PyCon US 2018) • Task Managements • Thea Flowers - Break the Cycle: Three excellent Python tools to automate repetitive tasks (PyCon US 2019)
  149. Related Talks • Test • Chun-Yu Tseng - 快快樂樂成為 Coding

    Ninja (by pytest) (PyCon APAC 2015) • Florian Bruhin – Pytest: Rapid Simple Testing (Swiss Python Summit 16) • Security • Terri Oda - Python Security Tools (PyCon US 2019) • Tennessee Leeuwenburg - Watch out for Safety Bandits! (PyCon AU 2018) • Multiple Topic • Dustin Ingram - Modern development environments for Pythonistas (PyCon JP 2019)
  150. None