Python Table Manners - A Clean Style @ PyCon CA 2019

Transcript

1. Python Table Manners A Clean Style
2. Slide
3. Collaboration Note
4. $whoami Wei Lee / 李唯 Software Engineer @ Rakuten Slice Volunteer @ PyCon TW @clleew Lee-W http://lee-w.github.io
5. PyCon Tour 2019 (Unexpected) 1.PyCon US ! 2.PyCon JP " 3.PyCon TW # 4.PyCon CA $
6. PyCon Tour 2019 (Unexpected) 1.PyCon US ! ⇦ First PyCon Oversea 2.PyCon JP " 3.PyCon TW # 4.PyCon CA $
7. PyCon Tour 2019 (Unexpected) 1.PyCon US ! 2.PyCon JP " ⇦ First Lightning Talk 3.PyCon TW # 4.PyCon CA $
8. PyCon Tour 2019 (Unexpected) 1.PyCon US ! 2.PyCon JP " 3.PyCon TW # ⇦ First Volunteering 4.PyCon CA $
9. PyCon Tour 2019 (Unexpected) 1.PyCon US ! 2.PyCon JP " 3.PyCon TW # 4.PyCon CA $ ⇦ First Regular Talk in a PyCon!
10. What I’m going to talk about • A brief introduction of tools solving the following problems • Dependency Management • Testing • Style Check • Task Management • git commit • Security • Continuous Integration
11. Some examples come from rg-cli • pycontw/pycontw-postevent-report-generator • A PyCon TW attendees data analyser • Work in progress. Welcome to contribute
12. I won’t cover…
13. I won’t cover…
15. Dependency Management
16. How we used to start a Python project python -m venv ./venv source venv/bin/activate pip freeze >> requirements.txt
17. Looks good. Right?
18. Is this story familiar to you? Your code does not work. But, it worked on my machine.
19. Is this story familiar to you? Your code does not work. But, it worked on my machine. It seems you miss some dependencies.
20. Is this story familiar to you? Your code does not work. But, it worked on my machine. It seems you miss some dependencies. Ahh… I forgot to activate the virtual environment when I tested it…
22. Sometimes we just forgot to… • Activate / deactivate virtual environment • Add package into requirements.txt
23. Till…
24. Pipenv
 Python Dev Workflow for Humans
25. Why Pipenv? • Virtual environments + Package management • No more manual update dependencies
 We use Pipfile and Pipfile.lock .
26. Initial Virtual Environment # Initial pipenv # Create Pipfile and Pipfile.lock if not yet exist pipenv install
27. Empty Pipfile [[source]] name = "pypi" url = "https://pypi.org/simple" verify_ssl = true [dev-packages] [packages] [requires] python_version = "3.7"
28. Install Package # Install package pipenv install <package>==<version> # Uninstall package pipenv uninstall <package>
29. Pipfile with “requests” [[source]] name = "pypi" url = "https://pypi.org/simple" verify_ssl = true [dev-packages] [packages] requests = “*" [requires] python_version = "3.7"
30. Pipfile.lock with “requests” { ...... "default": { ...... "requests": { "hashes": [ "sha256:11e007a8a2aa0323f5a921e9e6a2d7e4e67d9877e85773fba9ba6419025cbeb4", "sha256:9cf5292fcd0f598c671cfc1e0d7d1a7f13bb8085e9a590f48c010551dc6c4b31" ], "index": "pypi", "version": "==2.22.0" }, "urllib3": { "hashes": [ "sha256:3de946ffbed6e6746608990594d08faac602528ac7015ac28d33cee6a45b7398", "sha256:9a107b99a5393caf59c7aa3c1249c16e6879447533d0887f4336dde834c7be86" ], "version": "==1.25.6" } }, ...... }
31. Pipfile.lock with “requests” { ...... "default": { ...... "requests": { "hashes": [ "sha256:11e007a8a2aa0323f5a921e9e6a2d7e4e67d9877e85773fba9ba6419025cbeb4", "sha256:9cf5292fcd0f598c671cfc1e0d7d1a7f13bb8085e9a590f48c010551dc6c4b31" ], "index": "pypi", "version": "==2.22.0" }, "urllib3": { "hashes": [ "sha256:3de946ffbed6e6746608990594d08faac602528ac7015ac28d33cee6a45b7398", "sha256:9a107b99a5393caf59c7aa3c1249c16e6879447533d0887f4336dde834c7be86" ], "version": "==1.25.6" } }, ...... } You’re guaranteed to install the exact same package next time.
33. Install package only in dev # Install dev only package pipenv install <package>==<version> --dev
34. Pipfile with “pytest" [[source]] name = "pypi" url = "https://pypi.org/simple" verify_ssl = true [dev-packages] pytest = "*" [packages] requests = "*" [requires] python_version = "3.7"
35. Run in virtual environment # Run command inside pipenv shell pipenv run <Command> # e.g. pipenv run python your_program.py
36. I don’t recommend using • Reason 1: You might accidentally install something inside the environment through 
 pip install • Reason 2: You can do almost anything through pipenv run <command> pipenv shell
37. Testing Don't let your customer debug for you
38. Add new features to a project without tests is like…
40. unittest 
 Unit testing framework in Python Standard Library
41. Pytest
42. Why pytest • More Pythonic • Compatible with old unittest tests • No boilerplate code required • No more assert.+ (e.g., assertEqual), just assert • Better test discovery • Advance features: mark, parameterize and etc. • Plenty of plugins
44. Running unittest in rg-cli python -m unittest discover -s ./ -p 'test_*.py' !"" test … #"" test_sponsor.py !"" test_title.py
45. Well… It’s not all unittest’s fault. • test directory need to be a package
 (even for pytest)
 
 
 
 • After that we can use python -m unittest !"" test #"" __init__.py ... #"" test_sponsor.py !"" test_title.py
46. Running pytest in rg-cli pytest
47. Running pytest in rg-cli pytest
48. Run test in pytest pipenv install pytest --dev pipenv run pytest After using pipenv (or any other virtual environment tool), you should always run in a virtual environment. Note that I won’t add the pipenv run prefix future pages.
49. unittest style tests import unittest from atta.partner import sponsor class TestSponsor(unittest.TestCase): def setUp(self): sponsors = sponsor.get_all_sponsors('./data/packages.yaml', './data/sponsors.yaml') self.sponsors = sponsors ...... def test_sponsor_number(self): self.assertEqual(len(self.sponsors), 1) .......
50. unittest style tests import unittest from atta.partner import sponsor class TestSponsor(unittest.TestCase): def setUp(self): sponsors = sponsor.get_all_sponsors('./data/packages.yaml', './data/sponsors.yaml') self.sponsors = sponsors ...... def test_sponsor_number(self): self.assertEqual(len(self.sponsors), 1) .......
51. unittest style tests import unittest from atta.partner import sponsor class TestSponsor(unittest.TestCase): def setUp(self): sponsors = sponsor.get_all_sponsors('./data/packages.yaml', './data/sponsors.yaml') self.sponsors = sponsors ...... def test_sponsor_number(self): self.assertEqual(len(self.sponsors), 1) .......
52. unittest style tests import unittest from atta.partner import sponsor class TestSponsor(unittest.TestCase): def setUp(self): sponsors = sponsor.get_all_sponsors('./data/packages.yaml', './data/sponsors.yaml') self.sponsors = sponsors ...... def test_sponsor_number(self): self.assertEqual(len(self.sponsors), 1) .......
53. pytest style tests import pytest from report_generator.partner import sponsor class TestSponsor: @pytest.fixture(scope="class") def sponsors(self): return sponsor.get_all_sponsors("test/data/packages.yaml", “test/data/sponsors.yaml") ...... def test_sponsor_number(self, sponsors): assert len(sponsors) == 1 ......
54. pytest style tests import pytest from report_generator.partner import sponsor class TestSponsor: @pytest.fixture(scope="class") def sponsors(self): return sponsor.get_all_sponsors("test/data/packages.yaml", “test/data/sponsors.yaml") ...... def test_sponsor_number(self, sponsors): assert len(sponsors) == 1 ......
55. pytest style tests import pytest from report_generator.partner import sponsor class TestSponsor: @pytest.fixture(scope="class") def sponsors(self): return sponsor.get_all_sponsors("test/data/packages.yaml", “test/data/sponsors.yaml") ...... def test_sponsor_number(self, sponsors): assert len(sponsors) == 1 ......
56. pytest style tests import pytest from report_generator.partner import sponsor @pytest.fixture(scope="class") def sponsors(): return sponsor.get_all_sponsors("test/data/packages.yaml", "test/data/sponsors.yaml") def test_sponsor_number(sponsors): assert len(sponsors) == 1
57. pytest style tests import pytest from report_generator.partner import sponsor @pytest.fixture(scope="class") def sponsors(): return sponsor.get_all_sponsors("test/data/packages.yaml", "test/data/sponsors.yaml") def test_sponsor_number(sponsors): assert len(sponsors) == 1
58. Pytest - powerful plugins • pytest-mock: Replace objects that are hard to test with fake objects • pytest-cov: Generate test coverage report • pytest-xdist: Distributed testing • and etc.
59. Style Check Check syntax correctness and make your code Pythonic
61. Style check tools • Check syntax and coding style • Flake8 • Pylint • Static Typing • Mypy
62. Flake8
63. • Run Flake8 • Code to Check
 
 • Warning Run Flake8 import os os = "My Operating system" ./bad_code.py:4:1: F811 redefinition of unused 'os' from line 1 ./bad_code.py:4:5: E222 multiple spaces after operator flake8
64. Config Flake8 - .flake8 [flake8] ignore = # F632: use ==/!= to compare str, bytes, and int literals F632, max-line-length = 119
65. Pylint
66. Config Pylint - .pylintrc • Run Pylint • Generate default config file
 • What I usually config in my .pylintrc pylint --generate-rcfile >> .pylintrc ... disable=print-statement, ... max-line-length=119 pylint <package> ......
67. Flake8 v.s. Pylint • Flake8: Faster • Pylint: Stricter and more optional • How do I use them? • Enforce flake8 but run pylint for reference only
68. Mypy
69. Run mypy mypy -p <package> -p <package> ...... --ignore-missing-imports
70. Run mypy - code to check from typing import List def func(val: List[str]): print(val) func([1, 2, 3])
71. Run mypy - Error Message wrong_type_hint.py:8: error: List item 0 has incompatible type "int"; expected “str” wrong_type_hint.py:8: error: List item 1 has incompatible type "int"; expected “str" wrong_type_hint.py:8: error: List item 2 has incompatible type "int"; expected "str"
72. One step forward Fix the style automatically
73. Black
74. Run Black black <package> ...
75. How Black reformat
76. How Black reformat
77. Why Black? • It’s not configurable.
78. Why Black? • It’s not configurable. No more arguing about which style is better
79. –The Zen of Python, by Tim Peters “There should be one-- and preferably only one --obvious way to do it.”
80. How I combine these tools pipenv run black -l 119 <package> pipenv run flake8 pipenv run mypy -p <package> --ignore-missing-imports pipenv run pylint <package>
82. Task Management Break the loop and repeat no more
83. Invoke • Task execution tool that can manage your commands. • It’s like Makefile in Python. • No more type or memorize long commands!
84. Before invoke in rg-cli
85. Before invoke in rg-cli
86. After invoke in rg-cli
87. List all commands inv -l
88. How ? - tasks.py from invoke import task PIPENV_PREFIX = "pipenv run" ...... @task def install(cmd): """Install script in pipenv environment""" cmd.run(f"{PIPENV_PREFIX} python setup.py install")
89. How ? - tasks.py from invoke import task PIPENV_PREFIX = "pipenv run" ...... @task def install(cmd): """Install script in pipenv environment""" cmd.run(f"{PIPENV_PREFIX} python setup.py install")
90. How ? - tasks.py from invoke import task PIPENV_PREFIX = "pipenv run" ...... @task def install(cmd): """Install script in pipenv environment""" cmd.run(f"{PIPENV_PREFIX} python setup.py install")
91. Modulize - Namespace #"" tasks $ #"" __init__.py $ #"" build.py $ #"" common.py $ #"" env.py $ #"" style.py $ !"" test.py
92. Autocomplete inv --print-completion-script=zsh >> ~/.zshrc
93. Autocomplete inv --print-completion-script=zsh >> ~/.zshrc
94. Why not use Makefile? 1.We’re Python developers! 2.Some tasks might not be easy to handle through shell script. 3.Python + Shell Script
 It’s the best of the both world!
95. Commit message A clear way back to the past
96. If you’re like him…
97. You’ll hate yourself when you need to roll back your code.
98. Commitizen
99. Commitizen in Python
100. Commit - cz commit type of this change
101. Commit - cz commit type of this change
102. Commit - cz commit scope of this change
103. Commit - cz commit
 a short description
104. Regulated git commits
105. Regulated git commits
106. Other Commitizen features • Bump version automatically based on commits • Customizable commit rules
107. Other Commitizen features • Bump version automatically based on commits • Customizable commit rules • Auto generate changelog (WIP)
108. Let’s Sprint!
110. Security
113. Safety Dependency Checking
114. Check vulnerability in current environment safety check
115. Check vulnerability in current environment safety check
116. But if you use pipenv … pipenv check
117. How does Safety work? • Search the vulnerability in CVS 
 (Common Vulnerabilities and Exposures) • Free DB: update monthly • Paid DB: update realtime
118. Bandit Static Analysis
119. Check Common Security Issue in Python Code bandit -r <package>
120. Check Common Security Issue in Python Code bandit -r <package>
121. Check Common Security Issue in Python Code bandit -r <package>
122. Check Common Security Issue in Python Code bandit -r <package>
123. Check Common Security Issue in Python Code bandit -r <package>
124. Not all the warnings should be fixed • Use bandit as a reference.
125. Skip the warning in bandit • Add # nosec after the line code • Config in .bandit
126. Continuous Integration Assemble all the trivial steps
127. drone
128. drone
129. Why Drone • Plenty of Plugins • CI configuration as code • Easy to config • Container-based • Run on local easily ($ drone exec) • Easy to setup
130. Install drone-cli
131. Drone in rg-cli
132. Steps
133. Config Drone - .drone.yml --- kind: pipeline type: docker name: default steps: - name: setup image: python:3.7-slim pull: if-not-exists volumes: - name: system_dependencies path: /usr/local - name: virtual_env path: /root/.local/share/virtualenvs commands: - pip install pipenv invoke - inv env.init-dev
 
 ... 
 volumes: - name: system_dependencies temp: {} - name: virtual_env temp: {}
134. Config Drone - .drone.yml --- kind: pipeline type: docker name: default steps: - name: setup image: python:3.7-slim pull: if-not-exists volumes: - name: system_dependencies path: /usr/local - name: virtual_env path: /root/.local/share/virtualenvs commands: - pip install pipenv invoke - inv env.init-dev
 
 ... 
 volumes: - name: system_dependencies temp: {} - name: virtual_env temp: {}
135. Config Drone - .drone.yml --- kind: pipeline type: docker name: default steps: - name: setup image: python:3.7-slim pull: if-not-exists volumes: - name: system_dependencies path: /usr/local - name: virtual_env path: /root/.local/share/virtualenvs commands: - pip install pipenv invoke - inv env.init-dev
 
 ... 
 volumes: - name: system_dependencies temp: {} - name: virtual_env temp: {}
136. Config Drone - .drone.yml --- kind: pipeline type: docker name: default steps: - name: setup image: python:3.7-slim pull: if-not-exists volumes: - name: system_dependencies path: /usr/local - name: virtual_env path: /root/.local/share/virtualenvs commands: - pip install pipenv invoke - inv env.init-dev
 
 ... 
 volumes: - name: system_dependencies temp: {} - name: virtual_env temp: {}
137. Config Drone - .drone.yml - name: check package security image: python:3.7-slim pull: if-not-exists volumes: - name: system_dependencies path: /usr/local - name: virtual_env path: /root/.local/share/virtualenvs commands: - inv secure depends_on: - setup
138. Config Drone - .drone.yml - name: check package security image: python:3.7-slim pull: if-not-exists volumes: - name: system_dependencies path: /usr/local - name: virtual_env path: /root/.local/share/virtualenvs commands: - inv secure depends_on: - setup
139. Config Drone - .drone.yml - name: check package security image: python:3.7-slim pull: if-not-exists volumes: - name: system_dependencies path: /usr/local - name: virtual_env path: /root/.local/share/virtualenvs commands: - inv secure depends_on: - setup
140. Run CI locally drone exec
141. What you need to do next 1.Push your code change to your remote git repo 2.Let drone check automatically 3. 4.Celebrate
143. Conclusion 1. Initial project and install packages through pipenv 2. Write unit tests to ensure functionality correctness and run pytest 3. Format your code through black 4. Check coding style through flake8, pylint, mypy 5. Manage all the commands through invoke 6. Regulate commit message through commitizen
144. Conclusion (Cont.) 7. Check vulnerability through Safety and bandit 8. Setup Drone CI on your remote git repo and let it check for you automatically
146. Tools • Dependency Management • Poetry: Manage packages and support package releasing (replace setup.py) • pip-tools: Compile requirements to pin package versions and without leaving pip and venv
147. Tools • Testing • Hypothesis • nox • Project Creation • Cookiecutter • PyScaffold
148. Related Talks • Dependency Management • Tzu-ping Chung - 這樣的開發環境沒問題嗎？ (PyCon TW 2018) • Kenneth Reitz - Pipenv: The Future of Python Dependency Management (PyCon US 2018) • Style Check • Kyle Knapp - Automating Code Quality - Kyle Knapp (PyCon US 2018) • Task Managements • Thea Flowers - Break the Cycle: Three excellent Python tools to automate repetitive tasks (PyCon US 2019)
149. Related Talks • Test • Chun-Yu Tseng - 快快樂樂成為 Coding Ninja (by pytest) (PyCon APAC 2015) • Florian Bruhin – Pytest: Rapid Simple Testing (Swiss Python Summit 16) • Security • Terri Oda - Python Security Tools (PyCon US 2019) • Tennessee Leeuwenburg - Watch out for Safety Bandits! (PyCon AU 2018) • Multiple Topic • Dustin Ingram - Modern development environments for Pythonistas (PyCon JP 2019)