Pipfile, pipenv, pip… what?!

Иван Чернов (Ostrovok.ru) @ Moscow Python Meetup 50

"Python имеет всеми нами любимый пакетный менеджер pip, но прогресс не стоит на месте и сообщество сделало новый удобный иструмент pipenv. Давайте посмотрим на него и ответим пора ли на него переходить?"
Видео: http://www.moscowpython.ru/meetup/50/pip-what/

Transcript

1. Pipfile, pipenv, pip… what?! Ivan Chernov, November 2017
2. About me • Python developer @ Ostrovok.ru • Maintain and develop: ◦ Site backend ◦ Email marketing platform ◦ Booking service
3. Agenda • Your project workflow • Improve it with <spoiler> • Why we need Pipfile? • Quick intro to pipenv • Summary
4. Setup your project $ pip install django
5. Setup your project $ pip install django $ pip freeze > requirements.txt
6. Setup your project $ pip install django $ pip freeze > requirements.txt $ cat requirements.txt
7. Setup your project $ pip install django $ pip freeze > requirements.txt $ cat requirements.txt ... docker-compose==1.16.1 docker-pycreds==0.2.1 Django==1.11.5 ...
8. >_< Virtualenv first $ python -m venv .venv
9. >_< Virtualenv first $ python -m venv .venv $ . .venv/bin/activate
10. >_< Virtualenv first $ python -m venv .venv $ . .venv/bin/activate (.venv) $ pip install django
11. >_< Virtualenv first $ python -m venv .venv $ . .venv/bin/activate (.venv) $ pip install django (.venv) $ pip freeze > requirements.txt
12. >_< Virtualenv first $ python -m venv .venv $ . .venv/bin/activate (.venv) $ pip install django (.venv) $ pip freeze > requirements.txt (.venv) $ cat requirements.txt
13. >_< Virtualenv first $ python -m venv .venv $ . .venv/bin/activate (.venv) $ pip install django (.venv) $ pip freeze > requirements.txt (.venv) $ cat requirements.txt Django==1.11.5 pytz==2017.2
15. How many deps? $ cat requirements.txt | wc -l
16. Too many... $ cat requirements.txt | wc -l 125
17. Deps question I $ cat requirements.txt | wc -l 125 • Are all of deps pinned?
18. Deps question II $ cat requirements.txt | wc -l 125 • Are all of deps pinned? • No conflicts in sub-deps?
19. Deps question III $ cat requirements.txt | wc -l 125 • Are all of deps pinned? • No conflicts in sub-deps? • How to view dependency tree?
20. Deps question IV $ cat requirements.txt | wc -l 125 • Are all of deps pinned? • No conflicts in sub-deps? • How to view dependency tree? • How to divide prod/dev/test deps?
21. Use pipenv pip-tools
22. Intro to pip-tools $ [sudo] pip install pip-tools
23. Intro to pip-tools $ [sudo] pip install pip-tools $ echo django >> requirements.in
24. Intro to pip-tools $ [sudo] pip install pip-tools $ echo django >> requirements.in $ pip-compile --output-file requirements.txt requirements.in
25. Intro to pip-tools $ [sudo] pip install pip-tools $ echo django >> requirements.in $ pip-compile --output-file requirements.txt requirements.in $ cat requirements.txt
26. Intro to pip-tools $ [sudo] pip install pip-tools $ echo django >> requirements.in $ pip-compile --output-file requirements.txt requirements.in $ cat requirements.txt django==1.11.5 pytz==2017.2 # via django
27. Intro to pip-tools $ [sudo] pip install pip-tools $ echo django >> requirements.in $ pip-compile --output-file requirements.txt requirements.in $ cat requirements.txt django==1.11.5 pytz==2017.2 # via django
29. Problem solved(?)
30. Moving to libraries • We have setuptools
31. Moving to libraries • We have setuptools • Which is used in setup.py
32. Moving to libraries • We have setuptools • Which is used in setup.py • And what if setup.py contains external deps?
33. Moving to libraries • We have setuptools • Which is used in setup.py • And what if setup.py contains external deps? • Pip will fail to install
34. Pipfile (PEP 518) • Rationale: give dev ability to specify build system • Implementation: ◦ toml file in root called Pipfile ◦ json file for locking Pipfile.lock ◦ WIP installation through pip install -p
35. Rejected formats (from PEP) • JSON - human-readable, but not human-editable
36. Rejected formats (from PEP) • JSON - human-readable, but not human-editable • Configparser - Python stdlib, but confusing
37. Rejected formats (from PEP) • JSON - human-readable, but not human-editable • Configparser - Python stdlib, but confusing • Python literals (dict) - Cannot be parsed in other languages
38. Rejected formats (from PEP) • JSON - human-readable, but not human-editable • Configparser - Python stdlib, but confusing • Python literals (dict) - Cannot be parsed in other languages • YAML
39. Rejected formats (from PEP) • JSON - human-readable, but not human-editable • Configparser - Python stdlib, but confusing • Python literals (dict) - Cannot be parsed in other languages • YAML ◦ Specification == 86 pages (== too difficult)
40. Rejected formats (from PEP) • JSON - human-readable, but not human-editable • Configparser - Python stdlib, but confusing • Python literals (dict) - Cannot be parsed in other languages • YAML ◦ Specification == 86 pages (== too difficult) ◦ Is not safe by default (command execution vuln.)
41. Rejected formats (from PEP) • JSON - human-readable, but not human-editable • Configparser - Python stdlib, but confusing • Python literals (dict) - Cannot be parsed in other languages • YAML ◦ Specification == 86 pages (== too difficult) ◦ Is not safe by default (command execution vuln.) ◦ Pip will need to vendor PyYAML (~1k LOC Python & C code)
42. Pipfile I [[source]] url = 'https://pypi.python.org/simple' verify_ssl = true name = 'pypi'
43. Pipfile I [[source]] url = 'https://pypi.python.org/simple' verify_ssl = true name = 'pypi'
44. Pipfile I [[source]] url = 'https://pypi.python.org/simple' verify_ssl = true name = 'pypi'
45. Pipfile II [packages] requests = '>2' django = { git = 'https://github.com/django/django.git', ref = '1.11.5', editable = true } pywinusb = { version = "*", os_name = "=='nt'", index="pypi"} [dev-packages] pytest = '*'
46. Pipfile II [packages] requests = '>2' django = { git = 'https://github.com/django/django.git', ref = '1.11.5', editable = true } pywinusb = { version = "*", os_name = "=='nt'", index="pypi"} [dev-packages] pytest = '*'
47. Pipfile II [packages] requests = '>2' django = { git = 'https://github.com/django/django.git', ref = '1.11.5', editable = true } pywinusb = { version = "*", os_name = "=='nt'", index="pypi"} [dev-packages] pytest = '*'
48. Pipfile II [packages] requests = '>2' django = { git = 'https://github.com/django/django.git', ref = '1.11.5', editable = true } pywinusb = { version = "*", os_name = "=='nt'", index="pypi"} [dev-packages] pytest = '*'
49. Pipfile II [packages] requests = '>2' django = { git = 'https://github.com/django/django.git', ref = '1.11.5', editable = true } pywinusb = { version = "*", os_name = "=='nt'", index="pypi"} [dev-packages] pytest = '*'
50. How to use it?
51. [sudo] pip install pipenv
52. What is Pipenv? • Sacred Marriage of Pipfile, Pip, Pip-tools, & Virtualenv. • From Kenneth Reitz: requests, autoenv • Automatically manages ◦ Pipfile ◦ Pipfile.lock ◦ virtualenv
53. Project with requirements.txt $ cd your/project/dir $ pipenv install Requirements.txt found, instead of Pipfile! Converting…
54. Custom pypi fail :( $ cd your/project/dir $ pipenv install Requirements.txt found, instead of Pipfile! Converting… # but for custom pypi, there will be error on lock :( $ sed -i s/pypi.python.org/pypi.example.org/g Pipfile
55. $ cd your/project/dir $ pipenv run python django-admin startproject How to use? I
56. $ cd your/project/dir $ pipenv shell (.venv) $ django-admin How to use? II
57. How to use with Docker? $ pipenv lock # In Dockerfile WORKDIR /your/dir/ COPY Pipfile Pipfile.lock /your/dir/ RUN pipenv install --system # to install libs in system folder
58. Summary Pros • All your deps are locked Cons
59. Summary Pros • All your deps are locked • All your venvs are in .local Cons
60. Summary Pros • All your deps are locked • All your venvs are in .local • Ability to separate dev/prod Cons
61. Summary Pros • All your deps are locked • All your venvs are in .local • Ability to separate dev/prod • Pip will support this file Cons
62. Summary Pros • All your deps are locked • All your venvs are in .local • Ability to separate dev/prod • Pip will support this file Cons • Not mature :(
63. Summary Pros • All your deps are locked • All your venvs are in .local • Ability to separate dev/prod • Pip will support this file Cons • Not mature :( • Dockerfile will miss cache
64. Summary Pros • All your deps are locked • All your venvs are in .local • Ability to separate dev/prod • Pip will support this file Cons • Not mature :( • Dockerfile will miss cache • Harder to update package
65. Links • PEP https://www.python.org/dev/peps/pep-0518/ • Pipfile repo https://github.com/pypa/pipfile • Pipenv repo https://github.com/pypa/pipfile • Cheetsheet https://robots.thoughtbot.com/how-to-manage-your-python-projects-with- pipenv
66. Hiring! Yes, t.me/vanadium23
67. Questions? P.S. follow me → @vanadium23