Pipfile, pipenv, pip… what?!

Иван Чернов (Ostrovok.ru) @ Moscow Python Meetup 50

"Python имеет всеми нами любимый пакетный менеджер pip, но прогресс не стоит на месте и сообщество сделало новый удобный иструмент pipenv. Давайте посмотрим на него и ответим пора ли на него переходить?"
Видео: http://www.moscowpython.ru/meetup/50/pip-what/

Transcript

1. Pipfile, pipenv, pip…
   what?!
   Ivan Chernov, November 2017
   

   View Slide

2. About me
   ● Python developer @ Ostrovok.ru
   ● Maintain and develop:
   ○ Site backend
   ○ Email marketing platform
   ○ Booking service
   

   View Slide

3. Agenda
   ● Your project workflow
   ● Improve it with
   ● Why we need Pipfile?
   ● Quick intro to pipenv
   ● Summary
   

   View Slide

4. Setup your project
   $ pip install django
   

   View Slide

5. Setup your project
   $ pip install django
   $ pip freeze > requirements.txt
   

   View Slide

6. Setup your project
   $ pip install django
   $ pip freeze > requirements.txt
   $ cat requirements.txt
   

   View Slide

7. Setup your project
   $ pip install django
   $ pip freeze > requirements.txt
   $ cat requirements.txt
   ...
   docker-compose==1.16.1
   docker-pycreds==0.2.1
   Django==1.11.5
   ...
   

   View Slide

8. >_< Virtualenv first
   $ python -m venv .venv
   

   View Slide

9. >_< Virtualenv first
   $ python -m venv .venv
   $ . .venv/bin/activate
   

   View Slide

10. >_< Virtualenv first
    $ python -m venv .venv
    $ . .venv/bin/activate
    (.venv) $ pip install django
    

    View Slide

11. >_< Virtualenv first
    $ python -m venv .venv
    $ . .venv/bin/activate
    (.venv) $ pip install django
    (.venv) $ pip freeze > requirements.txt
    

    View Slide

12. >_< Virtualenv first
    $ python -m venv .venv
    $ . .venv/bin/activate
    (.venv) $ pip install django
    (.venv) $ pip freeze > requirements.txt
    (.venv) $ cat requirements.txt
    

    View Slide

13. >_< Virtualenv first
    $ python -m venv .venv
    $ . .venv/bin/activate
    (.venv) $ pip install django
    (.venv) $ pip freeze > requirements.txt
    (.venv) $ cat requirements.txt
    Django==1.11.5
    pytz==2017.2
    

    View Slide

14. View Slide

15. How many deps?
    $ cat requirements.txt | wc -l
    

    View Slide

16. Too many...
    $ cat requirements.txt | wc -l
    125
    

    View Slide

17. Deps question I
    $ cat requirements.txt | wc -l
    125
    ● Are all of deps pinned?
    

    View Slide

18. Deps question II
    $ cat requirements.txt | wc -l
    125
    ● Are all of deps pinned?
    ● No conflicts in sub-deps?
    

    View Slide

19. Deps question III
    $ cat requirements.txt | wc -l
    125
    ● Are all of deps pinned?
    ● No conflicts in sub-deps?
    ● How to view dependency tree?
    

    View Slide

20. Deps question IV
    $ cat requirements.txt | wc -l
    125
    ● Are all of deps pinned?
    ● No conflicts in sub-deps?
    ● How to view dependency tree?
    ● How to divide prod/dev/test deps?
    

    View Slide

21. Use pipenv pip-tools
    

    View Slide

22. Intro to pip-tools
    $ [sudo] pip install pip-tools
    

    View Slide

23. Intro to pip-tools
    $ [sudo] pip install pip-tools
    $ echo django >> requirements.in
    

    View Slide

24. Intro to pip-tools
    $ [sudo] pip install pip-tools
    $ echo django >> requirements.in
    $ pip-compile --output-file requirements.txt requirements.in
    

    View Slide

25. Intro to pip-tools
    $ [sudo] pip install pip-tools
    $ echo django >> requirements.in
    $ pip-compile --output-file requirements.txt requirements.in
    $ cat requirements.txt
    

    View Slide

26. Intro to pip-tools
    $ [sudo] pip install pip-tools
    $ echo django >> requirements.in
    $ pip-compile --output-file requirements.txt requirements.in
    $ cat requirements.txt
    django==1.11.5
    pytz==2017.2 # via django
    

    View Slide

27. Intro to pip-tools
    $ [sudo] pip install pip-tools
    $ echo django >> requirements.in
    $ pip-compile --output-file requirements.txt requirements.in
    $ cat requirements.txt
    django==1.11.5
    pytz==2017.2 # via django
    

    View Slide

28. View Slide

29. Problem solved(?)
    

    View Slide

30. Moving to libraries
    ● We have setuptools
    

    View Slide

31. Moving to libraries
    ● We have setuptools
    ● Which is used in setup.py
    

    View Slide

32. Moving to libraries
    ● We have setuptools
    ● Which is used in setup.py
    ● And what if setup.py contains external deps?
    

    View Slide

33. Moving to libraries
    ● We have setuptools
    ● Which is used in setup.py
    ● And what if setup.py contains external deps?
    ● Pip will fail to install
    

    View Slide

34. Pipfile (PEP 518)
    ● Rationale: give dev ability to specify build system
    ● Implementation:
    ○ toml file in root called Pipfile
    ○ json file for locking Pipfile.lock
    ○ WIP installation through pip install -p
    

    View Slide

35. Rejected formats (from PEP)
    ● JSON - human-readable, but not human-editable
    

    View Slide

36. Rejected formats (from PEP)
    ● JSON - human-readable, but not human-editable
    ● Configparser - Python stdlib, but confusing
    

    View Slide

37. Rejected formats (from PEP)
    ● JSON - human-readable, but not human-editable
    ● Configparser - Python stdlib, but confusing
    ● Python literals (dict) - Cannot be parsed in other
    languages
    

    View Slide

38. Rejected formats (from PEP)
    ● JSON - human-readable, but not human-editable
    ● Configparser - Python stdlib, but confusing
    ● Python literals (dict) - Cannot be parsed in other
    languages
    ● YAML
    

    View Slide

39. Rejected formats (from PEP)
    ● JSON - human-readable, but not human-editable
    ● Configparser - Python stdlib, but confusing
    ● Python literals (dict) - Cannot be parsed in other
    languages
    ● YAML
    ○ Specification == 86 pages (== too difficult)
    

    View Slide

40. Rejected formats (from PEP)
    ● JSON - human-readable, but not human-editable
    ● Configparser - Python stdlib, but confusing
    ● Python literals (dict) - Cannot be parsed in other
    languages
    ● YAML
    ○ Specification == 86 pages (== too difficult)
    ○ Is not safe by default (command execution vuln.)
    

    View Slide

41. Rejected formats (from PEP)
    ● JSON - human-readable, but not human-editable
    ● Configparser - Python stdlib, but confusing
    ● Python literals (dict) - Cannot be parsed in other
    languages
    ● YAML
    ○ Specification == 86 pages (== too difficult)
    ○ Is not safe by default (command execution vuln.)
    ○ Pip will need to vendor PyYAML (~1k LOC Python & C code)
    

    View Slide

42. Pipfile I
    [[source]]
    url = 'https://pypi.python.org/simple'
    verify_ssl = true
    name = 'pypi'
    

    View Slide

43. Pipfile I
    [[source]]
    url = 'https://pypi.python.org/simple'
    verify_ssl = true
    name = 'pypi'
    

    View Slide

44. Pipfile I
    [[source]]
    url = 'https://pypi.python.org/simple'
    verify_ssl = true
    name = 'pypi'
    

    View Slide

45. Pipfile II
    [packages]
    requests = '>2'
    django = { git = 'https://github.com/django/django.git', ref
    = '1.11.5', editable = true }
    pywinusb = { version = "*", os_name = "=='nt'",
    index="pypi"}
    [dev-packages]
    pytest = '*'
    

    View Slide

46. Pipfile II
    [packages]
    requests = '>2'
    django = { git = 'https://github.com/django/django.git', ref
    = '1.11.5', editable = true }
    pywinusb = { version = "*", os_name = "=='nt'",
    index="pypi"}
    [dev-packages]
    pytest = '*'
    

    View Slide

47. Pipfile II
    [packages]
    requests = '>2'
    django = { git = 'https://github.com/django/django.git', ref
    = '1.11.5', editable = true }
    pywinusb = { version = "*", os_name = "=='nt'",
    index="pypi"}
    [dev-packages]
    pytest = '*'
    

    View Slide

48. Pipfile II
    [packages]
    requests = '>2'
    django = { git = 'https://github.com/django/django.git', ref
    = '1.11.5', editable = true }
    pywinusb = { version = "*", os_name = "=='nt'",
    index="pypi"}
    [dev-packages]
    pytest = '*'
    

    View Slide

49. Pipfile II
    [packages]
    requests = '>2'
    django = { git = 'https://github.com/django/django.git', ref
    = '1.11.5', editable = true }
    pywinusb = { version = "*", os_name = "=='nt'",
    index="pypi"}
    [dev-packages]
    pytest = '*'
    

    View Slide

50. How to use it?
    

    View Slide

51. [sudo] pip install pipenv
    

    View Slide

52. What is Pipenv?
    ● Sacred Marriage of Pipfile, Pip, Pip-tools, & Virtualenv.
    ● From Kenneth Reitz: requests, autoenv
    ● Automatically manages
    ○ Pipfile
    ○ Pipfile.lock
    ○ virtualenv
    

    View Slide

53. Project with requirements.txt
    $ cd your/project/dir
    $ pipenv install
    Requirements.txt found, instead of Pipfile! Converting…
    

    View Slide

54. Custom pypi fail :(
    $ cd your/project/dir
    $ pipenv install
    Requirements.txt found, instead of Pipfile! Converting…
    # but for custom pypi, there will be error on lock :(
    $ sed -i s/pypi.python.org/pypi.example.org/g Pipfile
    

    View Slide

55. $ cd your/project/dir
    $ pipenv run python django-admin startproject
    How to use? I
    

    View Slide

56. $ cd your/project/dir
    $ pipenv shell
    (.venv) $ django-admin
    How to use? II
    

    View Slide

57. How to use with Docker?
    $ pipenv lock
    # In Dockerfile
    WORKDIR /your/dir/
    COPY Pipfile Pipfile.lock /your/dir/
    RUN pipenv install --system
    # to install libs in system folder
    

    View Slide

58. Summary
    Pros
    ● All your deps are locked
    Cons
    

    View Slide

59. Summary
    Pros
    ● All your deps are locked
    ● All your venvs are in .local
    Cons
    

    View Slide

60. Summary
    Pros
    ● All your deps are locked
    ● All your venvs are in .local
    ● Ability to separate dev/prod
    Cons
    

    View Slide

61. Summary
    Pros
    ● All your deps are locked
    ● All your venvs are in .local
    ● Ability to separate dev/prod
    ● Pip will support this file
    Cons
    

    View Slide

62. Summary
    Pros
    ● All your deps are locked
    ● All your venvs are in .local
    ● Ability to separate dev/prod
    ● Pip will support this file
    Cons
    ● Not mature :(
    

    View Slide

63. Summary
    Pros
    ● All your deps are locked
    ● All your venvs are in .local
    ● Ability to separate dev/prod
    ● Pip will support this file
    Cons
    ● Not mature :(
    ● Dockerfile will miss cache
    

    View Slide

64. Summary
    Pros
    ● All your deps are locked
    ● All your venvs are in .local
    ● Ability to separate dev/prod
    ● Pip will support this file
    Cons
    ● Not mature :(
    ● Dockerfile will miss cache
    ● Harder to update package
    

    View Slide

65. Links
    ● PEP https://www.python.org/dev/peps/pep-0518/
    ● Pipfile repo https://github.com/pypa/pipfile
    ● Pipenv repo https://github.com/pypa/pipfile
    ● Cheetsheet
    https://robots.thoughtbot.com/how-to-manage-your-python-projects-with-
    pipenv
    

    View Slide

66. Hiring!
    Yes, t.me/vanadium23
    

    View Slide

67. Questions?
    P.S. follow me → @vanadium23
    

    View Slide