Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Bye, bye Virtual Environments?

Bye, bye Virtual Environments?

Alex Khaerov

June 24, 2019
Tweet

More Decks by Alex Khaerov

Other Decks in Programming

Transcript

  1. @hayorov
    @hayorov
    Привет ✋

    View full-size slide

  2. Virtual Environments?
    Прощай,
    hayorov
    Alex Khaerov

    View full-size slide

  3. @hayorov
    Alex Khaerov
    company
    who I am
    Development Lead
    @hayorov

    View full-size slide

  4. @hayorov
    Alex Khaerov
    company
    who I am
    Development Lead
    doing software development in the recent decade
    junior speaker - Python, Kubernetes
    committee member (Moscow Python, Helm Summit)
    a huge fan of laptop stickers and newbie
    @hayorov

    View full-size slide

  5. @hayorov
    Chainstack
    multi-cloud and multi-blockchain platform as a service
    based in Singapore # and hiring
    Alex Khaerov
    company
    who I am
    Development Lead
    doing software development in the recent decade
    junior speaker - Python, Kubernetes
    committee member (Moscow Python, Helm Summit)
    a huge fan of laptop stickers and newbie
    @hayorov

    View full-size slide

  6. @hayorov
    easy_install
    requrements.txt
    pip
    virtualenv

    View full-size slide

  7. @hayorov
    easy_install
    requrements.txt
    pip
    virtualenv
    setup.py
    sdist
    wheels
    virtualenvwrapper
    pyenv

    View full-size slide

  8. @hayorov
    easy_install
    requrements.txt
    pip
    virtualenv

    setup.py
    sdist
    wheels
    virtualenvwrapper
    pyenv
    conan
    flit
    bento
    hatch
    pants
    conda
    pipenv
    pip-tools
    poetry

    View full-size slide

  9. @hayorov
    easy_install
    requrements.txt
    pip
    virtualenv

    PEP 517 A build-system independent
    format for source trees
    setup.py
    sdist
    wheels
    virtualenvwrapper
    pyenv
    conan
    flit
    bento
    hatch
    pants
    conda
    pipenv
    pip-tools
    poetry

    View full-size slide

  10. @hayorov
    easy_install
    requrements.txt
    pip
    virtualenv

    PEP 517 A build-system independent
    format for source trees
    PEP 518 Specifying Minimum Build
    System Requirements for Python Projects
    setup.py
    sdist
    wheels
    virtualenvwrapper
    pyenv
    conan
    flit
    bento
    hatch
    pants
    conda
    pipenv
    pip-tools
    poetry

    View full-size slide

  11. @hayorov
    easy_install
    requrements.txt
    pip
    virtualenv
    &
    PEP 517 A build-system independent
    format for source trees
    PEP 518 Specifying Minimum Build
    System Requirements for Python Projects
    PEP 582 Specifying Minimum Build
    System Requirements for Python Projects
    setup.py
    sdist
    wheels
    virtualenvwrapper
    pyenv
    conan
    flit
    bento
    hatch
    pants
    conda
    pipenv
    pip-tools
    poetry

    View full-size slide

  12. @hayorov
    KEEP CALM

    View full-size slide

  13. @hayorov
    Who's
    That
    Pokémon?
    Who's
    That
    Pokémon?
    ???
    ???

    View full-size slide

  14. @hayorov
    Who's
    That
    Pokémon?
    Who's
    That
    Pokémon?
    ???
    ???

    View full-size slide

  15. @hayorov
    Dependency Manager for

    View full-size slide

  16. @hayorov
    Dependency Manager for
    composer

    View full-size slide

  17. @hayorov
    Dependency Manager for
    composer

    View full-size slide

  18. @hayorov
    Dependency Manager for
    composer
    conductor

    View full-size slide

  19. @hayorov
    @hayorov
    Modules, libraries = requirements
    Where VirtualEnv coming from?
    Modern era (PEP 517/518)
    Why Pipenv/Poetry is not enough
    Welcome PEP 582
    Concerns
    So now what?
    Agenda

    View full-size slide

  20. @hayorov
    DON’T REPEAT YOURSELF

    View full-size slide

  21. @hayorov
    The Old Days

    View full-size slide

  22. @hayorov
    The Old Days
    Hey, what’s wrong with curl,
    tar and setup.py install?

    View full-size slide

  23. @hayorov
    Problems With This
    • “The Cheeseshop” (PyPi) was merely an index of
    packages, not a sole package host.
    • Packages were often hosted elsewhere.
    • Ran on a single server in
    ', 

    while serving the entire Python community.
    • Its use wasn’t a fraction of what it is today, 

    so it wasn’t a problem.

    View full-size slide

  24. @hayorov
    More Obvious Issues
    • Very manual process - not good for
    • Globally installed packages - impossible to have two
    versions of the same library installed.
    • People often just copied things into site-packages, manually.
    • Poor user experience.

    View full-size slide

  25. @hayorov
    Next Iteration

    View full-size slide

  26. @hayorov
    Improvements!
    • Much better user experience for installation.
    • Most packages were installed from PyPi.
    • Easier to automate programatically.
    • , no easy_uninstall.

    View full-size slide

  27. @hayorov
    2010 -> …
    • Pip became the de-facto replacement for easy_install.
    • Pinned requirements.txt file passed around.
    • Virtualenv became common practice.

    View full-size slide

  28. @hayorov
    2010 -> …
    • Pip became the de-facto replacement for easy_install.
    • Pinned requirements.txt file passed around.
    • Virtualenv became common practice.
    Pip + Virtualenv + requirements.txt =

    View full-size slide

  29. @hayorov
    Hatch, VirtualEnvManager, autoenv, fades, inve, pew, pipenv, pyenv-virtualenv, pyenv-virtualenvwrapper, pyenv, pyvenv, rvirtualenv,
    tox, v, venv, vex, virtual-python, virtualenv-burrito, virtualenv-mv, virtualenv, virtualenvwrapper-win, virtualenvwrapper, workingenv

    View full-size slide

  30. @hayorov
    Virtualenv
    Hatch, VirtualEnvManager, autoenv, fades, inve, pew, pipenv, pyenv-virtualenv, pyenv-virtualenvwrapper, pyenv, pyvenv, rvirtualenv,
    tox, v, venv, vex, virtual-python, virtualenv-burrito, virtualenv-mv, virtualenv, virtualenvwrapper-win, virtualenvwrapper, workingenv

    View full-size slide

  31. @hayorov
    Virtualenv
    Hatch, VirtualEnvManager, autoenv, fades, inve, pew, pipenv, pyenv-virtualenv, pyenv-virtualenvwrapper, pyenv, pyvenv, rvirtualenv,
    tox, v, venv, vex, virtual-python, virtualenv-burrito, virtualenv-mv, virtualenv, virtualenvwrapper-win, virtualenvwrapper, workingenv
    Oct 17, 2005
    virtual-python.py is added to EasyInstall.

    View full-size slide

  32. @hayorov
    Virtualenv
    Hatch, VirtualEnvManager, autoenv, fades, inve, pew, pipenv, pyenv-virtualenv, pyenv-virtualenvwrapper, pyenv, pyvenv, rvirtualenv,
    tox, v, venv, vex, virtual-python, virtualenv-burrito, virtualenv-mv, virtualenv, virtualenvwrapper-win, virtualenvwrapper, workingenv
    Jun 13, 2011
    PEP 405 (Python Virtual Environments)
    is created.
    Oct 17, 2005
    virtual-python.py is added to EasyInstall.

    View full-size slide

  33. @hayorov
    Virtualenv
    Hatch, VirtualEnvManager, autoenv, fades, inve, pew, pipenv, pyenv-virtualenv, pyenv-virtualenvwrapper, pyenv, pyvenv, rvirtualenv,
    tox, v, venv, vex, virtual-python, virtualenv-burrito, virtualenv-mv, virtualenv, virtualenvwrapper-win, virtualenvwrapper, workingenv
    Jun 13, 2011
    PEP 405 (Python Virtual Environments)
    is created.
    Oct 17, 2005
    virtual-python.py is added to EasyInstall.

    May 25, 2012
    PEP 405 is accepted
    for inclusion in Python 3.3.

    View full-size slide

  34. @hayorov
    Virtualenv
    Hatch, VirtualEnvManager, autoenv, fades, inve, pew, pipenv, pyenv-virtualenv, pyenv-virtualenvwrapper, pyenv, pyvenv, rvirtualenv,
    tox, v, venv, vex, virtual-python, virtualenv-burrito, virtualenv-mv, virtualenv, virtualenvwrapper-win, virtualenvwrapper, workingenv
    Jun 13, 2011
    PEP 405 (Python Virtual Environments)
    is created.
    Oct 17, 2005
    virtual-python.py is added to EasyInstall.

    May 25, 2012
    PEP 405 is accepted
    for inclusion in Python 3.3.
    Sep 13, 2015
    Python 3.5 is released and
    venv is recommended tool

    View full-size slide

  35. @hayorov
    Virtualenv
    Hatch, VirtualEnvManager, autoenv, fades, inve, pew, pipenv, pyenv-virtualenv, pyenv-virtualenvwrapper, pyenv, pyvenv, rvirtualenv,
    tox, v, venv, vex, virtual-python, virtualenv-burrito, virtualenv-mv, virtualenv, virtualenvwrapper-win, virtualenvwrapper, workingenv
    Jun 13, 2011
    PEP 405 (Python Virtual Environments)
    is created.
    Oct 17, 2005
    virtual-python.py is added to EasyInstall.

    May 25, 2012
    PEP 405 is accepted
    for inclusion in Python 3.3.
    Sep 13, 2015
    Python 3.5 is released and
    venv is recommended tool
    • Virtualenv is a directory containing a Python interpreter, a special pyvenv.cfg
    • The “standard tool for creating virtual environments” is venv.

    View full-size slide

  36. Pipenv Poetry
    2019

    View full-size slide

  37. Pipenv Poetry
    Was first with pyproject.toml-
    styled Pipfile/pipfile.lock
    2019

    View full-size slide

  38. Pipenv Poetry
    Was first with pyproject.toml-
    styled Pipfile/pipfile.lock
    Automates away Virtualenv
    entirely
    2019

    View full-size slide

  39. Pipenv Poetry
    Was first with pyproject.toml-
    styled Pipfile/pipfile.lock
    Automates away Virtualenv
    entirely
    Ensures deterministic build
    2019

    View full-size slide

  40. Pipenv Poetry
    Was first with pyproject.toml-
    styled Pipfile/pipfile.lock
    Automates away Virtualenv
    entirely
    Ensures deterministic build
    Other useful tools
    e.g. $ pipenv graph.
    2019

    View full-size slide

  41. Pipenv Poetry
    Was first with pyproject.toml-
    styled Pipfile/pipfile.lock
    Automates away Virtualenv
    entirely
    Ensures deterministic build
    Other useful tools
    e.g. $ pipenv graph.
    All the same

    2019

    View full-size slide

  42. Pipenv Poetry
    Was first with pyproject.toml-
    styled Pipfile/pipfile.lock
    Automates away Virtualenv
    entirely
    Ensures deterministic build
    Other useful tools
    e.g. $ pipenv graph.
    Exhaustive and fast dependency
    resolver.
    All the same

    2019

    View full-size slide

  43. Pipenv Poetry
    Was first with pyproject.toml-
    styled Pipfile/pipfile.lock
    Automates away Virtualenv
    entirely
    Ensures deterministic build
    Other useful tools
    e.g. $ pipenv graph.
    Exhaustive and fast dependency
    resolver.
    Emphasis on semantic versioning
    All the same

    2019

    View full-size slide

  44. Pipenv Poetry
    Was first with pyproject.toml-
    styled Pipfile/pipfile.lock
    Automates away Virtualenv
    entirely
    Ensures deterministic build
    Other useful tools
    e.g. $ pipenv graph.
    Exhaustive and fast dependency
    resolver.
    Emphasis on semantic versioning
    Easily build and package
    $ poetry publish
    All the same

    2019

    View full-size slide

  45. Pipenv Poetry
    Was first with pyproject.toml-
    styled Pipfile/pipfile.lock
    Automates away Virtualenv
    entirely
    Ensures deterministic build
    Other useful tools
    e.g. $ pipenv graph.
    Exhaustive and fast dependency
    resolver.
    Emphasis on semantic versioning
    Easily build and package
    $ poetry publish
    All the same

    2019

    View full-size slide

  46. @hayorov
    Pipenv/Poetry is not enough

    View full-size slide

  47. @hayorov
    Pipenv/Poetry is not enough
    • Learning curve 

    Explaining “virtual environments” is hard


    View full-size slide

  48. @hayorov
    Pipenv/Poetry is not enough
    • Learning curve 

    Explaining “virtual environments” is hard


    • Terminal isolation

    Activating/deactivating

    View full-size slide

  49. @hayorov
    Pipenv/Poetry is not enough
    • Learning curve 

    Explaining “virtual environments” is hard


    • Terminal isolation

    Activating/deactivating
    • Cognitive overhead

    Setting up, remembering installation location, 

    activating/deactivating

    View full-size slide

  50. @hayorov
    @hayorov

    View full-size slide

  51. @hayorov
    __pypackages __

    View full-size slide

  52. @hayorov
    PEP 582 - Python local packages directory
    “This PEP proposes to add to Python a mechanism
    to automatically recognize a directory
    and prefer importing packages installed in this location
    over user or global site-packages."
    __pypackages __

    View full-size slide

  53. @hayorov
    foo
    |
    |- myscript.py

    |
    |- my_lib
    | |_ __init __.py
    | |_my_lib.py

    View full-size slide

  54. @hayorov
    foo
    |
    |- myscript.py

    |
    |- my_lib
    | |_ __init __.py
    | |_my_lib.py
    |
    |_ __pypackages __
    |_ 3.8
    |- requests
    | |_ __init __.py
    |_ django
    |_ __init __.py

    View full-size slide

  55. @hayorov
    foo
    |
    |- myscript.py

    |
    |- my_lib
    | |_ __init __.py
    | |_my_lib.py
    |
    |_ __pypackages __
    |_ 3.8
    |- requests
    | |_ __init __.py
    |_ django
    |_ __init __.py
    Magic folder

    View full-size slide

  56. @hayorov
    foo
    |
    |- myscript.py

    |
    |- my_lib
    | |_ __init __.py
    | |_my_lib.py
    |
    |_ __pypackages __
    |_ 3.8
    |- requests
    | |_ __init __.py
    |_ django
    |_ __init __.py
    Magic folder
    PyVersion folder

    View full-size slide

  57. @hayorov
    foo
    |
    |- myscript.py

    |
    |- my_lib
    | |_ __init __.py
    | |_my_lib.py
    |
    |_ __pypackages __
    |_ 3.8
    |- requests
    | |_ __init __.py
    |_ django
    |_ __init __.py
    Magic folder
    PyVersion folder
    Full structure

    View full-size slide

  58. @hayorov
    Python binary is executed

    View full-size slide

  59. @hayorov
    Python binary is executed
    Determine its prefix (as stored in sys.prefix)
    In [2]: sys.prefix
    Out[2]: '/usr/local/Cellar/python/3.7.3/Frameworks/Python.framework/Versions/3.7'

    View full-size slide

  60. @hayorov
    Python binary is executed
    Determine its prefix (as stored in sys.prefix)
    In [2]: sys.prefix
    Out[2]: '/usr/local/Cellar/python/3.7.3/Frameworks/Python.framework/Versions/3.7'
    find the standard library and other key files, and determine site-package
    In [3]: sys.path
    Out[3]:
    ['/usr/local/bin',
    '/usr/local/Cellar/python/3.7.3/Frameworks/Python.framework/Versions/3.7/lib/python37.zip',
    '/usr/local/Cellar/python/3.7.3/Frameworks/Python.framework/Versions/3.7/lib/python3.7',
    '/usr/local/Cellar/python/3.7.3/Frameworks/Python.framework/Versions/3.7/lib/python3.7/lib-dynload'
    '',
    ‘/usr/local/lib/python3.7/site-packages'

    View full-size slide

  61. @hayorov
    Python binary is executed
    Determine its prefix (as stored in sys.prefix)
    In [2]: sys.prefix
    Out[2]: '/usr/local/Cellar/python/3.7.3/Frameworks/Python.framework/Versions/3.7'
    find the standard library and other key files, and determine site-package
    In [3]: sys.path
    Out[3]:
    ['/usr/local/bin',
    '/usr/local/Cellar/python/3.7.3/Frameworks/Python.framework/Versions/3.7/lib/python37.zip',
    '/usr/local/Cellar/python/3.7.3/Frameworks/Python.framework/Versions/3.7/lib/python3.7',
    '/usr/local/Cellar/python/3.7.3/Frameworks/Python.framework/Versions/3.7/lib/python3.7/lib-dynload'
    '',
    ‘/usr/local/lib/python3.7/site-packages'
    __pypackages __

    View full-size slide

  62. @hayorov
    Try it Today

    View full-size slide

  63. @hayorov
    Try it Today
    reference CPython
    implementation
    https://github.com/kushaldas/cpython/tree/pypackages

    View full-size slide

  64. @hayorov
    Try it Today
    reference CPython
    implementation

    need to build the binaries yourself
    https://github.com/kushaldas/cpython/tree/pypackages

    View full-size slide

  65. @hayorov
    Try it Today
    reference CPython
    implementation
    latest commit accfcb3 on 14 Sep 2018

    need to build the binaries yourself
    https://github.com/kushaldas/cpython/tree/pypackages

    View full-size slide

  66. @hayorov
    Try it Today
    reference CPython
    implementation
    proof of concept
    wrapper pythonloc
    latest commit accfcb3 on 14 Sep 2018

    need to build the binaries yourself
    https://github.com/kushaldas/cpython/tree/pypackages https://github.com/cs01/pythonloc

    View full-size slide

  67. @hayorov
    Try it Today
    reference CPython
    implementation
    proof of concept
    wrapper pythonloc
    latest commit accfcb3 on 14 Sep 2018

    need to build the binaries yourself
    https://github.com/kushaldas/cpython/tree/pypackages https://github.com/cs01/pythonloc
    pip install --user pythonloc

    View full-size slide

  68. @hayorov
    Try it Today
    reference CPython
    implementation
    proof of concept
    wrapper pythonloc
    latest commit accfcb3 on 14 Sep 2018

    need to build the binaries yourself
    https://github.com/kushaldas/cpython/tree/pypackages https://github.com/cs01/pythonloc
    pythonloc my_script.py
    pip install --user pythonloc

    View full-size slide

  69. @hayorov
    Try it Today
    reference CPython
    implementation
    proof of concept
    wrapper pythonloc
    latest commit accfcb3 on 14 Sep 2018

    need to build the binaries yourself
    https://github.com/kushaldas/cpython/tree/pypackages https://github.com/cs01/pythonloc
    pythonloc my_script.py
    pip install --user pythonloc
    piploc / pipfreezeloc

    View full-size slide

  70. @hayorov
    Try it Today
    reference CPython
    implementation
    proof of concept
    wrapper pythonloc
    latest commit accfcb3 on 14 Sep 2018

    need to build the binaries yourself
    https://github.com/kushaldas/cpython/tree/pypackages https://github.com/cs01/pythonloc
    pythonloc my_script.py
    pip install --user pythonloc
    piploc / pipfreezeloc

    View full-size slide

  71. @hayorov
    Concerns

    View full-size slide

  72. @hayorov
    Concerns
    What about entry points?

    pip and virtualenv associate them automatically, PEP582 - not.

    View full-size slide

  73. @hayorov
    Concerns
    What about entry points?

    pip and virtualenv associate them automatically, PEP582 - not.
    What about Pipenv (or poetry, or pip)?

    View full-size slide

  74. @hayorov
    Concerns
    What about entry points?

    pip and virtualenv associate them automatically, PEP582 - not.
    What about Pipenv (or poetry, or pip)?
    What about cross platform?

    __pypackages__/windows/3.6/lib or __pypackages__/unix/3.6/lib

    View full-size slide

  75. @hayorov
    What about the fallback to site-packages?

    __pypackages__ -> ? -> site-packages
    Concerns
    What about entry points?

    pip and virtualenv associate them automatically, PEP582 - not.
    What about Pipenv (or poetry, or pip)?
    What about cross platform?

    __pypackages__/windows/3.6/lib or __pypackages__/unix/3.6/lib

    View full-size slide

  76. @hayorov
    What about the fallback to site-packages?

    __pypackages__ -> ? -> site-packages
    Concerns
    What about entry points?

    pip and virtualenv associate them automatically, PEP582 - not.
    What about Pipenv (or poetry, or pip)?
    What about cross platform?

    __pypackages__/windows/3.6/lib or __pypackages__/unix/3.6/lib
    What about bloat?

    Yes - nodejs

    View full-size slide

  77. @hayorov
    So Now What?
    • Start a new project with Poetry
    • Already with Pipenv? Stay on it!
    • Be patient and wait for native support of PEP 582
    • Wanna try? cs01/pythonloc

    View full-size slide

  78. Thank you
    questions…

    Alex Khaerov
    t.me/hayorov
    http://bit.ly/xxx

    View full-size slide