Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Inside The Cheeseshop: How Python Packaging Works (SciPy 2018)

Inside The Cheeseshop: How Python Packaging Works (SciPy 2018)

Questions and confusion about the Python packaging ecosystem abound. What is this `setup.py` file? What's the difference between wheels and eggs? Do I use setuptools or distutils? Why should I use twine? Do I put my projects dependencies in a `requirements.txt` or in `setup.py`? How do I just get my module up on PyPI? Should I use `pip` or `conda`? Wait, what is Warehouse?

This talk identifies the key tools one might encounter when trying to distribute Python software, what they are used for, why they exist, and their history (including where their weird names come from). In addition, we'll see how they all work together, what it takes to make them work, and what the future has in store for Python packaging.

Dustin Ingram

July 11, 2018
Tweet

More Decks by Dustin Ingram

Other Decks in Technology

Transcript

  1. SciPy 2018
    SciPy 2018 — Dus/n Ingram — @di_codes

    View Slide

  2. Hi, I'm Dus+n

    View Slide

  3. Who am I?
    • Director @ PromptWorks in Aus4n, TX
    • PyTexas Organizer
    • Python Packaging Working Group (Packaging-WG)
    member
    • Python Packaging Authority (PyPA) member
    • Python Package Index (PyPI) maintainer
    SciPy 2018 — Dus/n Ingram — @di_codes

    View Slide

  4. {{ talk.title }}
    {{ talk.subtitle }}
    SciPy 2018 — Dus/n Ingram — @di_codes

    View Slide

  5. Python Packaging
    Ge#ng the Code You Wrote
    To People That Want It
    Using the Same Language You Wrote It In
    SciPy 2018 — Dus/n Ingram — @di_codes

    View Slide

  6. View Slide

  7. Python Packaging
    In Just Five Easy Steps!
    SciPy 2018 — Dus/n Ingram — @di_codes

    View Slide

  8. View Slide

  9. Python Packaging
    In Just FOUR Easy Steps!
    SciPy 2018 — Dus/n Ingram — @di_codes

    View Slide

  10. View Slide

  11. Python Packaging
    It's Rela*vely Painless Now
    Go Ahead and Use It
    SciPy 2018 — Dus/n Ingram — @di_codes

    View Slide

  12. View Slide

  13. Python Packaging
    We're S'll Trying To Make It Be5er
    SciPy 2018 — Dus/n Ingram — @di_codes

    View Slide

  14. View Slide

  15. Python Packaging
    Let's Just Throw It All Away
    And Start Over From Scratch
    SciPy 2018 — Dus/n Ingram — @di_codes

    View Slide

  16. View Slide

  17. Python Packaging
    Let Me Just Get You Up To Speed
    On Everything That's Changed Since Last Time
    SciPy 2018 — Dus/n Ingram — @di_codes

    View Slide

  18. View Slide

  19. Python Packaging
    There's A Lot Of Stuff Here
    You Might Not Need All Of It
    SciPy 2018 — Dus/n Ingram — @di_codes

    View Slide

  20. View Slide

  21. Python Packaging
    In the Simplest Terms Possible
    For Anyone That Cares
    SciPy 2018 — Dus/n Ingram — @di_codes

    View Slide

  22. View Slide

  23. Python Packaging
    So Easy A Caveman Could Do It
    SciPy 2018 — Dus/n Ingram — @di_codes

    View Slide

  24. View Slide

  25. Hello
    I Am a PyPI Maintainer
    At the Very Least
    I Should Be Able To Tell You
    How To Use PyPI
    SciPy 2018 — Dus/n Ingram — @di_codes

    View Slide

  26. View Slide

  27. Python Packaging
    SciPy 2018 — Dus/n Ingram — @di_codes

    View Slide

  28. Python Packaging
    But Indiana Jones-themed
    SciPy 2018 — Dus/n Ingram — @di_codes

    View Slide

  29. Python Packaging
    But Indiana Jones-themed
    (Also In French)
    SciPy 2018 — Dus/n Ingram — @di_codes

    View Slide

  30. View Slide

  31. Links
    • h#ps:/
    /www.youtube.com/watch?v=qOH-h-EKKac
    • h#ps:/
    /www.youtube.com/watch?v=5BqAeN-F9Qs
    • h#ps:/
    /www.youtube.com/watch?v=gc9dkktg1gU
    • h#ps:/
    /www.youtube.com/watch?
    v=W8A2bOKPtJU
    • h#ps:/
    /www.youtube.com/watch?v=eLPiPHr6TVI
    • h#ps:/
    /www.youtube.com/watch?
    SciPy 2018 — Dus/n Ingram — @di_codes

    View Slide

  32. The End
    SciPy 2018 — Dus/n Ingram — @di_codes

    View Slide

  33. Inside The Cheeseshop
    How Python Packaging Works
    SciPy 2018 — Dus/n Ingram — @di_codes

    View Slide

  34. View Slide

  35. View Slide

  36. View Slide

  37. View Slide

  38. View Slide

  39. totally_awesome_library.py
    SciPy 2018 — Dus/n Ingram — @di_codes

    View Slide

  40. Problem:
    How do I get this to users?
    SciPy 2018 — Dus/n Ingram — @di_codes

    View Slide

  41. Solu%on:
    Email?
    SciPy 2018 — Dus/n Ingram — @di_codes

    View Slide

  42. Solu%on:
    Personal Website?
    SciPy 2018 — Dus/n Ingram — @di_codes

    View Slide

  43. View Slide

  44. Problem:
    How do I find Python code?
    SciPy 2018 — Dus/n Ingram — @di_codes

    View Slide

  45. Solu%on:
    The Vaults of Parnassus
    SciPy 2018 — Dus/n Ingram — @di_codes

    View Slide

  46. View Slide

  47. Problem:
    How do I build this?
    SciPy 2018 — Dus/n Ingram — @di_codes

    View Slide

  48. Solu%on:
    distutils
    SciPy 2018 — Dus/n Ingram — @di_codes

    View Slide

  49. $ python setup.py
    SciPy 2018 — Dus/n Ingram — @di_codes

    View Slide

  50. $ python setup.py build
    SciPy 2018 — Dus/n Ingram — @di_codes

    View Slide

  51. Solu%on:
    Source Distribu-ons
    SciPy 2018 — Dus/n Ingram — @di_codes

    View Slide

  52. AKA:
    sdist
    SciPy 2018 — Dus/n Ingram — @di_codes

    View Slide

  53. $ python setup.py sdist
    SciPy 2018 — Dus/n Ingram — @di_codes

    View Slide

  54. Problem:
    Building takes too long
    SciPy 2018 — Dus/n Ingram — @di_codes

    View Slide

  55. Solu%on:
    Built Distribu+ons
    SciPy 2018 — Dus/n Ingram — @di_codes

    View Slide

  56. AKA:
    bdist
    SciPy 2018 — Dus/n Ingram — @di_codes

    View Slide

  57. $ python setup.py bdist
    SciPy 2018 — Dus/n Ingram — @di_codes

    View Slide

  58. Problem:
    How can I "do" packaging?
    SciPy 2018 — Dus/n Ingram — @di_codes

    View Slide

  59. Solu%on:
    Use pla(orm packaging?
    SciPy 2018 — Dus/n Ingram — @di_codes

    View Slide

  60. Sub-Problem:
    My pla'orm doesn't have
    packaging
    SciPy 2018 — Dus/n Ingram — @di_codes

    View Slide

  61. Sub-Problem:
    My pla'orm does have
    packaging, but I want things
    now!
    SciPy 2018 — Dus/n Ingram — @di_codes

    View Slide

  62. Solu%on:
    The Python Package Index
    SciPy 2018 — Dus/n Ingram — @di_codes

    View Slide

  63. View Slide

  64. AKA:
    "PyPI"
    SciPy 2018 — Dus/n Ingram — @di_codes

    View Slide

  65. !"#
    SciPy 2018 — Dus/n Ingram — @di_codes

    View Slide

  66. AKA:
    "The Cheeseshop"
    SciPy 2018 — Dus/n Ingram — @di_codes

    View Slide

  67. View Slide

  68. Problem:
    How can I specify
    dependencies?
    SciPy 2018 — Dus/n Ingram — @di_codes

    View Slide

  69. Solu%on:
    setuptools
    SciPy 2018 — Dus/n Ingram — @di_codes

    View Slide

  70. Problem:
    Installing is too hard!
    SciPy 2018 — Dus/n Ingram — @di_codes

    View Slide

  71. Solu%on:
    easy_install
    SciPy 2018 — Dus/n Ingram — @di_codes

    View Slide

  72. The Egg Distribu.on
    SciPy 2018 — Dus/n Ingram — @di_codes

    View Slide

  73. View Slide

  74. Problem:
    easy_install problems
    SciPy 2018 — Dus/n Ingram — @di_codes

    View Slide

  75. Solu%on:
    pyinstall
    SciPy 2018 — Dus/n Ingram — @di_codes

    View Slide

  76. Problem:
    "pyinstall" is too long
    SciPy 2018 — Dus/n Ingram — @di_codes

    View Slide

  77. Solu%on:
    pip
    SciPy 2018 — Dus/n Ingram — @di_codes

    View Slide

  78. eggs
    SciPy 2018 — Dus/n Ingram — @di_codes

    View Slide

  79. Problem:
    Applica'on dependencies?
    SciPy 2018 — Dus/n Ingram — @di_codes

    View Slide

  80. Solu%on:
    requirements.txt
    SciPy 2018 — Dus/n Ingram — @di_codes

    View Slide

  81. pip install -r requirements.txt
    SciPy 2018 — Dus/n Ingram — @di_codes

    View Slide

  82. Problem:
    Installing from PyPI is slow
    SciPy 2018 — Dus/n Ingram — @di_codes

    View Slide

  83. Problem:
    Trus%ng 3rd Party Domains
    SciPy 2018 — Dus/n Ingram — @di_codes

    View Slide

  84. Solu%on:
    PyPI begins hos-ng releases
    SciPy 2018 — Dus/n Ingram — @di_codes

    View Slide

  85. View Slide

  86. Problem:
    Scien&fic Python is a thing
    SciPy 2018 — Dus/n Ingram — @di_codes

    View Slide

  87. Sub-Problem:
    distu&ls is not up to the task
    SciPy 2018 — Dus/n Ingram — @di_codes

    View Slide

  88. Sub-Problem:
    Build environments?
    SciPy 2018 — Dus/n Ingram — @di_codes

    View Slide

  89. Sub-Problem:
    I need prebuilt binaries
    SciPy 2018 — Dus/n Ingram — @di_codes

    View Slide

  90. Sub-Problem:
    I need more than Python
    SciPy 2018 — Dus/n Ingram — @di_codes

    View Slide

  91. Sub-Problem:
    I don't even have Python
    SciPy 2018 — Dus/n Ingram — @di_codes

    View Slide

  92. Solu%on:
    Anaconda & Conda
    SciPy 2018 — Dus/n Ingram — @di_codes

    View Slide

  93. Problem:
    We need built distribu-ons
    (again)
    SciPy 2018 — Dus/n Ingram — @di_codes

    View Slide

  94. Solu%on:
    The Wheel Distribu.on
    SciPy 2018 — Dus/n Ingram — @di_codes

    View Slide

  95. View Slide

  96. View Slide

  97. Problem:
    PyPI is showing it's age
    SciPy 2018 — Dus/n Ingram — @di_codes

    View Slide

  98. View Slide

  99. View Slide

  100. View Slide

  101. View Slide

  102. Solu%on:
    Rewrite PyPI from scratch
    SciPy 2018 — Dus/n Ingram — @di_codes

    View Slide

  103. View Slide

  104. AKA:
    Warehouse
    SciPy 2018 — Dus/n Ingram — @di_codes

    View Slide

  105. View Slide

  106. The End
    SciPy 2018 — Dus/n Ingram — @di_codes

    View Slide

  107. Current Problems
    SciPy 2018 — Dus/n Ingram — @di_codes

    View Slide

  108. Problem:
    Packaging is s*ll kinda hard
    SciPy 2018 — Dus/n Ingram — @di_codes

    View Slide

  109. Solu%on:
    The Python Packaging Guide
    h"ps:/
    /packaging.python.org
    SciPy 2018 — Dus/n Ingram — @di_codes

    View Slide

  110. Solu%on:
    sampleproject
    h"ps:/
    /github.com/pypa/sampleproject
    SciPy 2018 — Dus/n Ingram — @di_codes

    View Slide

  111. Solu%on:
    General care & maintenance
    SciPy 2018 — Dus/n Ingram — @di_codes

    View Slide

  112. Problem:
    Packaging is a li+le too easy
    SciPy 2018 — Dus/n Ingram — @di_codes

    View Slide

  113. Python Packaging
    So Easy a Spammer Could Do It
    SciPy 2018 — Dus/n Ingram — @di_codes

    View Slide

  114. Problem:
    Reproducible Environments
    SciPy 2018 — Dus/n Ingram — @di_codes

    View Slide

  115. Solu%on:
    pipfile/pipfile.lock
    h"ps:/
    /github.com/pypa/pipfile
    SciPy 2018 — Dus/n Ingram — @di_codes

    View Slide

  116. Problem:
    Arbitrary code in setup.py
    SciPy 2018 — Dus/n Ingram — @di_codes

    View Slide

  117. View Slide

  118. Problem:
    A distu(ls/setuptools dance
    SciPy 2018 — Dus/n Ingram — @di_codes

    View Slide

  119. Solu%on:
    PEP 517/518
    SciPy 2018 — Dus/n Ingram — @di_codes

    View Slide

  120. View Slide

  121. View Slide

  122. pyproject.toml
    SciPy 2018 — Dus/n Ingram — @di_codes

    View Slide

  123. How to help
    SciPy 2018 — Dus/n Ingram — @di_codes

    View Slide

  124. View Slide

  125. How to get help
    SciPy 2018 — Dus/n Ingram — @di_codes

    View Slide

  126. How to get help
    • h#ps:/
    /packaging.python.org
    • Issue tracker for a given tool
    • On IRC: #pypa Freenode channel
    • h#ps:/
    /github.com/pypa/packaging-problems
    SciPy 2018 — Dus/n Ingram — @di_codes

    View Slide

  127. Talk to me
    • Github: @di
    • Twi,er: @di_codes
    • Email: [email protected]
    SciPy 2018 — Dus/n Ingram — @di_codes

    View Slide

  128. Summary
    SciPy 2018 — Dus/n Ingram — @di_codes

    View Slide

  129. Thanks!
    SciPy 2018 — Dus/n Ingram — @di_codes

    View Slide

  130. Talk to me
    • Github: @di
    • Twi,er: @di_codes
    • Email: [email protected]
    • Or, ask me ques,ons now!
    SciPy 2018 — Dus/n Ingram — @di_codes

    View Slide