Inside The Cheeseshop: How Python Packaging Works (SciPy 2018)

Questions and confusion about the Python packaging ecosystem abound. What is this `setup.py` file? What's the difference between wheels and eggs? Do I use setuptools or distutils? Why should I use twine? Do I put my projects dependencies in a `requirements.txt` or in `setup.py`? How do I just get my module up on PyPI? Should I use `pip` or `conda`? Wait, what is Warehouse?

This talk identifies the key tools one might encounter when trying to distribute Python software, what they are used for, why they exist, and their history (including where their weird names come from). In addition, we'll see how they all work together, what it takes to make them work, and what the future has in store for Python packaging.

Transcript

1. SciPy 2018
   SciPy 2018 — Dus/n Ingram — @di_codes
   

   View Slide

2. Hi, I'm Dus+n
   

   View Slide

3. Who am I?
   • Director @ PromptWorks in Aus4n, TX
   • PyTexas Organizer
   • Python Packaging Working Group (Packaging-WG)
   member
   • Python Packaging Authority (PyPA) member
   • Python Package Index (PyPI) maintainer
   SciPy 2018 — Dus/n Ingram — @di_codes
   

   View Slide

4. {{ talk.title }}
   {{ talk.subtitle }}
   SciPy 2018 — Dus/n Ingram — @di_codes
   

   View Slide

5. Python Packaging
   Ge#ng the Code You Wrote
   To People That Want It
   Using the Same Language You Wrote It In
   SciPy 2018 — Dus/n Ingram — @di_codes
   

   View Slide

6. View Slide

7. Python Packaging
   In Just Five Easy Steps!
   SciPy 2018 — Dus/n Ingram — @di_codes
   

   View Slide

8. View Slide

9. Python Packaging
   In Just FOUR Easy Steps!
   SciPy 2018 — Dus/n Ingram — @di_codes
   

   View Slide

10. View Slide

11. Python Packaging
    It's Rela*vely Painless Now
    Go Ahead and Use It
    SciPy 2018 — Dus/n Ingram — @di_codes
    

    View Slide

12. View Slide

13. Python Packaging
    We're S'll Trying To Make It Be5er
    SciPy 2018 — Dus/n Ingram — @di_codes
    

    View Slide

14. View Slide

15. Python Packaging
    Let's Just Throw It All Away
    And Start Over From Scratch
    SciPy 2018 — Dus/n Ingram — @di_codes
    

    View Slide

16. View Slide

17. Python Packaging
    Let Me Just Get You Up To Speed
    On Everything That's Changed Since Last Time
    SciPy 2018 — Dus/n Ingram — @di_codes
    

    View Slide

18. View Slide

19. Python Packaging
    There's A Lot Of Stuff Here
    You Might Not Need All Of It
    SciPy 2018 — Dus/n Ingram — @di_codes
    

    View Slide

20. View Slide

21. Python Packaging
    In the Simplest Terms Possible
    For Anyone That Cares
    SciPy 2018 — Dus/n Ingram — @di_codes
    

    View Slide

22. View Slide

23. Python Packaging
    So Easy A Caveman Could Do It
    SciPy 2018 — Dus/n Ingram — @di_codes
    

    View Slide

24. View Slide

25. Hello
    I Am a PyPI Maintainer
    At the Very Least
    I Should Be Able To Tell You
    How To Use PyPI
    SciPy 2018 — Dus/n Ingram — @di_codes
    

    View Slide

26. View Slide

27. Python Packaging
    SciPy 2018 — Dus/n Ingram — @di_codes
    

    View Slide

28. Python Packaging
    But Indiana Jones-themed
    SciPy 2018 — Dus/n Ingram — @di_codes
    

    View Slide

29. Python Packaging
    But Indiana Jones-themed
    (Also In French)
    SciPy 2018 — Dus/n Ingram — @di_codes
    

    View Slide

30. View Slide

31. Links
    • h#ps:/
    /www.youtube.com/watch?v=qOH-h-EKKac
    • h#ps:/
    /www.youtube.com/watch?v=5BqAeN-F9Qs
    • h#ps:/
    /www.youtube.com/watch?v=gc9dkktg1gU
    • h#ps:/
    /www.youtube.com/watch?
    v=W8A2bOKPtJU
    • h#ps:/
    /www.youtube.com/watch?v=eLPiPHr6TVI
    • h#ps:/
    /www.youtube.com/watch?
    SciPy 2018 — Dus/n Ingram — @di_codes
    

    View Slide

32. The End
    SciPy 2018 — Dus/n Ingram — @di_codes
    

    View Slide

33. Inside The Cheeseshop
    How Python Packaging Works
    SciPy 2018 — Dus/n Ingram — @di_codes
    

    View Slide

34. View Slide

35. View Slide

36. View Slide

37. View Slide

38. View Slide

39. totally_awesome_library.py
    SciPy 2018 — Dus/n Ingram — @di_codes
    

    View Slide

40. Problem:
    How do I get this to users?
    SciPy 2018 — Dus/n Ingram — @di_codes
    

    View Slide

41. Solu%on:
    Email?
    SciPy 2018 — Dus/n Ingram — @di_codes
    

    View Slide

42. Solu%on:
    Personal Website?
    SciPy 2018 — Dus/n Ingram — @di_codes
    

    View Slide

43. View Slide

44. Problem:
    How do I find Python code?
    SciPy 2018 — Dus/n Ingram — @di_codes
    

    View Slide

45. Solu%on:
    The Vaults of Parnassus
    SciPy 2018 — Dus/n Ingram — @di_codes
    

    View Slide

46. View Slide

47. Problem:
    How do I build this?
    SciPy 2018 — Dus/n Ingram — @di_codes
    

    View Slide

48. Solu%on:
    distutils
    SciPy 2018 — Dus/n Ingram — @di_codes
    

    View Slide

49. $ python setup.py
    SciPy 2018 — Dus/n Ingram — @di_codes
    

    View Slide

50. $ python setup.py build
    SciPy 2018 — Dus/n Ingram — @di_codes
    

    View Slide

51. Solu%on:
    Source Distribu-ons
    SciPy 2018 — Dus/n Ingram — @di_codes
    

    View Slide

52. AKA:
    sdist
    SciPy 2018 — Dus/n Ingram — @di_codes
    

    View Slide

53. $ python setup.py sdist
    SciPy 2018 — Dus/n Ingram — @di_codes
    

    View Slide

54. Problem:
    Building takes too long
    SciPy 2018 — Dus/n Ingram — @di_codes
    

    View Slide

55. Solu%on:
    Built Distribu+ons
    SciPy 2018 — Dus/n Ingram — @di_codes
    

    View Slide

56. AKA:
    bdist
    SciPy 2018 — Dus/n Ingram — @di_codes
    

    View Slide

57. $ python setup.py bdist
    SciPy 2018 — Dus/n Ingram — @di_codes
    

    View Slide

58. Problem:
    How can I "do" packaging?
    SciPy 2018 — Dus/n Ingram — @di_codes
    

    View Slide

59. Solu%on:
    Use pla(orm packaging?
    SciPy 2018 — Dus/n Ingram — @di_codes
    

    View Slide

60. Sub-Problem:
    My pla'orm doesn't have
    packaging
    SciPy 2018 — Dus/n Ingram — @di_codes
    

    View Slide

61. Sub-Problem:
    My pla'orm does have
    packaging, but I want things
    now!
    SciPy 2018 — Dus/n Ingram — @di_codes
    

    View Slide

62. Solu%on:
    The Python Package Index
    SciPy 2018 — Dus/n Ingram — @di_codes
    

    View Slide

63. View Slide

64. AKA:
    "PyPI"
    SciPy 2018 — Dus/n Ingram — @di_codes
    

    View Slide

65. !"#
    SciPy 2018 — Dus/n Ingram — @di_codes
    

    View Slide

66. AKA:
    "The Cheeseshop"
    SciPy 2018 — Dus/n Ingram — @di_codes
    

    View Slide

67. View Slide

68. Problem:
    How can I specify
    dependencies?
    SciPy 2018 — Dus/n Ingram — @di_codes
    

    View Slide

69. Solu%on:
    setuptools
    SciPy 2018 — Dus/n Ingram — @di_codes
    

    View Slide

70. Problem:
    Installing is too hard!
    SciPy 2018 — Dus/n Ingram — @di_codes
    

    View Slide

71. Solu%on:
    easy_install
    SciPy 2018 — Dus/n Ingram — @di_codes
    

    View Slide

72. The Egg Distribu.on
    SciPy 2018 — Dus/n Ingram — @di_codes
    

    View Slide

73. View Slide

74. Problem:
    easy_install problems
    SciPy 2018 — Dus/n Ingram — @di_codes
    

    View Slide

75. Solu%on:
    pyinstall
    SciPy 2018 — Dus/n Ingram — @di_codes
    

    View Slide

76. Problem:
    "pyinstall" is too long
    SciPy 2018 — Dus/n Ingram — @di_codes
    

    View Slide

77. Solu%on:
    pip
    SciPy 2018 — Dus/n Ingram — @di_codes
    

    View Slide

78. eggs
    SciPy 2018 — Dus/n Ingram — @di_codes
    

    View Slide

79. Problem:
    Applica'on dependencies?
    SciPy 2018 — Dus/n Ingram — @di_codes
    

    View Slide

80. Solu%on:
    requirements.txt
    SciPy 2018 — Dus/n Ingram — @di_codes
    

    View Slide

81. pip install -r requirements.txt
    SciPy 2018 — Dus/n Ingram — @di_codes
    

    View Slide

82. Problem:
    Installing from PyPI is slow
    SciPy 2018 — Dus/n Ingram — @di_codes
    

    View Slide

83. Problem:
    Trus%ng 3rd Party Domains
    SciPy 2018 — Dus/n Ingram — @di_codes
    

    View Slide

84. Solu%on:
    PyPI begins hos-ng releases
    SciPy 2018 — Dus/n Ingram — @di_codes
    

    View Slide

85. View Slide

86. Problem:
    Scien&fic Python is a thing
    SciPy 2018 — Dus/n Ingram — @di_codes
    

    View Slide

87. Sub-Problem:
    distu&ls is not up to the task
    SciPy 2018 — Dus/n Ingram — @di_codes
    

    View Slide

88. Sub-Problem:
    Build environments?
    SciPy 2018 — Dus/n Ingram — @di_codes
    

    View Slide

89. Sub-Problem:
    I need prebuilt binaries
    SciPy 2018 — Dus/n Ingram — @di_codes
    

    View Slide

90. Sub-Problem:
    I need more than Python
    SciPy 2018 — Dus/n Ingram — @di_codes
    

    View Slide

91. Sub-Problem:
    I don't even have Python
    SciPy 2018 — Dus/n Ingram — @di_codes
    

    View Slide

92. Solu%on:
    Anaconda & Conda
    SciPy 2018 — Dus/n Ingram — @di_codes
    

    View Slide

93. Problem:
    We need built distribu-ons
    (again)
    SciPy 2018 — Dus/n Ingram — @di_codes
    

    View Slide

94. Solu%on:
    The Wheel Distribu.on
    SciPy 2018 — Dus/n Ingram — @di_codes
    

    View Slide

95. View Slide

96. View Slide

97. Problem:
    PyPI is showing it's age
    SciPy 2018 — Dus/n Ingram — @di_codes
    

    View Slide

98. View Slide

99. View Slide

100. View Slide

101. View Slide

102. Solu%on:
     Rewrite PyPI from scratch
     SciPy 2018 — Dus/n Ingram — @di_codes
     

     View Slide

103. View Slide

104. AKA:
     Warehouse
     SciPy 2018 — Dus/n Ingram — @di_codes
     

     View Slide

105. View Slide

106. The End
     SciPy 2018 — Dus/n Ingram — @di_codes
     

     View Slide

107. Current Problems
     SciPy 2018 — Dus/n Ingram — @di_codes
     

     View Slide

108. Problem:
     Packaging is s*ll kinda hard
     SciPy 2018 — Dus/n Ingram — @di_codes
     

     View Slide

109. Solu%on:
     The Python Packaging Guide
     h"ps:/
     /packaging.python.org
     SciPy 2018 — Dus/n Ingram — @di_codes
     

     View Slide

110. Solu%on:
     sampleproject
     h"ps:/
     /github.com/pypa/sampleproject
     SciPy 2018 — Dus/n Ingram — @di_codes
     

     View Slide

111. Solu%on:
     General care & maintenance
     SciPy 2018 — Dus/n Ingram — @di_codes
     

     View Slide

112. Problem:
     Packaging is a li+le too easy
     SciPy 2018 — Dus/n Ingram — @di_codes
     

     View Slide

113. Python Packaging
     So Easy a Spammer Could Do It
     SciPy 2018 — Dus/n Ingram — @di_codes
     

     View Slide

114. Problem:
     Reproducible Environments
     SciPy 2018 — Dus/n Ingram — @di_codes
     

     View Slide

115. Solu%on:
     pipfile/pipfile.lock
     h"ps:/
     /github.com/pypa/pipfile
     SciPy 2018 — Dus/n Ingram — @di_codes
     

     View Slide

116. Problem:
     Arbitrary code in setup.py
     SciPy 2018 — Dus/n Ingram — @di_codes
     

     View Slide

117. View Slide

118. Problem:
     A distu(ls/setuptools dance
     SciPy 2018 — Dus/n Ingram — @di_codes
     

     View Slide

119. Solu%on:
     PEP 517/518
     SciPy 2018 — Dus/n Ingram — @di_codes
     

     View Slide

120. View Slide

121. View Slide

122. pyproject.toml
     SciPy 2018 — Dus/n Ingram — @di_codes
     

     View Slide

123. How to help
     SciPy 2018 — Dus/n Ingram — @di_codes
     

     View Slide

124. View Slide

125. How to get help
     SciPy 2018 — Dus/n Ingram — @di_codes
     

     View Slide

126. How to get help
     • h#ps:/
     /packaging.python.org
     • Issue tracker for a given tool
     • On IRC: #pypa Freenode channel
     • h#ps:/
     /github.com/pypa/packaging-problems
     SciPy 2018 — Dus/n Ingram — @di_codes
     

     View Slide

127. Talk to me
     • Github: @di
     • Twi,er: @di_codes
     • Email: di@python.org
     SciPy 2018 — Dus/n Ingram — @di_codes
     

     View Slide

128. Summary
     SciPy 2018 — Dus/n Ingram — @di_codes
     

     View Slide

129. Thanks!
     SciPy 2018 — Dus/n Ingram — @di_codes
     

     View Slide

130. Talk to me
     • Github: @di
     • Twi,er: @di_codes
     • Email: di@python.org
     • Or, ask me ques,ons now!
     SciPy 2018 — Dus/n Ingram — @di_codes
     

     View Slide