Time travel: Let’s learn from the history of Python packaging!

Ever wonder about the history of Python packaging? Learn the origins of tools like pip and poetry, and discover how big tech handled packaging before adopting these solutions.

Transcript

1. Kir Chou @ PyCon TW 2020 1 Time travel: Let’s learn from the history of Python packaging!
2. Kir Chou • PyCon TW Speaker 2017, 2018, 2019 • PyCon JP Speaker 2019, 2020 • SDE @ Amazon Search 2 @k1rch0u kir.chou note35
3. Agenda • Motivation • History of Python software distribution • Learn from history • Challenge before community’s python packaging solution • Challenge after community’s python packaging solution • Takeaway • Appendix 3
4. Motivation 4
5. 5 One day, our builder Tool team announced… Let’s migrate to idiomatic tools!
6. 6 Great! But why?
7. 7 @di_codes
8. History of Python software distribution 8
9. 9 1991 Python 1.0
10. 10 How to give/take the work to/from others?
11. How to give/take the work to/from others? 11 Give Take
12. How to give/take the work to/from others? 12 Give Take
14. RFC822 14 1991 2005 2003 Python 1.0 Python 2.0 2000 PEP241 PEP345 PEP314 disutils (egg) setuptools 2006 easy_install PEP405 tox virtualenv 20072008 2011 PyPI
15. 15 Metadata: PEP241 (2001), PEP314 (2003), PEP345 (2005) Build tool: distutils, setuptools
16. 16 setuptools: setup.py Metadata in EGG xxx.egg-info/PKG-INFO
17. 17 PyPI (Cheese Shop): PEP301 (2002/Nov) Package Installer: easy_install
18. 18 Isolated Environments (Build / Test / Release): tox, virtualenv (2007), venv@PEP405 (2011)
20. 20 1991 2011 Python 1.0 2008 Python 3.0 RFC822 PEP566 PEP459 PEP426 PEP427,440 PEP376 2005 2003 Python 2.0 2000 PEP241 PEP345 PEP314 disutils (egg) setuptools 2006 wheel easy_install bento/twine PyPI pip PEP405 tox virtualenv 2007 PEP394 PEP582 poetry pipenv 2012 2017 PyPI warehouse PEP262
21. 21 Deferred Metadata 2.0: PEP426 (2012), PEP459 (2013) Metadata 2.1: PEP566 (2017/Dec)
22. 22 PEP566: pyproject.toml
23. 23 Version & Dependency: PEP386 (2009/Superseded), pip-tools/pip-sync (2012) PEP440 (2013), PEP508 (2015)
24. 24 Package installer (2008): pip Publish Tool (2013): bento / twine / flit PyPI (2018): PyPI warehouse
25. 25 Database: PEP262 (2001/Deferred), PEP376 (2009) Wheel: PEP425 (2012) / PEP427 (2012)
26. 26 PEP376 Database of Installed Python Distributions xxx.dist-info/WHEEL
28. 28 One For All: Conda / Pipenv / Poetry PEP517 (2015), PEP518 (2016)
29. 29 Learn from history
30. 30 1. Metadata Schema Evolution
31. Evolution 31 2. Build Tools 1. Metadata Schema
32. 32 2. Build Tools 1. Metadata Schema 3. Publish Tools / Package Index Evolution
33. 33 2. Build Tools 1. Metadata Schema 3. Publish Tools / Package Index Evolution 4. Installer / Deployment Tools
34. 34 2. Build Tools 1. Metadata Schema 3. Publish Tools / Package Index 5. Multiple platform Evolution 4. Installer / Deployment Tools
35. 35 2. Build Tools 1. Metadata Schema 3. Publish Tools / Package Index 5. Multiple platform / version Evolution 4. Installer / Deployment Tools
36. 36 1991 Python 1.0 wheel pipenv poetry The pattern is repeated per generation 2008 Python 3.0 tox Python 2.0 2000 disutils (egg) setuptools virtualenv bento/twine easy_install pip PyPI warehouse PyPI
37. 1994 Amazon 1998 Google 37 Challenge BEFORE community’s python packaging solution
38. 38 1991 1998 Python 1.0 Google 1994 Amazon No Python packaging solution
39. 39 How to give/take the work to/from others?
40. Solve same problems in history • Build tool • build tool + dependency resolver • Support multiple Python version • Publish tool / Private Package Registry • Installer / Deployment tool • Support isolated environment • Support multiple platforms 40
41. 41 2. Internal Build Tools 1. Internal Package Metadata 3. Internal Package Registry 4. Internal Deployment Tools
42. 42 3p packages are all handled by internal tools
44. 44 Company Community X growth rate X Community solutions grow faster now! ~2012
45. Packaging problem is mostly general! • Build System • What is a Python package? • How does build tool resolve Python’s dependencies? • How to build a Python package? • Publish System / Private Package Registry • How to publish your Python package? • How to download other Python’s dependencies? • Deployment System • How to isolate your Python environment? • Which platform does the Python package support? 45
46. Shift to community solution
47. 47 Challenge AFTER community’s python packaging solution
48. 1. Ecosystem 48 Community’s solution Company’s solution Community’s solution Company’s solution one way two way
50. 50 Cloud Services Business Cloud Service Provider Developer Community Sponsorship Pay $ Invent New solution Adopt new community solution One for all, all for one
51. 51 2. More standards ∝ More tools
54. 54 Join PyCon! Join the community!
55. 55 Review the past to understand the present Takeaway History & Past Challenges Current Challenges Community
56. Thanks! 56
57. Appendix 57
58. History • Dustin Ingram @ SciPy 2018 Inside the Cheeseshop: How Python Packaging Works • History of packaging written by Martijn Faassen • Kenneth Reitz @ PyCon 2018 Pipenv: The Future of Python Dependency Management • Clinton Roy @ Kiwi PyCon X (2019) The Packaging Lifecycle with Poetry • History of wheel written by Daniel Holth 58
59. Packaging • Dave Forgac @ PyOhio 2015 Python Packaging from Init to Deploy • Elana Hashman @ PyCon 2019 The Black Magic of Python Wheel • Official Document: Packaging binary extensions (2013) 59
60. Runtime (Virtual) Environment • Carl Meyer @ PyCon 2011 Reverse-engineering Ian Bicking's brain: inside pip and virtualenv • Bernat Gabor @ EuroPython 2019 Status quo of virtual environments 60
61. 61 Why did 3 big cloud platforms provide Package Registry service JUST recently? 2019 May 2020 June 2020 July