first 90% of development time. The last 10% of the code accounts for the other 90% of development time.” — Tom Cargill, Bell Labs (probably talking about packaging) 5 The Rule
Standalone: only imports from the standard library • schema, ashes, boltons, bottle.py • Targets Python: easy to distribute and integrate ◦ “vendoring” 9 The smallest unit of Python
a “package” • Generally includes an __init__.py • Django, requests, hyperlink, face • Easy to install with pip ◦ pip installs packages, after all, right? 10 The molecule to the module’s atom
redistributable archives • A distribution is an archive of zero or more packages • Motivational case studies: PIL & Pillow. PyCrypto(dome). • Built by setuptools through setup.py ◦ Example: sdist ◦ Simple, source-only .tar.gz 11 Packages are not proper packages.
Pillow, gevent, lxml • wheel ◦ The shiny new binary distribution, or bdist ◦ Supports most Windows, Mac, & Linux ◦ No compiler necessary 12 Python is more than just .py files.
◦ .dll and .so ◦ libcrypto (OpenSSL), libxml2, libpng, etc. • Static vs dynamic linking ◦ Big wheels • conda! 16 The world is not written in Python (yet).
day out. Rarely a product (maybe SDKs) There are two kinds of packages Applications Services and other products with high-level, non-code interfaces. Most products. 20
debug and resolve • No dependency resolution or transactional installs • Even dev tools work better with pipsi ◦ Every application deserves its own env 22 Especially not from PyPI
team in 2011 • 30+ midtier apps, services, and batch jobs ◦ Max single service volume: 1.2 billion reqs/day (2016) ◦ Max single service throughput: 10,000 reqs/sec/worker (2016) • Multiprotocol, service-focused, gevent-based framework • Hundreds of users, almost all grassroots 26 http://github.com/paypal/support Wonderfully, spontaneously Python.
binary libraries) / 5 team members = ∞ static builds. 27 PayPal Python Environment Support Matrix (2014) Operating System Architecture Python Version Linux 32-bit/64-bit 2.6/2.7 Mac 64-bit 2.7 Solaris 32-bit 2.7 Windows 32-bit 2.7
Anaconda Environment Support Matrix (2015) Operating System Architecture Python Version Linux 32-bit/64-bit 2.7-3.x Mac 64-bit 2.7-3.x Windows 32-bit/64-bit 2.7-3.x
Python 2.4 ◦ Plus 2.6-2.7 (sort of) • No Anaconda. Couldn’t just target conda install • We could bring our own by putting Miniconda in an RPM... 31 Production-first development in practice.
conda install --download-only 2. Miniconda + .tar.bz2 archives into the RPM 3. Run a tiny installer script in the RPM postinstall section Ready to test and deploy! https://www.paypal-engineering.com/2016/09/07/python-packaging-at-paypal/ https://github.com/paypal/support/blob/master/examples/miniconda 32 One way to box a snake:
https://github.com/jcrist/conda-pack RPM yourself a conda, today! (Also worth a look: https://github.com/conda/constructor ) 33 The innovation never stops:
in its own partition! • Literally v1 was ISO9660 • E.g., AppImage / kdenlive 36 Even trendier than selfies! Anaconda meets old autorun CDs https://github.com/AppImage/AppImageKit https://github.com/appimage-packages/kdenlive
into an images for local, stage, and prod • Library dependencies from: yum, conda, pip, and sky (git repo) • Service deps specified in terms of docker images • Plugins + core shipped as PEX to local developers • Open-sourced just for you: https://github.com/shopkick/sky 40
Bring your own Miniconda • Custom installer script that calls conda ◦ docker build instrumentation severely lacking. Layer proliferation. • Tips: ◦ Run after yum, before pip ◦ Size is always an issue: nomkl unless you really need the speed, conda clean. ◦ --no-channel-priority useful for mixing and matching 41
Mahmoud Hashemi
mhashemi
reverentgeek
rmw
skipperchong
carmenhchung
keathley
productmarketing
clairelew
thoeni
jcasabona
