The Packaging Gradient (Extended Edition)

The Packaging Gradient (Extended Edition)

A slightly longer version of The Packaging Gradient, as requested from the BayPiggies meetup group.

Has a bit more about linking and a few refresher slides that help the overall flow of the presentation.


Mahmoud Hashemi

September 28, 2017


  1. 1.
  2. 2.

    Packaging ⊡ Turning your code into a self-contained artifact (file)

    ⊡ For reliable transfer to and use in production environments a. Data centers b. Laptops / Phones But how? The fine art of:
  3. 4.

    Packaging reality ⊡ Developers leave packaging until the end ⊡

    They discover what feels like a thousand options ⊡ They invent one more. a. Docs TBD
  4. 5.

    “” “The first 90% of the code accounts for the

    first 90% of development time. The last 10% of the code accounts for the other 90% of development time.” — Tom Cargill, Bell Labs The Rule
  5. 15.

    A few blocks? A wallet or purse. A few miles?

    A backpack at least California? A carry-on bag The USA? A suitcase, or maybe an RV Otherwise? Any of the above — and a passport The Packing Gradient How far you travel influences what you pack low size high portability (compatibility)
  6. 17.
  7. 18.

    The standalone Python module ⊡ A .py file is a

    module ⊡ Standalone: only imports from the standard library ⊡ schema, ashes, boltons, ⊡ Easy to distribute and integrate • “vendoring”
  8. 19.

    The pure-Python package ⊡ A directory with a in

    it is a package ⊡ Django, requests, hyperlink ⊡ Easy to install with pip • pip installs packages, after all, right?
  9. 20.

    Pardon my dist ⊡ A distribution is an archive of

    zero or more packages ⊡ Built by setuptools through ⊡ Example: sdist • Simple, source-only .tar.gz • Great for pure-Python modules and packages ⊡ Motivational case study: PIL & Pillow
  10. 21.

    The full Python package ⊡ Python s interoperability & performance

    ⊡ Pillow, numpy, SciPy, gevent ⊡ wheel • The shiny new binary distribution, or bdist • Supports most Windows, Mac, & Linux (We don’t talk about .eggs anymore.)
  11. 22.

    Linkin’ Libs ⊡ Static linking vs dynamic linking ⊡ Some

    wheels are bigger than others ⊡ System libraries • .dll & .so • libcrypto (OpenSSL) • libxml2 / libpng / libpython
  12. 25.

    Questions? ⊡ I sure hope so! ⊡ Even wheel won’t

    let you install most types of software ⊡ Python’s native package management will get you: • Libraries • Basic dev tools • (that’s it)
  13. 26.

    General purpose! No one packaging solution can contain all of

    Python’s power ⊡ Telephony infrastructure (Twilio) ⊡ Payments systems (PayPal) ⊡ Neuroscience and psychology (NeuroPy) ⊡ Numerical analysis and engineering (numpy, numba, and many more) ⊡ Animation (LucasArts, Disney, Dreamworks) ⊡ Gaming backends (Eve Online, Second Life, Battlefield, and so many others) ⊡ Email infrastructure (Mailman, Mailgun) ⊡ Media storage and processing (YouTube, Instagram, Dropbox) ⊡ Operations and systems management (Rackspace, OpenStack) ⊡ Natural language processing (NLTK) ⊡ Machine learning and computer vision (scikit-learn, Orange, SimpleCV) ⊡ Security and penetration testing (so many) ⊡ Big Data (Disco, Hadoop support) ⊡ Calendaring (Calendar Server/Apple iCal) ⊡ Search systems (ITA, Ultraseek, and Google) On: ⊡ Server ⊡ Console ⊡ Desktop ⊡ Embedded ⊡ Mobile ⊡ Browser
  14. 28.

    pip requirements ⊡ A working Python with pip installed ⊡

    A network connection, probably to the Internet ⊡ Preinstalled system libraries ⊡ Build tools for target packages (gcc/clang) • For when wheels aren’t available for your platform Standard pip installation requires:
  15. 29.

    pip requirements You. A developer willing to sit and watch

    dependencies download, and debug any version conflicts, build errors, etc. But pip’s most important dependency:
  16. 30.

    Packaging Python ⊡ Sublime Text ⊡ EVE Online ⊡ Reddit

    “Packaging Python” vs “Python’s Packaging” How do we ship applications?
  17. 32.

    ⊡ Python EXecutable ⊡ Single, runnable file ⊡ No setup

    or install ⊡ Builds on Python’s zipimport □ superzippy □ zipapp ⊡ Same constraints as wheel Packaging Python Applications PEX PEX’s internal structure: hashbang python + zip
  18. 33.

    Our app on PEX <3 MB OS-specific Similar to wheels

    System Python Linux / Mac No setup step Web download Wiki attachment Slack/Hipchat PEX is probably The Right Way™ to distribute Installation requires Artifact details Distribution options
  19. 34.

    Packaging Python Applications Anaconda’s internal layout (lib, include, bin, etc)

    ⊡ A whole new ecosystem ⊡ conda install applications □ postgres □ nginx □ python, itself ⊡ Built on OS and Python features □ Filesystem layout □ Python landmarking □ PatchELF Anaconda
  20. 35.

    Packaging Python Applications liblzma comes from Anaconda, libc comes from

    your system ldd is your friend! ⊡ Anaconda provides some self-containment ⊡ Some intentional dependence on the OS Aside: Linking
  21. 36.

    Packaging Python Applications ⊡ For fun, run ldd on □

    Random binaries □ python itself □ .sos (esp. under .../lib/python2.7/lib-dynload) Aside: Linking
  22. 37.

    Packaging Python Applications Anaconda’s internal layout (lib, include, bin, etc)

    ⊡ A whole new ecosystem ⊡ conda install applications □ postgres □ nginx □ python, itself ⊡ Built on OS and Python features □ Filesystem layout □ Python landmarking □ PatchELF Anaconda
  23. 39.

    Our app on conda Artifact details <1 MB OS-specific .tar.bz2

    Installation requires Anaconda Linux / Mac / Windows conda install step Distribution options conda is a good channel for distributing our tool to devs and data people
  24. 40.

    A bundle of py. ⊡ Bring your own Python! ⊡

    Generally entails an installer. □ Dropbox □ EVE Online □ Civilization IV □ All Kivy apps ⊡ Lots of options: □ cx_Freeze, PyInstaller, osnap, bbFreeze, py2exe, py2app, pynsist, nuitka, ... Freezers Packaging Python Applications
  25. 41.

    Our app, frozen Artifact details <30MB OS-specific .exe / .app

    / etc. Installation requires Linux / Mac / Windows Executable or double-click install System libraries Distribution options Direct download App stores Freezers represent the best option for consumer software.
  26. 42.

    So many wheels going round. ⊡ Servers need extra attention

    ⊡ Chef Omnibus ⊡ Higher-level self-containment ⊡ Copies of all dependent services ⊡ Case Study: GitLab □ □ Freezers (EE) Packaging Python Applications
  27. 43.

    Our app on Omnibus Artifact details <100MB .rpm / .deb

    Installation requires Debian-/RedHat-based Linux yum or apt Minimal system libraries Distribution options Direct download PPA / package repo Omnibus could probably ship a script, but works best for applications with multiple components (services, queues, batches)
  28. 44.

    Intermission: bits and pieces ⊡ virtualenv ⊡ RPM, DEB, etc.

    On their own, not so great for users ⊡ Example: dh-virtualenv □ Analogous to PEX: includes libraries, but not Python □ Artifacts managed by your Debian package manager Many technologies, many more part than whole.
  29. 45.

    Like selfies, but twice as trendy. ⊡ Bring your own

    userspace! ⊡ Anaconda meets old autostart CDs ⊡ Example: AppImage ⊡ Case study: kdenlive Userspace Images Packaging Python Applications
  30. 46.

    Our app in an image Artifact details <100MB ISO 9660

    Installation requires Linux chmod +x Distribution options Direct download Shared folder For Linux, an image is even better than a PEX, if a bit conservative.
  31. 47.

    Basically disposable, but they never seem to seal as well

    as they promise. ⊡ Userspace images □ + sandboxing □ + distribution ⊡ Flatpak / Snappy □ .deb vs .rpm round 2 ⊡ Docker (Moby?) “Containers” Packaging Python Applications
  32. 48.

    Our app, containerized Artifact details 100MB-2GB Often hidden Installation requires

    Linux (or equiv. virtualization) Container runtime “Pull” step Distribution options Linux app store Docker Hub/Registry Containers might be overkill for our script, but it wouldn’t top the charts for the most ridiculous image to date.
  33. 49.

    ⊡ Bring the whole dang kernel ⊡ Production-grade emulation ⊡

    Key to the cloud Virtual Machines Screenshot of OpenStack deployment in progress. Packaging Python Applications
  34. 50.

    Our app, virtualized Artifact details 100MB-8GB Many formats Installation requires

    Any OS Hypervisor Distribution options Direct download AWS OpenStack Glance While too unwieldy for our script, VMs are only getting more powerful.
  35. 51.

    ⊡ Just ship the box. ⊡ Appliances! Hardware High-latency delivery,

    but comes fully loaded. Packaging Python Applications
  36. 52.

    Our app on hardware Artifact details 4" x 3" x

    1" $50 Ships in 1-3 business weeks Installation requires Electricity Distribution options USPS DHL A good toss Coming soon to a Kickstarter near you.
  37. 53.

    Wrapping up We learned: ⊡ Packaging is the fine art

    of creating a distributable software artifact ⊡ A packaging design is not the last phase in a project ⊡ Python is general purpose, PyPI and pip are not ⊡ Because libraries are different than applications
  38. 54.

    Wrapping up 1 2 3 Packaging for Python libraries 1.

    .py - standalone modules 2. sdist - Pure-Python packages 3. wheel - Python packages (With room to spare for static vs. dynamic linking)
  39. 55.

    Wrapping up 1 1. PEX - libraries included 2. anaconda

    - Python ecosystem 3. freezers - Python included 4. images - system libraries included 5. containers - sandboxed images 6. virtual machines - kernel included 7. hardware - plug and play 2 3 4 5 6 7 Packaging for Python applications