Upgrade to Pro — share decks privately, control downloads, hide ads and more …

The Packaging Gradient (Extended Edition)

The Packaging Gradient (Extended Edition)

A slightly longer version of The Packaging Gradient, as requested from the BayPiggies meetup group. https://www.meetup.com/BAyPIGgies/events/242072266/

Has a bit more about linking and a few refresher slides that help the overall flow of the presentation.

Mahmoud Hashemi

September 28, 2017

More Decks by Mahmoud Hashemi

Other Decks in Programming


  1. Packaging ⊡ Turning your code into a self-contained artifact (file)

    ⊡ For reliable transfer to and use in production environments a. Data centers b. Laptops / Phones But how? The fine art of:
  2. Packaging reality ⊡ Developers leave packaging until the end ⊡

    They discover what feels like a thousand options ⊡ They invent one more. a. Docs TBD
  3. “” “The first 90% of the code accounts for the

    first 90% of development time. The last 10% of the code accounts for the other 90% of development time.” — Tom Cargill, Bell Labs The Rule
  4. A few blocks? A wallet or purse. A few miles?

    A backpack at least California? A carry-on bag The USA? A suitcase, or maybe an RV Otherwise? Any of the above — and a passport The Packing Gradient How far you travel influences what you pack low size high portability (compatibility)
  5. The standalone Python module ⊡ A .py file is a

    module ⊡ Standalone: only imports from the standard library ⊡ schema, ashes, boltons, bottle.py ⊡ Easy to distribute and integrate • “vendoring”
  6. The pure-Python package ⊡ A directory with a __init__.py in

    it is a package ⊡ Django, requests, hyperlink ⊡ Easy to install with pip • pip installs packages, after all, right?
  7. Pardon my dist ⊡ A distribution is an archive of

    zero or more packages ⊡ Built by setuptools through setup.py ⊡ Example: sdist • Simple, source-only .tar.gz • Great for pure-Python modules and packages ⊡ Motivational case study: PIL & Pillow
  8. The full Python package ⊡ Python s interoperability & performance

    ⊡ Pillow, numpy, SciPy, gevent ⊡ wheel • The shiny new binary distribution, or bdist • Supports most Windows, Mac, & Linux (We don’t talk about .eggs anymore.)
  9. Linkin’ Libs ⊡ Static linking vs dynamic linking ⊡ Some

    wheels are bigger than others ⊡ System libraries • .dll & .so • libcrypto (OpenSSL) • libxml2 / libpng / libpython
  10. Questions? ⊡ I sure hope so! ⊡ Even wheel won’t

    let you install most types of software ⊡ Python’s native package management will get you: • Libraries • Basic dev tools • (that’s it)
  11. General purpose! No one packaging solution can contain all of

    Python’s power ⊡ Telephony infrastructure (Twilio) ⊡ Payments systems (PayPal) ⊡ Neuroscience and psychology (NeuroPy) ⊡ Numerical analysis and engineering (numpy, numba, and many more) ⊡ Animation (LucasArts, Disney, Dreamworks) ⊡ Gaming backends (Eve Online, Second Life, Battlefield, and so many others) ⊡ Email infrastructure (Mailman, Mailgun) ⊡ Media storage and processing (YouTube, Instagram, Dropbox) ⊡ Operations and systems management (Rackspace, OpenStack) ⊡ Natural language processing (NLTK) ⊡ Machine learning and computer vision (scikit-learn, Orange, SimpleCV) ⊡ Security and penetration testing (so many) ⊡ Big Data (Disco, Hadoop support) ⊡ Calendaring (Calendar Server/Apple iCal) ⊡ Search systems (ITA, Ultraseek, and Google) On: ⊡ Server ⊡ Console ⊡ Desktop ⊡ Embedded ⊡ Mobile ⊡ Browser
  12. pip requirements ⊡ A working Python with pip installed ⊡

    A network connection, probably to the Internet ⊡ Preinstalled system libraries ⊡ Build tools for target packages (gcc/clang) • For when wheels aren’t available for your platform Standard pip installation requires:
  13. pip requirements You. A developer willing to sit and watch

    dependencies download, and debug any version conflicts, build errors, etc. But pip’s most important dependency:
  14. Packaging Python ⊡ Sublime Text ⊡ EVE Online ⊡ Reddit

    “Packaging Python” vs “Python’s Packaging” How do we ship applications?
  15. ⊡ Python EXecutable ⊡ Single, runnable file ⊡ No setup

    or install ⊡ Builds on Python’s zipimport □ superzippy □ zipapp ⊡ Same constraints as wheel Packaging Python Applications PEX PEX’s internal structure: hashbang python + zip
  16. Our app on PEX <3 MB OS-specific Similar to wheels

    System Python Linux / Mac No setup step Web download Wiki attachment Slack/Hipchat PEX is probably The Right Way™ to distribute useful.py. Installation requires Artifact details Distribution options
  17. Packaging Python Applications Anaconda’s internal layout (lib, include, bin, etc)

    ⊡ A whole new ecosystem ⊡ conda install applications □ postgres □ nginx □ python, itself ⊡ Built on OS and Python features □ Filesystem layout □ Python landmarking □ PatchELF Anaconda
  18. Packaging Python Applications liblzma comes from Anaconda, libc comes from

    your system ldd is your friend! ⊡ Anaconda provides some self-containment ⊡ Some intentional dependence on the OS Aside: Linking
  19. Packaging Python Applications ⊡ For fun, run ldd on □

    Random binaries □ python itself □ .sos (esp. under .../lib/python2.7/lib-dynload) Aside: Linking
  20. Packaging Python Applications Anaconda’s internal layout (lib, include, bin, etc)

    ⊡ A whole new ecosystem ⊡ conda install applications □ postgres □ nginx □ python, itself ⊡ Built on OS and Python features □ Filesystem layout □ Python landmarking □ PatchELF Anaconda
  21. Our app on conda Artifact details <1 MB OS-specific .tar.bz2

    Installation requires Anaconda Linux / Mac / Windows conda install step Distribution options anaconda.org conda is a good channel for distributing our tool to devs and data people
  22. A bundle of py. ⊡ Bring your own Python! ⊡

    Generally entails an installer. □ Dropbox □ EVE Online □ Civilization IV □ All Kivy apps ⊡ Lots of options: □ cx_Freeze, PyInstaller, osnap, bbFreeze, py2exe, py2app, pynsist, nuitka, ... Freezers Packaging Python Applications
  23. Our app, frozen Artifact details <30MB OS-specific .exe / .app

    / etc. Installation requires Linux / Mac / Windows Executable or double-click install System libraries Distribution options Direct download App stores Freezers represent the best option for consumer software.
  24. So many wheels going round. ⊡ Servers need extra attention

    ⊡ Chef Omnibus ⊡ Higher-level self-containment ⊡ Copies of all dependent services ⊡ Case Study: GitLab □ https://about.gitlab.com/installation/ □ https://gitlab.com/gitlab-org/omnibus-gitlab Freezers (EE) Packaging Python Applications
  25. Our app on Omnibus Artifact details <100MB .rpm / .deb

    Installation requires Debian-/RedHat-based Linux yum or apt Minimal system libraries Distribution options Direct download PPA / package repo Omnibus could probably ship a script, but works best for applications with multiple components (services, queues, batches)
  26. Intermission: bits and pieces ⊡ virtualenv ⊡ RPM, DEB, etc.

    On their own, not so great for users ⊡ Example: dh-virtualenv □ Analogous to PEX: includes libraries, but not Python □ Artifacts managed by your Debian package manager Many technologies, many more part than whole.
  27. Like selfies, but twice as trendy. ⊡ Bring your own

    userspace! ⊡ Anaconda meets old autostart CDs ⊡ Example: AppImage ⊡ Case study: kdenlive Userspace Images Packaging Python Applications
  28. Our app in an image Artifact details <100MB ISO 9660

    Installation requires Linux chmod +x Distribution options Direct download Shared folder For Linux, an image is even better than a PEX, if a bit conservative.
  29. Basically disposable, but they never seem to seal as well

    as they promise. ⊡ Userspace images □ + sandboxing □ + distribution ⊡ Flatpak / Snappy □ .deb vs .rpm round 2 ⊡ Docker (Moby?) “Containers” Packaging Python Applications
  30. Our app, containerized Artifact details 100MB-2GB Often hidden Installation requires

    Linux (or equiv. virtualization) Container runtime “Pull” step Distribution options Linux app store Docker Hub/Registry Containers might be overkill for our script, but it wouldn’t top the charts for the most ridiculous image to date.
  31. ⊡ Bring the whole dang kernel ⊡ Production-grade emulation ⊡

    Key to the cloud Virtual Machines Screenshot of OpenStack deployment in progress. Packaging Python Applications
  32. Our app, virtualized Artifact details 100MB-8GB Many formats Installation requires

    Any OS Hypervisor Distribution options Direct download AWS OpenStack Glance While too unwieldy for our script, VMs are only getting more powerful.
  33. ⊡ Just ship the box. ⊡ Appliances! Hardware High-latency delivery,

    but comes fully loaded. Packaging Python Applications
  34. Our app on hardware Artifact details 4" x 3" x

    1" $50 Ships in 1-3 business weeks Installation requires Electricity Distribution options USPS DHL A good toss Coming soon to a Kickstarter near you.
  35. Wrapping up We learned: ⊡ Packaging is the fine art

    of creating a distributable software artifact ⊡ A packaging design is not the last phase in a project ⊡ Python is general purpose, PyPI and pip are not ⊡ Because libraries are different than applications
  36. Wrapping up 1 2 3 Packaging for Python libraries 1.

    .py - standalone modules 2. sdist - Pure-Python packages 3. wheel - Python packages (With room to spare for static vs. dynamic linking)
  37. Wrapping up 1 1. PEX - libraries included 2. anaconda

    - Python ecosystem 3. freezers - Python included 4. images - system libraries included 5. containers - sandboxed images 6. virtual machines - kernel included 7. hardware - plug and play 2 3 4 5 6 7 Packaging for Python applications