Upgrade to Pro — share decks privately, control downloads, hide ads and more …

The Packaging Gradient (Extended Edition)

The Packaging Gradient (Extended Edition)

A slightly longer version of The Packaging Gradient, as requested from the BayPiggies meetup group. https://www.meetup.com/BAyPIGgies/events/242072266/

Has a bit more about linking and a few refresher slides that help the overall flow of the presentation.


Mahmoud Hashemi

September 28, 2017

More Decks by Mahmoud Hashemi

Other Decks in Programming


  1. The Packaging Gradient Mahmoud Hashemi September, 2017 A Holistic Guide

    to Software Distribution (Extended Edition)
  2. Packaging ⊡ Turning your code into a self-contained artifact (file)

    ⊡ For reliable transfer to and use in production environments a. Data centers b. Laptops / Phones But how? The fine art of:
  3. “” “TBD” — IRL packaging instructions

  4. Packaging reality ⊡ Developers leave packaging until the end ⊡

    They discover what feels like a thousand options ⊡ They invent one more. a. Docs TBD
  5. “” “The first 90% of the code accounts for the

    first 90% of development time. The last 10% of the code accounts for the other 90% of development time.” — Tom Cargill, Bell Labs The Rule
  6. Packaging is not the last step. Production environment is everything.


  8. Deploying the “script” home:~$ scp useful.py dest: --- dest:~$ python

  9. python: command not found Oh, sorry, this bashrc is weird.

    Just a second... 1 ❌
  10. ImportError: No module named requests This was working on my

    machine a second ago... 1 ❌❌
  11. SSL certificate error: [Errno 1] _ssl.c:504: error:0D0890A1:asn1 encoding routines:ASN1_verify: unknown

    message digest algorithm I thought we were friends. ❌❌❌
  12. The “script” import requests resp = requests.get('https://github.com/mahmoud') print(resp.status) # (that’s

  13. Mahmoud Hashemi September, 2017 A Convenient Analogy to Software Packaging

    The Packing Gradient
  14. ? How far are you traveling? (approximately)

  15. A few blocks? A wallet or purse. A few miles?

    A backpack at least California? A carry-on bag The USA? A suitcase, or maybe an RV Otherwise? Any of the above — and a passport The Packing Gradient How far you travel influences what you pack low size high portability (compatibility)
  16. The Packing Gradient Distance and accommodations determine what you pack.

    (hotel) (no hotel)
  17. What does this look like for software? Back to our

    regularly scheduled program. ?
  18. The standalone Python module ⊡ A .py file is a

    module ⊡ Standalone: only imports from the standard library ⊡ schema, ashes, boltons, bottle.py ⊡ Easy to distribute and integrate • “vendoring”
  19. The pure-Python package ⊡ A directory with a __init__.py in

    it is a package ⊡ Django, requests, hyperlink ⊡ Easy to install with pip • pip installs packages, after all, right?
  20. Pardon my dist ⊡ A distribution is an archive of

    zero or more packages ⊡ Built by setuptools through setup.py ⊡ Example: sdist • Simple, source-only .tar.gz • Great for pure-Python modules and packages ⊡ Motivational case study: PIL & Pillow
  21. The full Python package ⊡ Python s interoperability & performance

    ⊡ Pillow, numpy, SciPy, gevent ⊡ wheel • The shiny new binary distribution, or bdist • Supports most Windows, Mac, & Linux (We don’t talk about .eggs anymore.)
  22. Linkin’ Libs ⊡ Static linking vs dynamic linking ⊡ Some

    wheels are bigger than others ⊡ System libraries • .dll & .so • libcrypto (OpenSSL) • libxml2 / libpng / libpython
  23. In summary python setup.py sdist bdist_wheel upload The modern way

    to build and upload a Python package
  24. FIN Thanks for coming! github.com/mahmoud

  25. Questions? ⊡ I sure hope so! ⊡ Even wheel won’t

    let you install most types of software ⊡ Python’s native package management will get you: • Libraries • Basic dev tools • (that’s it)
  26. General purpose! No one packaging solution can contain all of

    Python’s power ⊡ Telephony infrastructure (Twilio) ⊡ Payments systems (PayPal) ⊡ Neuroscience and psychology (NeuroPy) ⊡ Numerical analysis and engineering (numpy, numba, and many more) ⊡ Animation (LucasArts, Disney, Dreamworks) ⊡ Gaming backends (Eve Online, Second Life, Battlefield, and so many others) ⊡ Email infrastructure (Mailman, Mailgun) ⊡ Media storage and processing (YouTube, Instagram, Dropbox) ⊡ Operations and systems management (Rackspace, OpenStack) ⊡ Natural language processing (NLTK) ⊡ Machine learning and computer vision (scikit-learn, Orange, SimpleCV) ⊡ Security and penetration testing (so many) ⊡ Big Data (Disco, Hadoop support) ⊡ Calendaring (Calendar Server/Apple iCal) ⊡ Search systems (ITA, Ultraseek, and Google) On: ⊡ Server ⊡ Console ⊡ Desktop ⊡ Embedded ⊡ Mobile ⊡ Browser
  27. PyPI is not an app store Python is general-purpose, but

    pip and PyPI are not
  28. pip requirements ⊡ A working Python with pip installed ⊡

    A network connection, probably to the Internet ⊡ Preinstalled system libraries ⊡ Build tools for target packages (gcc/clang) • For when wheels aren’t available for your platform Standard pip installation requires:
  29. pip requirements You. A developer willing to sit and watch

    dependencies download, and debug any version conflicts, build errors, etc. But pip’s most important dependency:
  30. Packaging Python ⊡ Sublime Text ⊡ EVE Online ⊡ Reddit

    “Packaging Python” vs “Python’s Packaging” How do we ship applications?
  31. The command-line Python application import requests resp = requests.get('https://github.com/mahmoud') print(resp.status)

    # (3 lines is all it takes to make an application)
  32. ⊡ Python EXecutable ⊡ Single, runnable file ⊡ No setup

    or install ⊡ Builds on Python’s zipimport □ superzippy □ zipapp ⊡ Same constraints as wheel Packaging Python Applications PEX PEX’s internal structure: hashbang python + zip
  33. Our app on PEX <3 MB OS-specific Similar to wheels

    System Python Linux / Mac No setup step Web download Wiki attachment Slack/Hipchat PEX is probably The Right Way™ to distribute useful.py. Installation requires Artifact details Distribution options
  34. Packaging Python Applications Anaconda’s internal layout (lib, include, bin, etc)

    ⊡ A whole new ecosystem ⊡ conda install applications □ postgres □ nginx □ python, itself ⊡ Built on OS and Python features □ Filesystem layout □ Python landmarking □ PatchELF Anaconda
  35. Packaging Python Applications liblzma comes from Anaconda, libc comes from

    your system ldd is your friend! ⊡ Anaconda provides some self-containment ⊡ Some intentional dependence on the OS Aside: Linking
  36. Packaging Python Applications ⊡ For fun, run ldd on □

    Random binaries □ python itself □ .sos (esp. under .../lib/python2.7/lib-dynload) Aside: Linking
  37. Packaging Python Applications Anaconda’s internal layout (lib, include, bin, etc)

    ⊡ A whole new ecosystem ⊡ conda install applications □ postgres □ nginx □ python, itself ⊡ Built on OS and Python features □ Filesystem layout □ Python landmarking □ PatchELF Anaconda
  38. Anaconda. Is. Awesome. Cross-platform, Python-first package management and social app

  39. Our app on conda Artifact details <1 MB OS-specific .tar.bz2

    Installation requires Anaconda Linux / Mac / Windows conda install step Distribution options anaconda.org conda is a good channel for distributing our tool to devs and data people
  40. A bundle of py. ⊡ Bring your own Python! ⊡

    Generally entails an installer. □ Dropbox □ EVE Online □ Civilization IV □ All Kivy apps ⊡ Lots of options: □ cx_Freeze, PyInstaller, osnap, bbFreeze, py2exe, py2app, pynsist, nuitka, ... Freezers Packaging Python Applications
  41. Our app, frozen Artifact details <30MB OS-specific .exe / .app

    / etc. Installation requires Linux / Mac / Windows Executable or double-click install System libraries Distribution options Direct download App stores Freezers represent the best option for consumer software.
  42. So many wheels going round. ⊡ Servers need extra attention

    ⊡ Chef Omnibus ⊡ Higher-level self-containment ⊡ Copies of all dependent services ⊡ Case Study: GitLab □ https://about.gitlab.com/installation/ □ https://gitlab.com/gitlab-org/omnibus-gitlab Freezers (EE) Packaging Python Applications
  43. Our app on Omnibus Artifact details <100MB .rpm / .deb

    Installation requires Debian-/RedHat-based Linux yum or apt Minimal system libraries Distribution options Direct download PPA / package repo Omnibus could probably ship a script, but works best for applications with multiple components (services, queues, batches)
  44. Intermission: bits and pieces ⊡ virtualenv ⊡ RPM, DEB, etc.

    On their own, not so great for users ⊡ Example: dh-virtualenv □ Analogous to PEX: includes libraries, but not Python □ Artifacts managed by your Debian package manager Many technologies, many more part than whole.
  45. Like selfies, but twice as trendy. ⊡ Bring your own

    userspace! ⊡ Anaconda meets old autostart CDs ⊡ Example: AppImage ⊡ Case study: kdenlive Userspace Images Packaging Python Applications
  46. Our app in an image Artifact details <100MB ISO 9660

    Installation requires Linux chmod +x Distribution options Direct download Shared folder For Linux, an image is even better than a PEX, if a bit conservative.
  47. Basically disposable, but they never seem to seal as well

    as they promise. ⊡ Userspace images □ + sandboxing □ + distribution ⊡ Flatpak / Snappy □ .deb vs .rpm round 2 ⊡ Docker (Moby?) “Containers” Packaging Python Applications
  48. Our app, containerized Artifact details 100MB-2GB Often hidden Installation requires

    Linux (or equiv. virtualization) Container runtime “Pull” step Distribution options Linux app store Docker Hub/Registry Containers might be overkill for our script, but it wouldn’t top the charts for the most ridiculous image to date.
  49. ⊡ Bring the whole dang kernel ⊡ Production-grade emulation ⊡

    Key to the cloud Virtual Machines Screenshot of OpenStack deployment in progress. Packaging Python Applications
  50. Our app, virtualized Artifact details 100MB-8GB Many formats Installation requires

    Any OS Hypervisor Distribution options Direct download AWS OpenStack Glance While too unwieldy for our script, VMs are only getting more powerful.
  51. ⊡ Just ship the box. ⊡ Appliances! Hardware High-latency delivery,

    but comes fully loaded. Packaging Python Applications
  52. Our app on hardware Artifact details 4" x 3" x

    1" $50 Ships in 1-3 business weeks Installation requires Electricity Distribution options USPS DHL A good toss Coming soon to a Kickstarter near you.
  53. Wrapping up We learned: ⊡ Packaging is the fine art

    of creating a distributable software artifact ⊡ A packaging design is not the last phase in a project ⊡ Python is general purpose, PyPI and pip are not ⊡ Because libraries are different than applications
  54. Wrapping up 1 2 3 Packaging for Python libraries 1.

    .py - standalone modules 2. sdist - Pure-Python packages 3. wheel - Python packages (With room to spare for static vs. dynamic linking)
  55. Wrapping up 1 1. PEX - libraries included 2. anaconda

    - Python ecosystem 3. freezers - Python included 4. images - system libraries included 5. containers - sandboxed images 6. virtual machines - kernel included 7. hardware - plug and play 2 3 4 5 6 7 Packaging for Python applications
  56. THANKS! Questions? bit.ly/pkgrad @mhashemi