2017 - The Packaging Gradient

Slide 1

Slide 1 text

The Packaging Gradient Mahmoud Hashemi August, 2017 A Holistic Guide to Software Distribution

Slide 2

Slide 2 text

Packaging ⊡ Turning your code into a self-contained artifact (file) ⊡ For reliable transfer to and use in production environments a. Data centers b. Laptops / Phones But how? The fine art of:

Slide 3

Slide 3 text

“” “TBD” — IRL packaging instructions

Slide 4

Slide 4 text

Packaging reality ⊡ Developers leave packaging until the end ⊡ They discover what feels like a thousand options ⊡ They invent one more. a. Docs TBD

Slide 5

Slide 5 text

“” “The first 90% of the code accounts for the first 90% of development time. The last 10% of the code accounts for the other 90% of development time.” — Tom Cargill, Bell Labs The Rule

Slide 6

Slide 6 text

Packaging is not the last step. Production environment is everything.

Slide 7

Slide 7 text

LIVE CODING DEMO

Slide 8

Slide 8 text

Deploying the “script” home:~$ scp useful.py dest: --- dest:~$ python useful.py

Slide 9

Slide 9 text

python: command not found Oh, sorry, this bashrc is weird. Just a second... 1 ❌

Slide 10

Slide 10 text

ImportError: No module named requests This was working on my machine a second ago... 1 ❌❌

Slide 11

Slide 11 text

SSL certificate error: [Errno 1] _ssl.c:504: error:0D0890A1:asn1 encoding routines:ASN1_verify: unknown message digest algorithm I thought we were friends. ❌❌❌

Slide 12

Slide 12 text

The “script” import requests resp = requests.get('https://github.com/mahmoud') print(resp.status) # (that’s it)

Slide 13

Slide 13 text

Mahmoud Hashemi August, 2017 A Convenient Analogy to Software Packaging The Packing Gradient

Slide 14

Slide 14 text

? How far are you traveling? (approximately)

Slide 15

Slide 15 text

A few blocks? A wallet or purse. A few miles? A backpack at least California? A carry-on bag The USA? A suitcase, or maybe an RV Otherwise? Any of the above — and a passport The Packing Gradient How far you travel influences what you pack low size high portability

Slide 16

Slide 16 text

The Packing Gradient Distance and accommodations determine what you pack. (hotel) (no hotel)

Slide 17

Slide 17 text

What does this look like for software? Back to our regularly scheduled program. ?

Slide 18

Slide 18 text

The standalone Python module ⊡ A .py file is a module ⊡ Standalone: only imports from the standard library ⊡ schema, ashes, boltons, bottle.py ⊡ Easy to distribute and integrate

Slide 19

Slide 19 text

The pure-Python package ⊡ A directory with a __init__.py in it is a package ⊡ Django, requests, hyperlink ⊡ Easy to install with pip ● pip installs packages, after all, right?

Slide 20

Slide 20 text

Pardon my dist ⊡ A distribution is an archive of zero or more packages ⊡ Built by setuptools through setup.py ⊡ Example: sdist ● Simple, source-only .tar.gz ● Great for pure-Python modules and packages ⊡ Motivational case study: PIL & Pillow

Slide 21

Slide 21 text

The full Python package ⊡ Python s interoperability & performance ⊡ Pillow, numpy, SciPy, gevent ⊡ wheel ● The shiny new binary distribution, or bdist ● Supports most Windows, Mac, & Linux (We don’t talk about .eggs anymore.)

Slide 22

Slide 22 text

Linkin’ Libs ⊡ Static linking vs dynamic linking ⊡ Some wheels are bigger than others ⊡ System libraries ● .dll & .so ● libcrypto (OpenSSL) ● libxml2 / libpng / libpython

Slide 23

Slide 23 text

In summary python setup.py sdist bdist_wheel upload The modern way to build and upload a Python package

Slide 24

Slide 24 text

FIN Thanks for coming! github.com/mahmoud

Slide 25

Slide 25 text

Questions? ⊡ I sure hope so! ⊡ Even wheel won’t let you install most types of software ⊡ Python’s native package management will get you: ● Libraries ● Basic dev tools ● (that’s it)

Slide 26

Slide 26 text

PyPI is not an app store Python is general-purpose, but pip and PyPI are not

Slide 27

Slide 27 text

pip requirements ⊡ A working Python with pip installed ⊡ A network connection, probably to the Internet ⊡ Preinstalled system libraries ⊡ Build tools for target packages (gcc/clang) ● For when wheels aren’t available for your platform Standard pip installation requires:

Slide 28

Slide 28 text

pip requirements You. A developer willing to sit and watch dependencies download, and debug any version conflicts, build errors, etc. But pip’s most important dependency:

Slide 29

Slide 29 text

Packaging Python ⊡ Sublime Text ⊡ EVE Online ⊡ Reddit “Python Packaging” vs “Packaging Python” How do we ship applications?

Slide 30

Slide 30 text

⊡ Python EXecutable ⊡ Single, runnable file ⊡ No setup or install ⊡ Builds on Python’s zipimport □ superzippy □ zipapp ⊡ Same constraints as wheel Packaging Python Applications PEX PEX’s internal structure: hashbang python + zip

Slide 31

Slide 31 text

Our app on PEX <3 MB OS-specific Similar to wheels System Python Linux / Mac No setup step Web download Wiki attachment Slack/Hipchat PEX is probably The Right Way™ to distribute useful.py. Installation requires Artifact details Distribution options

Slide 32

Slide 32 text

Packaging Python Applications Anaconda’s internal layout (lib, include, bin, etc) ⊡ A whole new ecosystem ⊡ conda install applications □ postgres □ nginx □ python, itself ⊡ Built on OS and Python features □ Filesystem layout □ Python landmarking □ PatchELF Anaconda

Slide 33

Slide 33 text

Anaconda. Is. Awesome. Cross-platform, Python-first package management and social app store.

Slide 34

Slide 34 text

Our app on conda Artifact details <1 MB OS-specific .tar.bz2 Installation requires Anaconda Linux / Mac / Windows conda install step Distribution options anaconda.org conda is a good channel for distributing our tool to devs and data people

Slide 35

Slide 35 text

A bundle of py. ⊡ Bring your own Python! ⊡ Generally entails an installer. □ Dropbox □ EVE Online □ Civilization IV □ All Kivy apps ⊡ Lots of options: □ cx_Freeze, PyInstaller, osnap, bbFreeze, py2exe, py2app, pynsist, nuitka, ... Freezers Packaging Python Applications

Slide 36

Slide 36 text

Our app, frozen Artifact details <30MB OS-specific .exe / .app / etc. Installation requires Linux / Mac / Windows Executable or double-click install System libraries Distribution options Direct download App stores Freezers represent the best option for consumer software.

Slide 37

Slide 37 text

So many wheels going round. ⊡ Servers need extra attention ⊡ Chef Omnibus ⊡ Higher-level self-containment ⊡ Copies of all dependent services ⊡ Case Study: GitLab □ https://about.gitlab.com/installation/ □ https://gitlab.com/gitlab-org/omnibus-gitlab Freezers (EE) Packaging Python Applications

Slide 38

Slide 38 text

Our app on Omnibus Artifact details <100MB .rpm / .deb Installation requires Debian-/RedHat-based Linux yum or apt Minimal system libraries Distribution options Direct download PPA / package repo Omnibus could probably ship a script, but works best for applications with multiple components (services, queues, batches)

Slide 39

Slide 39 text

Intermission: bits and pieces ⊡ virtualenv ⊡ RPM, DEB, etc. On their own, not so great for users ⊡ Example: dh-virtualenv □ Analogous to PEX: includes libraries, but not Python □ Artifacts managed by your Debian package manager Many technologies, many more part than whole.

Slide 40

Slide 40 text

Like selfies, but twice as trendy. ⊡ Bring your own userspace! ⊡ Anaconda meets old autostart CDs ⊡ Example: AppImage ⊡ Case study: kdenlive Userspace Images Packaging Python Applications

Slide 41

Slide 41 text

Our app in an image Artifact details <100MB ISO 9660 Installation requires Linux chmod +x Distribution options Direct download Shared folder For Linux, an image is even better than a PEX, if a bit conservative.

Slide 42

Slide 42 text

Basically disposable, but they never seem to seal as well as they promise. ⊡ Userspace images □ + sandboxing □ + distribution ⊡ Flatpak / Snappy □ .deb vs .rpm round 2 ⊡ Docker (Moby?) “Containers” Packaging Python Applications

Slide 43

Slide 43 text

Our app, containerized Artifact details 100MB-2GB Often hidden Installation requires Linux (or equiv. virtualization) Container runtime “Pull” step Distribution options Linux app store Docker Hub/Registry Containers might be overkill for our script, but it wouldn’t top the charts for the most ridiculous image to date.

Slide 44

Slide 44 text

⊡ Bring the whole dang kernel ⊡ Production-grade emulation ⊡ Key to the cloud Virtual Machines Screenshot of OpenStack deployment in progress. Packaging Python Applications

Slide 45

Slide 45 text

Our app, virtualized Artifact details 100MB-8GB Many formats Installation requires Any OS Hypervisor Distribution options Direct download AWS OpenStack Glance While too unwieldy for our script, VMs are only getting more powerful.

Slide 46

Slide 46 text

⊡ Just ship the box. ⊡ Appliances! Hardware High-latency delivery, but comes fully loaded. Packaging Python Applications

Slide 47

Slide 47 text

Our app on hardware Artifact details 4" x 3" x 1" $50 Ships in 1-3 business weeks Installation requires Electricity Distribution options USPS DHL A good toss Coming soon to a Kickstarter near you.

Slide 48

Slide 48 text

Wrapping up We learned: ⊡ Packaging is the fine art of creating a distributable software artifact ⊡ A packaging design is not the last phase in a project ⊡ Python is general purpose, PyPI and pip are not ⊡ Because libraries are different than applications

Slide 49

Slide 49 text

Wrapping up 1 2 3 Packaging for Python libraries 1. .py - standalone modules 2. sdist - Pure-Python packages 3. wheel - Python packages (With room to spare for static vs. dynamic linking)

Slide 50

Slide 50 text

Wrapping up 1 1. PEX - libraries included 2. anaconda - Python ecosystem 3. freezers - Python included 4. images - system libraries included 5. containers - sandboxed images 6. virtual machines - kernel included 7. hardware - plug and play 2 3 4 5 6 7 Packaging for Python applications

Slide 51

Slide 51 text

THANKS! Questions? bit.ly/pkgrad @mhashemi