Upgrade to Pro — share decks privately, control downloads, hide ads and more …

The Packaging Gradient

The Packaging Gradient

There are many options to package software, particularly Python software. How do all these options fit together, and how does one decide which option is right for them? This guide takes a holistic approach, as presented at PyBay 2017.

Mahmoud Hashemi

August 12, 2017
Tweet

More Decks by Mahmoud Hashemi

Other Decks in Programming

Transcript

  1. The Packaging Gradient
    Mahmoud Hashemi
    August, 2017
    A Holistic Guide to Software Distribution

    View Slide

  2. Packaging
    ⊡ Turning your code into a self-contained artifact (file)
    ⊡ For reliable transfer to and use in production environments
    a. Data centers
    b. Laptops / Phones
    But how?
    The fine art of:

    View Slide

  3. “”
    “TBD”
    — IRL packaging instructions

    View Slide

  4. Packaging reality
    ⊡ Developers leave packaging until the end
    ⊡ They discover what feels like a thousand options
    ⊡ They invent one more.
    a. Docs TBD

    View Slide

  5. “”
    “The first 90% of the code accounts for the first 90% of
    development time.
    The last 10% of the code accounts for the other 90% of
    development time.”
    — Tom Cargill, Bell Labs The
    Rule

    View Slide

  6. Packaging is not the last step.
    Production environment is everything.

    View Slide

  7. LIVE
    CODING
    DEMO

    View Slide

  8. Deploying the “script”
    home:~$ scp useful.py dest:
    ---
    dest:~$ python useful.py

    View Slide

  9. python: command not found
    Oh, sorry, this bashrc is weird. Just a second...
    1

    View Slide

  10. ImportError: No module named requests
    This was working on my machine a second ago...
    1
    ❌❌

    View Slide

  11. SSL certificate error: [Errno 1]
    _ssl.c:504: error:0D0890A1:asn1
    encoding routines:ASN1_verify:
    unknown message digest algorithm
    I thought we were friends.
    ❌❌❌

    View Slide

  12. The “script”
    import requests
    resp = requests.get('https://github.com/mahmoud')
    print(resp.status)
    # (that’s it)

    View Slide

  13. Mahmoud Hashemi
    August, 2017
    A Convenient Analogy to Software Packaging
    The Packing Gradient

    View Slide

  14. ?
    How far are you traveling?
    (approximately)

    View Slide

  15. A few blocks? A wallet or purse.
    A few miles? A backpack at least
    California? A carry-on bag
    The USA? A suitcase, or maybe an RV
    Otherwise? Any of the above — and a passport
    The Packing Gradient
    How far you travel influences what you pack
    low size
    high
    portability

    View Slide

  16. The Packing Gradient
    Distance and accommodations determine what you pack.
    (hotel) (no hotel)

    View Slide

  17. What does this look like for software?
    Back to our regularly scheduled program.
    ?

    View Slide

  18. The standalone Python module
    ⊡ A .py file is a module
    ⊡ Standalone: only imports from the standard library
    ⊡ schema, ashes, boltons, bottle.py
    ⊡ Easy to distribute and integrate

    View Slide

  19. The pure-Python package
    ⊡ A directory with a __init__.py in it is a package
    ⊡ Django, requests, hyperlink
    ⊡ Easy to install with pip
    ● pip installs packages, after all, right?

    View Slide

  20. Pardon my dist
    ⊡ A distribution is an archive of zero or more packages
    ⊡ Built by setuptools through setup.py
    ⊡ Example: sdist
    ● Simple, source-only .tar.gz
    ● Great for pure-Python modules and packages
    ⊡ Motivational case study: PIL & Pillow

    View Slide

  21. The full Python package
    ⊡ Python s interoperability & performance
    ⊡ Pillow, numpy, SciPy, gevent
    ⊡ wheel
    ● The shiny new binary distribution, or bdist
    ● Supports most Windows, Mac, & Linux
    (We don’t talk about .eggs anymore.)

    View Slide

  22. Linkin’ Libs
    ⊡ Static linking vs dynamic linking
    ⊡ Some wheels are bigger than others
    ⊡ System libraries
    ● .dll & .so
    ● libcrypto (OpenSSL)
    ● libxml2 / libpng / libpython

    View Slide

  23. In summary
    python setup.py sdist bdist_wheel upload
    The modern way to build and upload a Python package

    View Slide

  24. FIN
    Thanks for coming!
    github.com/mahmoud

    View Slide

  25. Questions?
    ⊡ I sure hope so!
    ⊡ Even wheel won’t let you install most types of software
    ⊡ Python’s native package management will get you:
    ● Libraries
    ● Basic dev tools
    ● (that’s it)

    View Slide

  26. PyPI is not an app store
    Python is general-purpose, but pip and PyPI are not

    View Slide

  27. pip requirements
    ⊡ A working Python with pip installed
    ⊡ A network connection, probably to the Internet
    ⊡ Preinstalled system libraries
    ⊡ Build tools for target packages (gcc/clang)
    ● For when wheels aren’t available for your platform
    Standard pip installation requires:

    View Slide

  28. pip requirements
    You.
    A developer willing to sit and watch dependencies download,
    and debug any version conflicts, build errors, etc.
    But pip’s most important dependency:

    View Slide

  29. Packaging Python
    ⊡ Sublime Text
    ⊡ EVE Online
    ⊡ Reddit
    “Python Packaging” vs “Packaging Python”
    How do we ship applications?

    View Slide

  30. ⊡ Python EXecutable
    ⊡ Single, runnable file
    ⊡ No setup or install
    ⊡ Builds on Python’s zipimport
    □ superzippy
    □ zipapp
    ⊡ Same constraints as wheel
    Packaging Python Applications
    PEX
    PEX’s internal structure: hashbang python + zip

    View Slide

  31. Our app on PEX
    <3 MB
    OS-specific
    Similar to wheels
    System Python
    Linux / Mac
    No setup step
    Web download
    Wiki attachment
    Slack/Hipchat
    PEX is probably The Right Way™ to distribute useful.py.
    Installation requires Artifact details Distribution options

    View Slide

  32. Packaging Python Applications
    Anaconda’s internal layout (lib, include, bin, etc)
    ⊡ A whole new ecosystem
    ⊡ conda install applications
    □ postgres
    □ nginx
    □ python, itself
    ⊡ Built on OS and Python features
    □ Filesystem layout
    □ Python landmarking
    □ PatchELF
    Anaconda

    View Slide

  33. Anaconda. Is. Awesome.
    Cross-platform, Python-first package management
    and social app store.

    View Slide

  34. Our app on conda
    Artifact details
    <1 MB
    OS-specific
    .tar.bz2
    Installation requires
    Anaconda
    Linux / Mac / Windows
    conda install step
    Distribution options
    anaconda.org
    conda is a good channel for distributing our tool to devs and data people

    View Slide

  35. A bundle of py.
    ⊡ Bring your own Python!
    ⊡ Generally entails an installer.
    □ Dropbox
    □ EVE Online
    □ Civilization IV
    □ All Kivy apps
    ⊡ Lots of options:
    □ cx_Freeze, PyInstaller, osnap, bbFreeze,
    py2exe, py2app, pynsist, nuitka, ...
    Freezers
    Packaging Python Applications

    View Slide

  36. Our app, frozen
    Artifact details
    <30MB
    OS-specific
    .exe / .app / etc.
    Installation requires
    Linux / Mac / Windows
    Executable or
    double-click install
    System libraries
    Distribution options
    Direct download
    App stores
    Freezers represent the best option for consumer software.

    View Slide

  37. So many wheels going round.
    ⊡ Servers need extra attention
    ⊡ Chef Omnibus
    ⊡ Higher-level self-containment
    ⊡ Copies of all dependent services
    ⊡ Case Study: GitLab
    □ https://about.gitlab.com/installation/
    □ https://gitlab.com/gitlab-org/omnibus-gitlab
    Freezers (EE)
    Packaging Python Applications

    View Slide

  38. Our app on Omnibus
    Artifact details
    <100MB
    .rpm / .deb
    Installation requires
    Debian-/RedHat-based
    Linux
    yum or apt
    Minimal system libraries
    Distribution options
    Direct download
    PPA / package repo
    Omnibus could probably ship a script, but works best for applications with
    multiple components (services, queues, batches)

    View Slide

  39. Intermission: bits and pieces
    ⊡ virtualenv
    ⊡ RPM, DEB, etc.
    On their own, not so great for users
    ⊡ Example: dh-virtualenv
    □ Analogous to PEX: includes libraries, but not Python
    □ Artifacts managed by your Debian package manager
    Many technologies, many more part than whole.

    View Slide

  40. Like selfies, but twice as trendy.
    ⊡ Bring your own userspace!
    ⊡ Anaconda meets old autostart CDs
    ⊡ Example: AppImage
    ⊡ Case study: kdenlive
    Userspace Images
    Packaging Python Applications

    View Slide

  41. Our app in an image
    Artifact details
    <100MB
    ISO 9660
    Installation requires
    Linux
    chmod +x
    Distribution options
    Direct download
    Shared folder
    For Linux, an image is even better than a PEX, if a bit conservative.

    View Slide

  42. Basically disposable, but they never seem to seal as well as
    they promise.
    ⊡ Userspace images
    □ + sandboxing
    □ + distribution
    ⊡ Flatpak / Snappy
    □ .deb vs .rpm round 2
    ⊡ Docker (Moby?)
    “Containers”
    Packaging Python Applications

    View Slide

  43. Our app, containerized
    Artifact details
    100MB-2GB
    Often hidden
    Installation requires
    Linux
    (or equiv. virtualization)
    Container runtime
    “Pull” step
    Distribution options
    Linux app store
    Docker Hub/Registry
    Containers might be overkill for our script, but it wouldn’t top the charts for
    the most ridiculous image to date.

    View Slide

  44. ⊡ Bring the whole dang kernel
    ⊡ Production-grade emulation
    ⊡ Key to the cloud
    Virtual Machines
    Screenshot of OpenStack deployment in progress.
    Packaging Python Applications

    View Slide

  45. Our app, virtualized
    Artifact details
    100MB-8GB
    Many formats
    Installation requires
    Any OS
    Hypervisor
    Distribution options
    Direct download
    AWS
    OpenStack Glance
    While too unwieldy for our script, VMs are only getting more powerful.

    View Slide

  46. ⊡ Just ship the box.
    ⊡ Appliances!
    Hardware
    High-latency delivery, but comes fully loaded.
    Packaging Python Applications

    View Slide

  47. Our app on hardware
    Artifact details
    4" x 3" x 1"
    $50
    Ships in 1-3 business
    weeks
    Installation requires
    Electricity
    Distribution options
    USPS
    DHL
    A good toss
    Coming soon to a Kickstarter near you.

    View Slide

  48. Wrapping up
    We learned:
    ⊡ Packaging is the fine art of creating a distributable software artifact
    ⊡ A packaging design is not the last phase in a project
    ⊡ Python is general purpose, PyPI and pip are not
    ⊡ Because libraries are different than applications

    View Slide

  49. Wrapping up
    1 2 3
    Packaging for Python libraries
    1. .py - standalone modules
    2. sdist - Pure-Python packages
    3. wheel - Python packages
    (With room to spare for static vs. dynamic linking)

    View Slide

  50. Wrapping up
    1
    1. PEX - libraries included
    2. anaconda - Python ecosystem
    3. freezers - Python included
    4. images - system libraries included
    5. containers - sandboxed images
    6. virtual machines - kernel included
    7. hardware - plug and play
    2 3 4 5 6 7
    Packaging for Python applications

    View Slide

  51. THANKS!
    Questions?
    bit.ly/pkgrad
    @mhashemi

    View Slide