$30 off During Our Annual Pro Sale. View Details »

The Packaging Gradient (Extended Edition)

The Packaging Gradient (Extended Edition)

A slightly longer version of The Packaging Gradient, as requested from the BayPiggies meetup group. https://www.meetup.com/BAyPIGgies/events/242072266/

Has a bit more about linking and a few refresher slides that help the overall flow of the presentation.

Mahmoud Hashemi

September 28, 2017
Tweet

More Decks by Mahmoud Hashemi

Other Decks in Programming

Transcript

  1. The Packaging Gradient
    Mahmoud Hashemi
    September, 2017
    A Holistic Guide to Software Distribution
    (Extended Edition)

    View Slide

  2. Packaging
    ⊡ Turning your code into a self-contained artifact (file)
    ⊡ For reliable transfer to and use in production environments
    a. Data centers
    b. Laptops / Phones
    But how?
    The fine art of:

    View Slide

  3. “”
    “TBD”
    — IRL packaging instructions

    View Slide

  4. Packaging reality
    ⊡ Developers leave packaging until the end
    ⊡ They discover what feels like a thousand options
    ⊡ They invent one more.
    a. Docs TBD

    View Slide

  5. “”
    “The first 90% of the code accounts for the first 90% of
    development time.
    The last 10% of the code accounts for the other 90% of
    development time.”
    — Tom Cargill, Bell Labs The
    Rule

    View Slide

  6. Packaging is not the last step.
    Production environment is everything.

    View Slide

  7. LIVE
    CODING
    DEMO

    View Slide

  8. Deploying the “script”
    home:~$ scp useful.py dest:
    ---
    dest:~$ python useful.py

    View Slide

  9. python: command not found
    Oh, sorry, this bashrc is weird. Just a second...
    1

    View Slide

  10. ImportError: No module named requests
    This was working on my machine a second ago...
    1
    ❌❌

    View Slide

  11. SSL certificate error: [Errno 1]
    _ssl.c:504: error:0D0890A1:asn1
    encoding routines:ASN1_verify:
    unknown message digest algorithm
    I thought we were friends.
    ❌❌❌

    View Slide

  12. The “script”
    import requests
    resp = requests.get('https://github.com/mahmoud')
    print(resp.status)
    # (that’s it)

    View Slide

  13. Mahmoud Hashemi
    September, 2017
    A Convenient Analogy to Software Packaging
    The Packing Gradient

    View Slide

  14. ?
    How far are you traveling?
    (approximately)

    View Slide

  15. A few blocks? A wallet or purse.
    A few miles? A backpack at least
    California? A carry-on bag
    The USA? A suitcase, or maybe an RV
    Otherwise? Any of the above — and a passport
    The Packing Gradient
    How far you travel influences what you pack
    low size
    high
    portability
    (compatibility)

    View Slide

  16. The Packing Gradient
    Distance and accommodations determine what you pack.
    (hotel) (no hotel)

    View Slide

  17. What does this look like for software?
    Back to our regularly scheduled program.
    ?

    View Slide

  18. The standalone Python module
    ⊡ A .py file is a module
    ⊡ Standalone: only imports from the standard library
    ⊡ schema, ashes, boltons, bottle.py
    ⊡ Easy to distribute and integrate
    ● “vendoring”

    View Slide

  19. The pure-Python package
    ⊡ A directory with a __init__.py in it is a package
    ⊡ Django, requests, hyperlink
    ⊡ Easy to install with pip
    ● pip installs packages, after all, right?

    View Slide

  20. Pardon my dist
    ⊡ A distribution is an archive of zero or more packages
    ⊡ Built by setuptools through setup.py
    ⊡ Example: sdist
    ● Simple, source-only .tar.gz
    ● Great for pure-Python modules and packages
    ⊡ Motivational case study: PIL & Pillow

    View Slide

  21. The full Python package
    ⊡ Python s interoperability & performance
    ⊡ Pillow, numpy, SciPy, gevent
    ⊡ wheel
    ● The shiny new binary distribution, or bdist
    ● Supports most Windows, Mac, & Linux
    (We don’t talk about .eggs anymore.)

    View Slide

  22. Linkin’ Libs
    ⊡ Static linking vs dynamic linking
    ⊡ Some wheels are bigger than others
    ⊡ System libraries
    ● .dll & .so
    ● libcrypto (OpenSSL)
    ● libxml2 / libpng / libpython

    View Slide

  23. In summary
    python setup.py sdist bdist_wheel upload
    The modern way to build and upload a Python package

    View Slide

  24. FIN
    Thanks for coming!
    github.com/mahmoud

    View Slide

  25. Questions?
    ⊡ I sure hope so!
    ⊡ Even wheel won’t let you install most types of software
    ⊡ Python’s native package management will get you:
    ● Libraries
    ● Basic dev tools
    ● (that’s it)

    View Slide

  26. General purpose!
    No one packaging solution can contain all of Python’s power
    ⊡ Telephony infrastructure (Twilio)
    ⊡ Payments systems (PayPal)
    ⊡ Neuroscience and psychology (NeuroPy)
    ⊡ Numerical analysis and engineering (numpy, numba, and many more)
    ⊡ Animation (LucasArts, Disney, Dreamworks)
    ⊡ Gaming backends (Eve Online, Second Life, Battlefield, and so many others)
    ⊡ Email infrastructure (Mailman, Mailgun)
    ⊡ Media storage and processing (YouTube, Instagram, Dropbox)
    ⊡ Operations and systems management (Rackspace, OpenStack)
    ⊡ Natural language processing (NLTK)
    ⊡ Machine learning and computer vision (scikit-learn, Orange, SimpleCV)
    ⊡ Security and penetration testing (so many)
    ⊡ Big Data (Disco, Hadoop support)
    ⊡ Calendaring (Calendar Server/Apple iCal)
    ⊡ Search systems (ITA, Ultraseek, and Google)
    On:
    ⊡ Server
    ⊡ Console
    ⊡ Desktop
    ⊡ Embedded
    ⊡ Mobile
    ⊡ Browser

    View Slide

  27. PyPI is not an app store
    Python is general-purpose, but pip and PyPI are not

    View Slide

  28. pip requirements
    ⊡ A working Python with pip installed
    ⊡ A network connection, probably to the Internet
    ⊡ Preinstalled system libraries
    ⊡ Build tools for target packages (gcc/clang)
    ● For when wheels aren’t available for your platform
    Standard pip installation requires:

    View Slide

  29. pip requirements
    You.
    A developer willing to sit and watch dependencies download,
    and debug any version conflicts, build errors, etc.
    But pip’s most important dependency:

    View Slide

  30. Packaging Python
    ⊡ Sublime Text
    ⊡ EVE Online
    ⊡ Reddit
    “Packaging Python” vs “Python’s Packaging”
    How do we ship applications?

    View Slide

  31. The command-line Python application
    import requests
    resp = requests.get('https://github.com/mahmoud')
    print(resp.status)
    # (3 lines is all it takes to make an application)

    View Slide

  32. ⊡ Python EXecutable
    ⊡ Single, runnable file
    ⊡ No setup or install
    ⊡ Builds on Python’s zipimport
    □ superzippy
    □ zipapp
    ⊡ Same constraints as wheel
    Packaging Python Applications
    PEX
    PEX’s internal structure: hashbang python + zip

    View Slide

  33. Our app on PEX
    <3 MB
    OS-specific
    Similar to wheels
    System Python
    Linux / Mac
    No setup step
    Web download
    Wiki attachment
    Slack/Hipchat
    PEX is probably The Right Way™ to distribute useful.py.
    Installation requires Artifact details Distribution options

    View Slide

  34. Packaging Python Applications
    Anaconda’s internal layout (lib, include, bin, etc)
    ⊡ A whole new ecosystem
    ⊡ conda install applications
    □ postgres
    □ nginx
    □ python, itself
    ⊡ Built on OS and Python features
    □ Filesystem layout
    □ Python landmarking
    □ PatchELF
    Anaconda

    View Slide

  35. Packaging Python Applications
    liblzma comes from Anaconda, libc comes from your system
    ldd is your friend!
    ⊡ Anaconda provides some self-containment
    ⊡ Some intentional dependence on the OS
    Aside: Linking

    View Slide

  36. Packaging Python Applications
    ⊡ For fun, run ldd on
    □ Random binaries
    □ python itself
    □ .sos (esp. under .../lib/python2.7/lib-dynload)
    Aside: Linking

    View Slide

  37. Packaging Python Applications
    Anaconda’s internal layout (lib, include, bin, etc)
    ⊡ A whole new ecosystem
    ⊡ conda install applications
    □ postgres
    □ nginx
    □ python, itself
    ⊡ Built on OS and Python features
    □ Filesystem layout
    □ Python landmarking
    □ PatchELF
    Anaconda

    View Slide

  38. Anaconda. Is. Awesome.
    Cross-platform, Python-first package management
    and social app store.

    View Slide

  39. Our app on conda
    Artifact details
    <1 MB
    OS-specific
    .tar.bz2
    Installation requires
    Anaconda
    Linux / Mac / Windows
    conda install step
    Distribution options
    anaconda.org
    conda is a good channel for distributing our tool to devs and data people

    View Slide

  40. A bundle of py.
    ⊡ Bring your own Python!
    ⊡ Generally entails an installer.
    □ Dropbox
    □ EVE Online
    □ Civilization IV
    □ All Kivy apps
    ⊡ Lots of options:
    □ cx_Freeze, PyInstaller, osnap, bbFreeze,
    py2exe, py2app, pynsist, nuitka, ...
    Freezers
    Packaging Python Applications

    View Slide

  41. Our app, frozen
    Artifact details
    <30MB
    OS-specific
    .exe / .app / etc.
    Installation requires
    Linux / Mac / Windows
    Executable or
    double-click install
    System libraries
    Distribution options
    Direct download
    App stores
    Freezers represent the best option for consumer software.

    View Slide

  42. So many wheels going round.
    ⊡ Servers need extra attention
    ⊡ Chef Omnibus
    ⊡ Higher-level self-containment
    ⊡ Copies of all dependent services
    ⊡ Case Study: GitLab
    □ https://about.gitlab.com/installation/
    □ https://gitlab.com/gitlab-org/omnibus-gitlab
    Freezers (EE)
    Packaging Python Applications

    View Slide

  43. Our app on Omnibus
    Artifact details
    <100MB
    .rpm / .deb
    Installation requires
    Debian-/RedHat-based
    Linux
    yum or apt
    Minimal system libraries
    Distribution options
    Direct download
    PPA / package repo
    Omnibus could probably ship a script, but works best for applications with
    multiple components (services, queues, batches)

    View Slide

  44. Intermission: bits and pieces
    ⊡ virtualenv
    ⊡ RPM, DEB, etc.
    On their own, not so great for users
    ⊡ Example: dh-virtualenv
    □ Analogous to PEX: includes libraries, but not Python
    □ Artifacts managed by your Debian package manager
    Many technologies, many more part than whole.

    View Slide

  45. Like selfies, but twice as trendy.
    ⊡ Bring your own userspace!
    ⊡ Anaconda meets old autostart CDs
    ⊡ Example: AppImage
    ⊡ Case study: kdenlive
    Userspace Images
    Packaging Python Applications

    View Slide

  46. Our app in an image
    Artifact details
    <100MB
    ISO 9660
    Installation requires
    Linux
    chmod +x
    Distribution options
    Direct download
    Shared folder
    For Linux, an image is even better than a PEX, if a bit conservative.

    View Slide

  47. Basically disposable, but they never seem to seal as well as
    they promise.
    ⊡ Userspace images
    □ + sandboxing
    □ + distribution
    ⊡ Flatpak / Snappy
    □ .deb vs .rpm round 2
    ⊡ Docker (Moby?)
    “Containers”
    Packaging Python Applications

    View Slide

  48. Our app, containerized
    Artifact details
    100MB-2GB
    Often hidden
    Installation requires
    Linux
    (or equiv. virtualization)
    Container runtime
    “Pull” step
    Distribution options
    Linux app store
    Docker Hub/Registry
    Containers might be overkill for our script, but it wouldn’t top the charts for
    the most ridiculous image to date.

    View Slide

  49. ⊡ Bring the whole dang kernel
    ⊡ Production-grade emulation
    ⊡ Key to the cloud
    Virtual Machines
    Screenshot of OpenStack deployment in progress.
    Packaging Python Applications

    View Slide

  50. Our app, virtualized
    Artifact details
    100MB-8GB
    Many formats
    Installation requires
    Any OS
    Hypervisor
    Distribution options
    Direct download
    AWS
    OpenStack Glance
    While too unwieldy for our script, VMs are only getting more powerful.

    View Slide

  51. ⊡ Just ship the box.
    ⊡ Appliances!
    Hardware
    High-latency delivery, but comes fully loaded.
    Packaging Python Applications

    View Slide

  52. Our app on hardware
    Artifact details
    4" x 3" x 1"
    $50
    Ships in 1-3 business
    weeks
    Installation requires
    Electricity
    Distribution options
    USPS
    DHL
    A good toss
    Coming soon to a Kickstarter near you.

    View Slide

  53. Wrapping up
    We learned:
    ⊡ Packaging is the fine art of creating a distributable software artifact
    ⊡ A packaging design is not the last phase in a project
    ⊡ Python is general purpose, PyPI and pip are not
    ⊡ Because libraries are different than applications

    View Slide

  54. Wrapping up
    1 2 3
    Packaging for Python libraries
    1. .py - standalone modules
    2. sdist - Pure-Python packages
    3. wheel - Python packages
    (With room to spare for static vs. dynamic linking)

    View Slide

  55. Wrapping up
    1
    1. PEX - libraries included
    2. anaconda - Python ecosystem
    3. freezers - Python included
    4. images - system libraries included
    5. containers - sandboxed images
    6. virtual machines - kernel included
    7. hardware - plug and play
    2 3 4 5 6 7
    Packaging for Python applications

    View Slide

  56. THANKS!
    Questions?
    bit.ly/pkgrad
    @mhashemi

    View Slide