The Packaging Gradient (Extended Edition)

A slightly longer version of The Packaging Gradient, as requested from the BayPiggies meetup group. https://www.meetup.com/BAyPIGgies/events/242072266/

Has a bit more about linking and a few refresher slides that help the overall flow of the presentation.

Transcript

1. The Packaging Gradient
   Mahmoud Hashemi
   September, 2017
   A Holistic Guide to Software Distribution
   (Extended Edition)
   

   View Slide

2. Packaging
   ⊡ Turning your code into a self-contained artifact (file)
   ⊡ For reliable transfer to and use in production environments
   a. Data centers
   b. Laptops / Phones
   But how?
   The fine art of:
   

   View Slide

3. “”
   “TBD”
   — IRL packaging instructions
   

   View Slide

4. Packaging reality
   ⊡ Developers leave packaging until the end
   ⊡ They discover what feels like a thousand options
   ⊡ They invent one more.
   a. Docs TBD
   

   View Slide

5. “”
   “The first 90% of the code accounts for the first 90% of
   development time.
   The last 10% of the code accounts for the other 90% of
   development time.”
   — Tom Cargill, Bell Labs The
   Rule
   

   View Slide

6. Packaging is not the last step.
   Production environment is everything.
   

   View Slide

7. LIVE
   CODING
   DEMO
   

   View Slide

8. Deploying the “script”
   home:~$ scp useful.py dest:
   ---
   dest:~$ python useful.py
   

   View Slide

9. python: command not found
   Oh, sorry, this bashrc is weird. Just a second...
   1
   ❌
   

   View Slide

10. ImportError: No module named requests
    This was working on my machine a second ago...
    1
    ❌❌
    

    View Slide

11. SSL certificate error: [Errno 1]
    _ssl.c:504: error:0D0890A1:asn1
    encoding routines:ASN1_verify:
    unknown message digest algorithm
    I thought we were friends.
    ❌❌❌
    

    View Slide

12. The “script”
    import requests
    resp = requests.get('https://github.com/mahmoud')
    print(resp.status)
    # (that’s it)
    

    View Slide

13. Mahmoud Hashemi
    September, 2017
    A Convenient Analogy to Software Packaging
    The Packing Gradient
    

    View Slide

14. ?
    How far are you traveling?
    (approximately)
    

    View Slide

15. A few blocks? A wallet or purse.
    A few miles? A backpack at least
    California? A carry-on bag
    The USA? A suitcase, or maybe an RV
    Otherwise? Any of the above — and a passport
    The Packing Gradient
    How far you travel influences what you pack
    low size
    high
    portability
    (compatibility)
    

    View Slide

16. The Packing Gradient
    Distance and accommodations determine what you pack.
    (hotel) (no hotel)
    

    View Slide

17. What does this look like for software?
    Back to our regularly scheduled program.
    ?
    

    View Slide

18. The standalone Python module
    ⊡ A .py file is a module
    ⊡ Standalone: only imports from the standard library
    ⊡ schema, ashes, boltons, bottle.py
    ⊡ Easy to distribute and integrate
    ● “vendoring”
    

    View Slide

19. The pure-Python package
    ⊡ A directory with a __init__.py in it is a package
    ⊡ Django, requests, hyperlink
    ⊡ Easy to install with pip
    ● pip installs packages, after all, right?
    

    View Slide

20. Pardon my dist
    ⊡ A distribution is an archive of zero or more packages
    ⊡ Built by setuptools through setup.py
    ⊡ Example: sdist
    ● Simple, source-only .tar.gz
    ● Great for pure-Python modules and packages
    ⊡ Motivational case study: PIL & Pillow
    

    View Slide

21. The full Python package
    ⊡ Python s interoperability & performance
    ⊡ Pillow, numpy, SciPy, gevent
    ⊡ wheel
    ● The shiny new binary distribution, or bdist
    ● Supports most Windows, Mac, & Linux
    (We don’t talk about .eggs anymore.)
    

    View Slide

22. Linkin’ Libs
    ⊡ Static linking vs dynamic linking
    ⊡ Some wheels are bigger than others
    ⊡ System libraries
    ● .dll & .so
    ● libcrypto (OpenSSL)
    ● libxml2 / libpng / libpython
    

    View Slide

23. In summary
    python setup.py sdist bdist_wheel upload
    The modern way to build and upload a Python package
    

    View Slide

24. FIN
    Thanks for coming!
    github.com/mahmoud
    

    View Slide

25. Questions?
    ⊡ I sure hope so!
    ⊡ Even wheel won’t let you install most types of software
    ⊡ Python’s native package management will get you:
    ● Libraries
    ● Basic dev tools
    ● (that’s it)
    

    View Slide

26. General purpose!
    No one packaging solution can contain all of Python’s power
    ⊡ Telephony infrastructure (Twilio)
    ⊡ Payments systems (PayPal)
    ⊡ Neuroscience and psychology (NeuroPy)
    ⊡ Numerical analysis and engineering (numpy, numba, and many more)
    ⊡ Animation (LucasArts, Disney, Dreamworks)
    ⊡ Gaming backends (Eve Online, Second Life, Battlefield, and so many others)
    ⊡ Email infrastructure (Mailman, Mailgun)
    ⊡ Media storage and processing (YouTube, Instagram, Dropbox)
    ⊡ Operations and systems management (Rackspace, OpenStack)
    ⊡ Natural language processing (NLTK)
    ⊡ Machine learning and computer vision (scikit-learn, Orange, SimpleCV)
    ⊡ Security and penetration testing (so many)
    ⊡ Big Data (Disco, Hadoop support)
    ⊡ Calendaring (Calendar Server/Apple iCal)
    ⊡ Search systems (ITA, Ultraseek, and Google)
    On:
    ⊡ Server
    ⊡ Console
    ⊡ Desktop
    ⊡ Embedded
    ⊡ Mobile
    ⊡ Browser
    

    View Slide

27. PyPI is not an app store
    Python is general-purpose, but pip and PyPI are not
    

    View Slide

28. pip requirements
    ⊡ A working Python with pip installed
    ⊡ A network connection, probably to the Internet
    ⊡ Preinstalled system libraries
    ⊡ Build tools for target packages (gcc/clang)
    ● For when wheels aren’t available for your platform
    Standard pip installation requires:
    

    View Slide

29. pip requirements
    You.
    A developer willing to sit and watch dependencies download,
    and debug any version conflicts, build errors, etc.
    But pip’s most important dependency:
    

    View Slide

30. Packaging Python
    ⊡ Sublime Text
    ⊡ EVE Online
    ⊡ Reddit
    “Packaging Python” vs “Python’s Packaging”
    How do we ship applications?
    

    View Slide

31. The command-line Python application
    import requests
    resp = requests.get('https://github.com/mahmoud')
    print(resp.status)
    # (3 lines is all it takes to make an application)
    

    View Slide

32. ⊡ Python EXecutable
    ⊡ Single, runnable file
    ⊡ No setup or install
    ⊡ Builds on Python’s zipimport
    □ superzippy
    □ zipapp
    ⊡ Same constraints as wheel
    Packaging Python Applications
    PEX
    PEX’s internal structure: hashbang python + zip
    

    View Slide

33. Our app on PEX
    <3 MB
    OS-specific
    Similar to wheels
    System Python
    Linux / Mac
    No setup step
    Web download
    Wiki attachment
    Slack/Hipchat
    PEX is probably The Right Way™ to distribute useful.py.
    Installation requires Artifact details Distribution options
    

    View Slide

34. Packaging Python Applications
    Anaconda’s internal layout (lib, include, bin, etc)
    ⊡ A whole new ecosystem
    ⊡ conda install applications
    □ postgres
    □ nginx
    □ python, itself
    ⊡ Built on OS and Python features
    □ Filesystem layout
    □ Python landmarking
    □ PatchELF
    Anaconda
    

    View Slide

35. Packaging Python Applications
    liblzma comes from Anaconda, libc comes from your system
    ldd is your friend!
    ⊡ Anaconda provides some self-containment
    ⊡ Some intentional dependence on the OS
    Aside: Linking
    

    View Slide

36. Packaging Python Applications
    ⊡ For fun, run ldd on
    □ Random binaries
    □ python itself
    □ .sos (esp. under .../lib/python2.7/lib-dynload)
    Aside: Linking
    

    View Slide

37. Packaging Python Applications
    Anaconda’s internal layout (lib, include, bin, etc)
    ⊡ A whole new ecosystem
    ⊡ conda install applications
    □ postgres
    □ nginx
    □ python, itself
    ⊡ Built on OS and Python features
    □ Filesystem layout
    □ Python landmarking
    □ PatchELF
    Anaconda
    

    View Slide

38. Anaconda. Is. Awesome.
    Cross-platform, Python-first package management
    and social app store.
    

    View Slide

39. Our app on conda
    Artifact details
    <1 MB
    OS-specific
    .tar.bz2
    Installation requires
    Anaconda
    Linux / Mac / Windows
    conda install step
    Distribution options
    anaconda.org
    conda is a good channel for distributing our tool to devs and data people
    

    View Slide

40. A bundle of py.
    ⊡ Bring your own Python!
    ⊡ Generally entails an installer.
    □ Dropbox
    □ EVE Online
    □ Civilization IV
    □ All Kivy apps
    ⊡ Lots of options:
    □ cx_Freeze, PyInstaller, osnap, bbFreeze,
    py2exe, py2app, pynsist, nuitka, ...
    Freezers
    Packaging Python Applications
    

    View Slide

41. Our app, frozen
    Artifact details
    <30MB
    OS-specific
    .exe / .app / etc.
    Installation requires
    Linux / Mac / Windows
    Executable or
    double-click install
    System libraries
    Distribution options
    Direct download
    App stores
    Freezers represent the best option for consumer software.
    

    View Slide

42. So many wheels going round.
    ⊡ Servers need extra attention
    ⊡ Chef Omnibus
    ⊡ Higher-level self-containment
    ⊡ Copies of all dependent services
    ⊡ Case Study: GitLab
    □ https://about.gitlab.com/installation/
    □ https://gitlab.com/gitlab-org/omnibus-gitlab
    Freezers (EE)
    Packaging Python Applications
    

    View Slide

43. Our app on Omnibus
    Artifact details
    <100MB
    .rpm / .deb
    Installation requires
    Debian-/RedHat-based
    Linux
    yum or apt
    Minimal system libraries
    Distribution options
    Direct download
    PPA / package repo
    Omnibus could probably ship a script, but works best for applications with
    multiple components (services, queues, batches)
    

    View Slide

44. Intermission: bits and pieces
    ⊡ virtualenv
    ⊡ RPM, DEB, etc.
    On their own, not so great for users
    ⊡ Example: dh-virtualenv
    □ Analogous to PEX: includes libraries, but not Python
    □ Artifacts managed by your Debian package manager
    Many technologies, many more part than whole.
    

    View Slide

45. Like selfies, but twice as trendy.
    ⊡ Bring your own userspace!
    ⊡ Anaconda meets old autostart CDs
    ⊡ Example: AppImage
    ⊡ Case study: kdenlive
    Userspace Images
    Packaging Python Applications
    

    View Slide

46. Our app in an image
    Artifact details
    <100MB
    ISO 9660
    Installation requires
    Linux
    chmod +x
    Distribution options
    Direct download
    Shared folder
    For Linux, an image is even better than a PEX, if a bit conservative.
    

    View Slide

47. Basically disposable, but they never seem to seal as well as
    they promise.
    ⊡ Userspace images
    □ + sandboxing
    □ + distribution
    ⊡ Flatpak / Snappy
    □ .deb vs .rpm round 2
    ⊡ Docker (Moby?)
    “Containers”
    Packaging Python Applications
    

    View Slide

48. Our app, containerized
    Artifact details
    100MB-2GB
    Often hidden
    Installation requires
    Linux
    (or equiv. virtualization)
    Container runtime
    “Pull” step
    Distribution options
    Linux app store
    Docker Hub/Registry
    Containers might be overkill for our script, but it wouldn’t top the charts for
    the most ridiculous image to date.
    

    View Slide

49. ⊡ Bring the whole dang kernel
    ⊡ Production-grade emulation
    ⊡ Key to the cloud
    Virtual Machines
    Screenshot of OpenStack deployment in progress.
    Packaging Python Applications
    

    View Slide

50. Our app, virtualized
    Artifact details
    100MB-8GB
    Many formats
    Installation requires
    Any OS
    Hypervisor
    Distribution options
    Direct download
    AWS
    OpenStack Glance
    While too unwieldy for our script, VMs are only getting more powerful.
    

    View Slide

51. ⊡ Just ship the box.
    ⊡ Appliances!
    Hardware
    High-latency delivery, but comes fully loaded.
    Packaging Python Applications
    

    View Slide

52. Our app on hardware
    Artifact details
    4" x 3" x 1"
    $50
    Ships in 1-3 business
    weeks
    Installation requires
    Electricity
    Distribution options
    USPS
    DHL
    A good toss
    Coming soon to a Kickstarter near you.
    

    View Slide

53. Wrapping up
    We learned:
    ⊡ Packaging is the fine art of creating a distributable software artifact
    ⊡ A packaging design is not the last phase in a project
    ⊡ Python is general purpose, PyPI and pip are not
    ⊡ Because libraries are different than applications
    

    View Slide

54. Wrapping up
    1 2 3
    Packaging for Python libraries
    1. .py - standalone modules
    2. sdist - Pure-Python packages
    3. wheel - Python packages
    (With room to spare for static vs. dynamic linking)
    

    View Slide

55. Wrapping up
    1
    1. PEX - libraries included
    2. anaconda - Python ecosystem
    3. freezers - Python included
    4. images - system libraries included
    5. containers - sandboxed images
    6. virtual machines - kernel included
    7. hardware - plug and play
    2 3 4 5 6 7
    Packaging for Python applications
    

    View Slide

56. THANKS!
    Questions?
    bit.ly/pkgrad
    @mhashemi
    

    View Slide