The Future of Linux Packaging​

What's the future of Linux distributions? Containers, npm, linuxbrew, snaps join their friends container-optimized Linux and minimal distro images and make the future of APT and RPM a little bit... cloudy. How is open source software delivery and package management changing, and where are distributions going and what it means for Linux system administrators?

Originally presented at Texas LinuxFest 2019 in Irving, TX

Transcript

1. The Future of
   Linux Packaging
   José Miguel Parrella
   (@bureado)
   Texas LinuxFest
   2019
   

   View Slide

2. Agenda
   Beyond
   distributing
   software
   Tensions are more
   than cosmetics
   High expectations
   on package
   managers
   An assessment
   framework for
   package managers
   Attributes of a
   post-modern
   package manager
   

   View Slide

3. View Slide

4. From distributing software...
   Data model
   Application
   Configuration
   Libraries
   Utilities
   Kernel & firmware
   Managed DB
   Docker Hub
   Ansible
   nix
   snapd
   apt + fwupd
   Image +
   ephemeral
   partition
   curl | sh
   Mix of packages
   and layers,
   multiple cadences
   

   View Slide

5. ...to distributing state
   Data model
   Configuration
   Application
   Libraries
   Utilities
   Kernel
   • Linux ended up everywhere
   • New forms of Linux
   • The network became faster and more
   reliable
   • We changed how we look at distributed
   systems
   • It became harder to always represent
   state as a file
   • Pets gave way to cattle
   Data model
   Configuration
   Application
   Libraries
   Utilities
   Kernel
   Data model
   Configuration
   Application
   Libraries
   Utilities
   Kernel
   Data model
   Configuration
   Application
   Libraries
   Utilities
   Kernel
   Data model
   Configuration
   Application
   Libraries
   Utilities
   Kernel & firmware
   

   View Slide

6. It's been getting specialized!
   • Library managers
   • From CPAN and PyPI to NPM and
   Golang packages
   • Next-gen package management
   • Container image specification,
   Helm charts/CNAB & registries
   • Use cases where provenance
   is controlled by final distributor
   (embedded, firmware)
   Ecosystem Debian Upstream Coverage
   Ruby 1100 9300 11.83%
   Perl 3700 31000 11.94%
   Python 3700 118000 3.14%
   Node.js 1300 350000 0.37%
   All-up libs 30K 2.8M 1.07%
   Docker Hub ? 2.3M N/A
   Helm Charts
   CNAB Bundles
   Portable
   Services
   ~0 ? ~0
   Source: libraries.io, APT lists, Docker Hub (2018)
   

   View Slide

7. View Slide

8. View Slide

9. View Slide

10. View Slide

11. View Slide

12. View Slide

13. What
    problems are
    we trying to
    solve?
    • Push the packaging responsibility upstream?
    • Be able to distribute non-free software more
    effectively and/or monetize?
    • Provide additional container and application
    security capabilities?
    • Reduce the size of a Linux distribution?
    • Make it easier to package for Linux by removing
    dependency tracking?
    • Immutable, composable and reproducible
    systems?
    • Support cloud distribution models?
    • Make the software experience easier and
    better for Linux desktop users?
    

    View Slide

14. Package managers Associated distros (if applicable)
    • Atomic unit of software distribution (snap,
    bundle, etc.)
    • What the unit actually is (tarred source,
    squashfs, etc.)
    • What the unit metadata describes
    (dependencies, origin, checksums, etc.)
    • Where the unit comes from (repository
    equivalent)
    • Core repository concepts (e.g., channels,
    governance, login, proprietary software, etc.)
    • How are updates delivered?
    • What’s the isolation/sandboxing story?
    • Universe (size) and type of apps
    • How packages are built (developer tooling)
    • Source vs. binary, binary caches, etc.
    • Any components of the system not managed as a
    unit?
    • Upgrade/rollback strategy (e.g., dual partitions
    for CoreOS)
    • What software is available (e.g., in bundles) and
    what for?
    • What versions of said software? How is
    upstream tracked?
    • Is it a rolling release?
    • How are end users expected to bring their
    applications?
    • How system state is described (e.g., version
    hashes, all-up system release numbers)
    • Coexistence with other packaging systems
    • How is package provenance validated?
    What type of software is available? How is it distributed? Who seems to be gravitating towards it?
    

    View Slide

15. View Slide

16. What else do
    we expect
    from package
    managers?
    • Rich dependencies/metadata
    • Ability to automate operations (particularly
    upgrades)
    • Ability to compose from multiple repositories
    • Signatures and other mechanisms to prevent
    tampering
    o Mixing multiple release cadences
    o Distributing multiple kinds of artifacts
    oSource provenance/supply chain security
    

    View Slide

17. Attributes of
    the post-
    modern
    package
    manager
    SW supply chain
    security
    Reproduciibilityand trusted
    builds
    Identity and fingerprinting
    Smarter repos
    Client-aware
    Flighting
    Beyond CDN
    System-wide
    observability
    Package manager
    coexistence
    System-wide indices,
    metadata
    

    View Slide

18. Thank you!
    • Slides posted to Speaker Deck
    • Find me on Twitter and GitHub: @bureado
    • Web: www.jmp.soy
    • See you tonight at Lightning Talks!
    

    View Slide

19. Framework (sample)
    Name​ Atomic unit​
    Unit
    source​
    Univers
    e​
    Isolation​ Runtimes​ Core value prop​ Use case focus​ Related​ Depends​
    Snap​ Snap(squashfs)​ Stores​
    1,500 (
    *)​
    AppArmor*,offers
    classicmode
    Core snap​
    Autoupdates
    and bundling​
    Proprietary apps and
    IoT likely​
    UbuntuCo
    re
    Systemd
    Flatpak
    Package(OSTre
    e,OCI)​
    Flathub 290
    Sandbox
    plusbubblewrap
    Runtimes(GNOM
    E,etc.)​
    Cross-
    distroportability
    Desktopapplications
    OSTree
    Atomic​
    AppStrea
    m​
    OSTree
    Systemd
    bubblewr
    ap
    Nix​ Paths​ Nixpkgs 6,500
    None (other than
    the Nixstorepath)​
    N/A​
    Atomic upgrades and
    multi-versioning
    Universal,declarativesy
    stems
    NixOS N/A​
    Guix Paths​ Hydra​ 7,660
    None (other than
    the GNUstorepath)​
    N/A​
    Ease of use​
    Transactions​
    Buildreproducibility
    Easy packaging?
    Universal​ GuixSD Guile​
    AppImag
    e
    AppImage Distribute
    d​
    150+(*
    ) to 380
    Optional via firejail N/A​ 1 app = 1 file​ Desktopapplications Firejail N/A​
    swupd Bundles​ Repos​ 180 N/A​
    A few
    (e.g.,perl,python)​
    No packages,binary
    deltasonly
    Deterministicupgrades
    in the cloud​
    Clear
    Linux​
    N/A
    ​Helm Charts​ Hub 280 Kubernetes pods​ N/A
    Complex, multi-node
    apps​
    ​Data, Dev Tools ​CNAB K8S​
    

    View Slide