Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Reinventing Container Linux for the Wasm Era (a...

Reinventing Container Linux for the Wasm Era (and More) with System Extensions

Presented at Open Source Summit North America 2024, 17 April 2024.
Video recording: https://www.youtube.com/watch?v=OpMik3XSCi8&list=PLbzoR-pLrL6poagnac0dQuTXcmNvUHVOj&index=93

Abstract:
More than 10 years ago, CoreOS introduced the concept of a "container-optimized Linux", inspired by the ChromeOS concept of an immutable, image-based desktop operating system, that was designed for just one thing: deploying containers. Since then the Linux community has evolved tools for building distributions. In particular, systemd recently introduced system extensions (or sysexts) which have been embraced by a number of distros as an elegant way to combine components at provisioning time. After reviewing this background and explaining the necessary concepts, this session will look at how sysexts can be applied in the context of a Container Linux to build an immutable but configurable operating system adapted to a range of use cases. We will look in particular at how the Flatcar project has leveraged sysexts to replace torcx for platform-specific images, custom container runtimes, and versions tailored for deploying Wasm-based applications. The concepts, however, come from the multi-vendor UAPI group, and promise to be the foundation for how many modern Linux distros manage the balance between configurability and stability at scale.

Andy Randall

April 22, 2024
Tweet

More Decks by Andy Randall

Other Decks in Technology

Transcript

  1. Reinventing Container Linux for the Wasm Era (and More) with

    System Extensions Andrew Randall Principal PM Manager Azure Core Linux
  2. The Linux distro dichotomy Kernel + systemd Thousands of included

    packages Tens of thousands of additional optional packages Fully mutable filesystem General Purpose Linux ✅ Flexible, works for just about any application
  3. The Linux distro dichotomy Kernel + systemd Thousands of included

    packages Tens of thousands of additional optional packages Fully mutable filesystem General Purpose Linux Flexible, works for just about any application Sources: Palo Alto Neworks, Security Magazine
  4. The Linux distro dichotomy Kernel + systemd Thousands of included

    packages Tens of thousands of additional optional packages Fully mutable filesystem General Purpose Linux ✅ Flexible, works for just about any application ❌ Large attack surface area ❌ Manageability ❌ Snowflakes / config drift
  5. The Linux distro dichotomy Kernel + systemd Thousands of included

    packages Tens of thousands of additional optional packages Fully mutable filesystem General Purpose Linux Kernel + systemd Minimal (10s/100s) collection of packages Container workloads loaded at runtime Immutable filesystem Special Purpose Linux ✅ Minimal attack surface area ✅ Manageability at scale ✅ Repeatable deployments
  6. The Linux distro dichotomy Kernel + systemd Thousands of included

    packages Tens of thousands of additional optional packages Fully mutable filesystem General Purpose Linux Kernel + systemd Minimal (10s/100s) collection of packages Container workloads loaded at runtime Immutable filesystem Special Purpose Linux Minimal attack surface area Manageability at scale Repeatable deployments
  7. The Linux distro dichotomy Kernel + systemd Thousands of included

    packages Tens of thousands of additional optional packages Fully mutable filesystem General Purpose Linux Kernel + systemd Minimal (10s/100s) collection of packages Container workloads loaded at runtime Immutable filesystem Special Purpose Linux ✅ Minimal attack surface area ✅ Manageability at scale ✅ Repeatable deployments ❌ Inflexible - advanced knowledge required to modify base image
  8. Composable (Image-based) Linux Kernel + systemd Thousands of included packages

    Tens of thousands of additional optional packages Fully mutable filesystem Kernel + systemd OS extension layers loaded at boot time Container, Wasm modules, etc., loaded at runtime Immutable filesystem General Purpose Linux Composable Linux ✅ Minimal attack surface area ✅ Manageability at scale ✅ Repeatable deployments ✅ Easy to create custom OS flavors from composable system extension layers Kernel + systemd Minimal (10s/100s) collection of packages Container workloads loaded at runtime Immutable filesystem Special Purpose Linux
  9. Anatomy of a System Extension (sysext) /usr /opt An overlay

    file system containing /usr & /opt Packaged as a disk image* Loaded at boot time by systemd-sysext * typically; can also be plain directory or btrfs subvolume https://www.freedesktop.org/software/systemd/man/latest/systemd-sysext.html sysext
  10. Recent Applications in Flatcar Container Linux: 1) Torcx Replacement /

    Custom Container Runtimes ž torcx (from CoreOS) ž custom, tarball-based, complex, inflexible ž No behavior change for default (e.g. Docker, containerd) ž Easily add new runtimes (e.g. Podman) alongside or replacing standard ones sysext sysext sysext
  11. Recent Applications in Flatcar Container Linux: 2) OEM Partition ž

    Separate partition fixed at build time for platform- specific tools/agents ž Not upgradeable without reprovisioning entire node ž Sysext for each target platform ž In-place upgrades /oem sysext sysext sysext
  12. Recent Applications in Flatcar Container Linux: 3) Cluster API ž

    Custom worker node images combine OS + K8s control plane ž K8s + OS versions tied ž No in-place updates ž K8s control plane as sysext ž Stock distro images ž OS + K8s versions decoupled ž In-place updates sysext
  13. Creating Sysexts: the Flatcar Sysext Bakery files to bake +

    config + metadata https://github.com/flatcar/sysext-bakery/blob/main/README.md bake.sh sysext image (.raw) sysext Kind of like docker build Kind of like your dockerfile
  14. Publishing sysexts Create checksum upload sysext image + checksum +

    update conf to http endpoint (e.g. GitHub as part of build pipeline) Kind of like docker push sha256sum *.raw > SHA256SUMS CONF Create update conf file (optional)
  15. Baked Goods, Ready to Consume https://github.com/flatcar/sysext-bakery/releases/tag/latest • docker • docker-compose

    • kubernetes • wasmcloud* • wasmtime* • cri-o (PR in progress) • k3s (PR in progress) * we’ll come to these later
  16. A Brief Detour into Flatcar provisioning YAML Butane config (human

    readable) Butane transpiler JSON Ignition config (machine readable) This is where we want to specify the sysext(s) to use https://coreos.github.io/butane/
  17. Provisioning Flatcar with a Sysext variant: flatcar version: 1.0.0 storage:

    files: - path: /opt/extensions/wasmtime/wasmtime-17.0.1-x86-64.raw contents: source: https://github.com/flatcar/sysext-bakery/releases/download/latest/wasmtime-17.0.1-x86-64.raw links: - target: /opt/extensions/wasmtime/wasmtime-17.0.1-x86-64.raw path: /etc/extensions/wasmtime.raw hard: false YAML
  18. What about updates? OS-independent sysexts OS-dependent sysexts OS images •

    E.g. standalone go binary, no OS dependencies • systemd-sysupdate • simple semver based mechanism over https • Needs to update in lockstep with OS due to dependencies • Use OS update mechanism • Flatcar update server (Nebraska) extended to support sysexts • Sysexts part of OS image à updated as part of OS update
  19. Configuring for Updates of OS-independent Sysexts variant: flatcar version: 1.0.0

    storage: files: - path: /opt/extensions/wasmtime/wasmtime-17.0.1-x86-64.raw contents: source: https://github.com/flatcar/sysext-bakery/releases/download/latest/wasmtime-17.0.1-x86-64.raw - path: /etc/sysupdate.wasmtime.d/wasmtime.conf contents: source: https://github.com/flatcar/sysext-bakery/releases/download/latest/wasmtime.conf links: - target: /opt/extensions/wasmtime/wasmtime-17.0.1-x86-64.raw path: /etc/extensions/wasmtime.raw hard: false systemd: units: - name: systemd-sysupdate.timer enabled: true - name: systemd-sysupdate.service dropins: - name: wasmtime.conf contents: | [Service] ExecStartPre=/usr/lib/systemd/systemd-sysupdate -C wasmtime update - name: sysext.conf contents: | [Service] ExecStartPost=systemctl restart systemd-sysext YAML
  20. What if I don’t want to pull the image at

    runtime? bake_flatcar_image.sh sysext(.raw) new flatcar image including sysext https://github.com/flatcar/sysext-bakery?tab=readme-ov-file#baking-sysexts-into-flatcar-os-images // Create a qemu image (latest stable) with pre-baked wasmcloud bake_flatcar_image.sh --fetch --vendor qemu_uefi wasmcloud:wasmcloud-0.82.0-x86-64.raw
  21. Putting it all together: Wasm-Optimized Linux Wasm = Web Assembly

    By default, provably secure sandbox Most languages compile to it Runs on most OSes, architectures VERY small size, super fast start Wasm modules run in a Wasm runtime
  22. Putting it all together: Wasm-Optimized Linux Kernel + systemd Core

    wasm utils loaded during init or baked into image Wasm modules loaded at runtime Immutable filesystem Wasm-Optimized Linux No Docker sysext active – no docker binaries in OS! Worth noting that if you disable a sysext, the binaries disappear from the OS file system. Might be important for e.g. compliance.
  23. So many Wasm runtimes and tools to choose from Lunatic

    Modsurfer nerdctl runwasi SpiderLightning (slight) Spin wamr (wasm-micro- runtime) wasm3 WasmCloud WasmEdge Runtime wasmtime WaVe WaZero
  24. More Wasm Goodness in Ralph’s Bakery https://github.com/squillace/sysext-bakery This is a

    great playground for all: feel free to submit additional sysexts here or upstream Flatcar sysext-bakery for mature projects
  25. Takeaways • systemd-sysext is a promising new way to compose

    custom Linux distros • Immutable + minimal (with all the benefits that brings), but also flexible • Flatcar has already embraced as the way forward for enabling flexible deployments and customization • Great platform for production environments for Wasm and more
  26. Get involved! CNCF Special Purpose OS Working Group tag-runtime.cncf.io/wgs/spos The

    Linux Userspace API (UAPI) Group uapi-group.org Flatcar Container Linux Project github.com/flatcar/Flatcar