Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Deep Dive: Cloud Native Buildpacks - KubeCon EU 2019

hone
May 22, 2019

Deep Dive: Cloud Native Buildpacks - KubeCon EU 2019

hone

May 22, 2019
Tweet

More Decks by hone

Other Decks in Programming

Transcript

  1. New Buildpack API Build Detect Analysis Export where metadata about

    OCI layers generated during a previous build are made available to buildpacks where the remote layers are replaced by the generated layers where an optimal selection of compatible buildpacks is chosen and a build plan is created where buildpacks use that metadata to generate only the OCI layers that need to be replaced
  2. Components • Platform ◦ pack - Local CLI for CNB

    ◦ knative-integration - template for using CNB with knative/tekton • Implementation ◦ lifecycle - Implementation of the Buildpack API v3 ◦ libbuildpack - Go language binding for the CNB API • Core ◦ spec - Buildpack API v3 specification ◦ rfcs - RFCs for changes to CNB
  3. Heroku 18 Stack • Built on top of Ubuntu 18.04

    LTS • github.com/heroku/stack-images • devcenter.heroku.com/articles/stack-packages • 2 Docker Images ◦ heroku/heroku:18 ◦ heroku/heroku:18-build
  4. Heroku-18 CNB Build Image (heroku/pack:18-build) FROM heroku/heroku:18-build # setup user

    ARG pack_uid=1000 ARG pack_gid=1000 ENV CNB_USER_ID=${pack_uid} ENV CNB_GROUP_ID=${pack_gid} RUN groupadd pack --gid ${pack_gid} && \ useradd -u ${pack_uid} -g ${pack_gid} -ms /bin/bash heroku # Set stack info ENV STACK "heroku-18" ENV CNB_STACK_ID "heroku-18" LABEL io.buildpacks.stack.id="heroku-18"
  5. Heroku-18 CNB Run Image (heroku/pack:18) FROM heroku/heroku:18 # setup user

    ARG pack_uid=1000 ARG pack_gid=1000 ENV CNB_USER_ID=${pack_uid} ENV CNB_GROUP_ID=${pack_gid} RUN groupadd pack --gid ${pack_gid} && \ useradd -u ${pack_uid} -g ${pack_gid} -ms /bin/bash heroku # Set stack info ENV STACK "heroku-18" ENV CNB_STACK_ID "heroku-18" LABEL io.buildpacks.stack.id="heroku-18"
  6. builder.toml (buildpack groups) [[groups]] [[groups.buildpacks]] id = "heroku/ruby" version =

    "0.0.1" [[groups.buildpacks]] id = "heroku/procfile" version = "0.2" optional = true Ruby Procfile
  7. Builder Image (heroku/buildpacks:18) $ pack create-builder builder.toml • adds buildpacks

    to /buildpacks • injects buildpack lifecycle binaries • validates user id • validates that buildpacks support stack • creates directories: /workspace /layers /platform/env • chown folders • sets working dir
  8. Heroku Buildpacks • Cloud Native Buildpacks ◦ Ruby ◦ Java/JVM

    • v2a Buildpacks + Shim ◦ Node.js ◦ Python ◦ PHP ◦ Go Ruby Procfile Java/JVM Node.js + CNB Shim Python + CNB Shim
  9. Heroku CNB Ruby Buildpack Goals: • Migration path from v2a

    buildpack • Maintenance of both v2a + v3 buildpack • Buildpacks represent a composable unit of work
  10. Detect Compatibility if [ -z "$CNB_STACK_ID" ]; then # v2

    API APP_DIR=$1 else PLATFORM_DIR=$1 PLAN=$2 # working is the cwd now # v3 API APP_DIR=$(pwd) fi
  11. Heroku Ruby Buildpack Ruby Files: /layers/<bp1>/ruby/{bin,lib}/… Launch + Build Env:

    PATH=/layers/<bp1>/ruby/bin Rubygems Files: /layers/<bp1>/gems/... Launch Env + Build Env: GEM_HOME=/layers/<bp1>/gems/ GEM_PATH=/layers/<bp1>/gems/ /Gemfile /Gemfile.lock /app Ruby 2.6.3 Linux x86_64 heroku-18 sha256(Gemfile.lock) Ruby 2.6.3 Linux x86_64 heroku-18
  12. builder.toml (buildpack groups) [[groups]] [[groups.buildpacks]] id = "heroku/ruby" version =

    "0.0.1" [[groups.buildpacks]] id = "heroku/procfile" version = "0.2" optional = true Ruby Procfile
  13. Cloud Foundry Node.js Engine Buildpack Cloud Foundry Yarn Buildpack Cloud

    Foundry NPM Buildpack Cloud Foundry Node.js Engine Buildpack Group #1 Group #2 Build Configuration (for detection step)
  14. Cloud Foundry Node.js Engine Buildpack Cloud Foundry Yarn Buildpack Source

    Code /package.json /package-lock.json /app.js One job: provide Node.js if the app or subsequent buildpacks need it. PASS
  15. Cloud Foundry Node.js Engine Buildpack Cloud Foundry Yarn Buildpack Source

    Code /package.json /package-lock.json /app.js PASS FAIL One job: use Yarn to install and validate node modules. Fails: Missing yarn.lock
  16. Cloud Foundry Node.js Engine Buildpack Cloud Foundry Yarn Buildpack Cloud

    Foundry NPM Buildpack Cloud Foundry Node.js Engine Buildpack Group #1 Group #2 Build Configuration (for detection step)
  17. Cloud Foundry Node.js Engine Buildpack Cloud Foundry NPM Buildpack Source

    Code /package.json /package-lock.json /app.js One job: provide Node.js if the app or subsequent buildpacks need it. PASS
  18. Cloud Foundry Node.js Engine Buildpack Cloud Foundry NPM Buildpack Source

    Code /package.json /package-lock.json /app.js PASS One job: use NPM to install and validate node modules. package.json has: “engine”: “10.3.1”, so we add [nodejs] version = “10.3.1” to the build plan. PASS
  19. Cloud Foundry Node.js Engine Buildpack Cloud Foundry Yarn Buildpack Cloud

    Foundry NPM Buildpack Cloud Foundry Node.js Engine Buildpack Group #1 Group #2 Build Configuration (for detection step)
  20. Cloud Foundry Node.js Engine Buildpack Cloud Foundry NPM Buildpack /package.json

    /package-lock.json /app.js Node.js Files: /layers/<bp1>/nodejs/{bin,lib}/… Launch + Build Env: NODE_HOME=/layers/<bp1>/nodejs [nodejs] version = “10.3.1” Node.js 10.3.1 Linux x86_64
  21. Cloud Foundry Node.js Engine Buildpack Cloud Foundry NPM Buildpack Node.js

    Files: /layers/<bp1>/nodejs/{bin,lib}/… Launch + Build Env: NODE_HOME=/layers/<bp1>/nodejs [nodejs] version = “10.3.1” Node Modules Files: /layers/<bp2>/modules/... Launch Env: NODE_PATH=/layers/<bp2>/modules /package.json /package-lock.json /app.js Node.js 10.3.1 Linux x86_64 sha256(package-lock.json) (empty)
  22. Node.js Files: /layers/<bp1>/nodejs/{bin,lib}/… Launch + Build Env: NODE_HOME=/layers/<bp1>/nodejs [nodejs] version

    = “10.3.1” Node Modules Files: /layers/<bp2>/modules/... Launch Env: NODE_PATH=/layers/<bp2>/modules /package.json /package-lock.json /app.js Node.js 10.3.1 Linux x86_64 sha256(package-lock.json) Application Layers Runtime Base Image Metadata OCI Image
  23. Node.js Files: /layers/<bp1>/nodejs/{bin,lib}/… Launch + Build Env: NODE_HOME=/layers/<bp1>/nodejs [nodejs] version

    = “10.3.1” Node Modules Files: /layers/<bp2>/modules/... Launch Env: NODE_PATH=/layers/<bp2>/modules /package.json /package-lock.json /app.js Node.js 10.3.1 Linux x86_64 sha256(package-lock.json) Application Layers Runtime Base Image Metadata OCI Image #2
  24. [[buildpacks]] id = "io.buildpacks.npm" name = "NPM Buildpack" version =

    "0.0.7" path = "./npm-cnb/" [buildpacks.metadata] # … [[buildpacks]] id = "io.buildpacks.yarn" name = "Yarn Buildpack" version = "0.0.6" path = "./yarn-cnb/" [buildpacks.metadata] # ... [[buildpacks]] id = "io.buildpacks.node" name = "Node Engine Buildpack" version = "0.0.5" path = "./node-cnb/" [buildpacks.metadata] # ... [[buildpacks]] id = "io.buildpacks.nodejs-pkg" name = "Node.js Package Manager Buildpack" version = "0.0.9" [[buildpacks.order]] [[buildpacks.order.groups]] id = "io.buildpacks.npm" version = "0.0.7" [[buildpacks.order]] [[buildpacks.order.groups]] id = "io.buildpacks.npm" version = "0.0.7" [[buildpacks]] id = "io.buildpacks.nodejs" name = "Node.js Ecosystem Buildpack" version = "0.1.0" [[buildpacks.order]] group = [ { id = "io.buildpacks.node", version = "0.0.5" }, { id = "io.buildpacks.nodejs-pkg", version = "0.0.9" } ] buildpack.toml
  25. /cnb/blobs/<sha256>/ /cnb/by-id/io.buildpacks.npm/0.0.7 -> /cnb/blobs/<sha256>/ /cnb/by-id/io.buildpacks.yarn/0.0.6 -> /cnb/blobs/<sha256>/ /cnb/blobs/<sha256>/ /cnb/by-id/io.buildpacks.nodejs/0.1.0 ->

    /cnb/blobs/<sha256>/ /cnb/by-id/io.buildpacks.nodejs-pkg/0.0.9 -> /cnb/blobs/<sha256>/ Stack Non-runnable OCI image, either in a Docker registry, or saved as .CNB file.
  26. Roadmap • Distribution RFC • Buildpack Registry • Support for

    scratch images • Spec for Windows Containers • Mixins (contract to support additional OS packages) • Inline Buildpacks
  27. Platform Integrations • Cloud Foundry • Heroku • Tekton •

    Riff • Dokku • Draft (potentially) • You?
  28. Try it out! • Second Public Beta – v0.2.0 ◦

    pack CLI ◦ Cloud Foundry and Heroku Buildpacks ◦ Ubuntu-based Operating System Image • Join us on Slack: ◦ slack.buildpacks.io • Join our Mailing List: ◦ lists.cncf.io/g/cncf-buildpacks