Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Deep Dive: Cloud Native Buildpacks - KubeCon EU 2019

B87c43d4be875c9b41cd436f5c364f75?s=47 hone
May 22, 2019

Deep Dive: Cloud Native Buildpacks - KubeCon EU 2019

B87c43d4be875c9b41cd436f5c364f75?s=128

hone

May 22, 2019
Tweet

More Decks by hone

Other Decks in Programming

Transcript

  1. None
  2. Terence Lee @hone02

  3. None
  4. New Buildpack API Build Detect Analysis Export where metadata about

    OCI layers generated during a previous build are made available to buildpacks where the remote layers are replaced by the generated layers where an optimal selection of compatible buildpacks is chosen and a build plan is created where buildpacks use that metadata to generate only the OCI layers that need to be replaced
  5. Components • Platform ◦ pack - Local CLI for CNB

    ◦ knative-integration - template for using CNB with knative/tekton • Implementation ◦ lifecycle - Implementation of the Buildpack API v3 ◦ libbuildpack - Go language binding for the CNB API • Core ◦ spec - Buildpack API v3 specification ◦ rfcs - RFCs for changes to CNB
  6. Deep Dive into Real Buildpacks

  7. None
  8. Creating a Builder Image

  9. builder.toml (stack) [stack] id = "heroku-18" build-image = "heroku/pack:18-build" run-image

    = "heroku/pack:18"
  10. Heroku 18 Stack • Built on top of Ubuntu 18.04

    LTS • github.com/heroku/stack-images • devcenter.heroku.com/articles/stack-packages • 2 Docker Images ◦ heroku/heroku:18 ◦ heroku/heroku:18-build
  11. Heroku-18 CNB Build Image (heroku/pack:18-build) FROM heroku/heroku:18-build # setup user

    ARG pack_uid=1000 ARG pack_gid=1000 ENV CNB_USER_ID=${pack_uid} ENV CNB_GROUP_ID=${pack_gid} RUN groupadd pack --gid ${pack_gid} && \ useradd -u ${pack_uid} -g ${pack_gid} -ms /bin/bash heroku # Set stack info ENV STACK "heroku-18" ENV CNB_STACK_ID "heroku-18" LABEL io.buildpacks.stack.id="heroku-18"
  12. Heroku-18 CNB Run Image (heroku/pack:18) FROM heroku/heroku:18 # setup user

    ARG pack_uid=1000 ARG pack_gid=1000 ENV CNB_USER_ID=${pack_uid} ENV CNB_GROUP_ID=${pack_gid} RUN groupadd pack --gid ${pack_gid} && \ useradd -u ${pack_uid} -g ${pack_gid} -ms /bin/bash heroku # Set stack info ENV STACK "heroku-18" ENV CNB_STACK_ID "heroku-18" LABEL io.buildpacks.stack.id="heroku-18"
  13. builder.toml (buildpacks) [[buildpacks]] id = "heroku/ruby" uri = "https://github.com/heroku/heroku-buildpack-ruby/..." [[buildpacks]]

    id = "heroku/procfile" uri = "https://github.com/heroku/procfile-cnb/releases/…"
  14. builder.toml (buildpack groups) [[groups]] [[groups.buildpacks]] id = "heroku/ruby" version =

    "0.0.1" [[groups.buildpacks]] id = "heroku/procfile" version = "0.2" optional = true Ruby Procfile
  15. Builder Image (heroku/buildpacks:18) $ pack create-builder builder.toml • adds buildpacks

    to /buildpacks • injects buildpack lifecycle binaries • validates user id • validates that buildpacks support stack • creates directories: /workspace /layers /platform/env • chown folders • sets working dir
  16. Docker Images • Builder ◦ heroku/buildpacks:18 • Stack ◦ Build:

    heroku/pack:18 ◦ Run: heroku/pack:18-build
  17. Heroku Buildpacks • Cloud Native Buildpacks ◦ Ruby ◦ Java/JVM

    • v2a Buildpacks + Shim ◦ Node.js ◦ Python ◦ PHP ◦ Go Ruby Procfile Java/JVM Node.js + CNB Shim Python + CNB Shim
  18. Heroku CNB Ruby Buildpack Goals: • Migration path from v2a

    buildpack • Maintenance of both v2a + v3 buildpack • Buildpacks represent a composable unit of work
  19. Heroku Ruby Buildpack Source Code /Gemfile /Gemfile.lock /app Checks for

    Gemfile PASS
  20. Detect Compatibility if [ -z "$CNB_STACK_ID" ]; then # v2

    API APP_DIR=$1 else PLATFORM_DIR=$1 PLAN=$2 # working is the cwd now # v3 API APP_DIR=$(pwd) fi
  21. Heroku Ruby Buildpack Ruby Files: /layers/<bp1>/ruby/{bin,lib}/… Launch + Build Env:

    PATH=/layers/<bp1>/ruby/bin Rubygems Files: /layers/<bp1>/gems/... Launch Env + Build Env: GEM_HOME=/layers/<bp1>/gems/ GEM_PATH=/layers/<bp1>/gems/ /Gemfile /Gemfile.lock /app Ruby 2.6.3 Linux x86_64 heroku-18 sha256(Gemfile.lock) Ruby 2.6.3 Linux x86_64 heroku-18
  22. builder.toml (buildpack groups) [[groups]] [[groups.buildpacks]] id = "heroku/ruby" version =

    "0.0.1" [[groups.buildpacks]] id = "heroku/procfile" version = "0.2" optional = true Ruby Procfile
  23. Heroku CNB Procfile Buildpack Compatibility with existing Heroku apps

  24. Procfile format <process type>: <command>

  25. Ruby Procfile web: bin/rails servers -p $PORT -e $RAILS_ENV

  26. launch.toml [[processes]] type = "<process type>" command = "<command>"

  27. launch.toml [[processes]] type = "web" command = "bin/rails servers -p

    $PORT -e $RAILS_ENV"
  28. None
  29. Decomposition Cloud Foundry Buildpacks as modular, transparent, source code processors

  30. Cloud Foundry Node.js Buildpack (v2)

  31. Cloud Foundry Node.js Engine Buildpack Cloud Foundry Yarn Buildpack Cloud

    Foundry NPM Buildpack Cloud Foundry Node.js Engine Buildpack Group #1 Group #2 Build Configuration (for detection step)
  32. Buildpack Detection A mechanism for automated buildpack, tool, and dependency

    selection
  33. Cloud Foundry Node.js Engine Buildpack Cloud Foundry Yarn Buildpack Source

    Code /package.json /package-lock.json /app.js One job: provide Node.js if the app or subsequent buildpacks need it. PASS
  34. Cloud Foundry Node.js Engine Buildpack Cloud Foundry Yarn Buildpack Source

    Code /package.json /package-lock.json /app.js PASS FAIL One job: use Yarn to install and validate node modules. Fails: Missing yarn.lock
  35. Cloud Foundry Node.js Engine Buildpack Cloud Foundry Yarn Buildpack Cloud

    Foundry NPM Buildpack Cloud Foundry Node.js Engine Buildpack Group #1 Group #2 Build Configuration (for detection step)
  36. Cloud Foundry Node.js Engine Buildpack Cloud Foundry NPM Buildpack Source

    Code /package.json /package-lock.json /app.js One job: provide Node.js if the app or subsequent buildpacks need it. PASS
  37. Cloud Foundry Node.js Engine Buildpack Cloud Foundry NPM Buildpack Source

    Code /package.json /package-lock.json /app.js PASS One job: use NPM to install and validate node modules. package.json has: “engine”: “10.3.1”, so we add [nodejs] version = “10.3.1” to the build plan. PASS
  38. Cloud Foundry Node.js Engine Buildpack Cloud Foundry Yarn Buildpack Cloud

    Foundry NPM Buildpack Cloud Foundry Node.js Engine Buildpack Group #1 Group #2 Build Configuration (for detection step)
  39. Buildpack Build An unprivileged, reproducible mechanism for incremental construction of

    OCI images.
  40. Cloud Foundry Node.js Engine Buildpack Cloud Foundry NPM Buildpack /package.json

    /package-lock.json /app.js Node.js Files: /layers/<bp1>/nodejs/{bin,lib}/… Launch + Build Env: NODE_HOME=/layers/<bp1>/nodejs [nodejs] version = “10.3.1” Node.js 10.3.1 Linux x86_64
  41. Cloud Foundry Node.js Engine Buildpack Cloud Foundry NPM Buildpack Node.js

    Files: /layers/<bp1>/nodejs/{bin,lib}/… Launch + Build Env: NODE_HOME=/layers/<bp1>/nodejs [nodejs] version = “10.3.1” Node Modules Files: /layers/<bp2>/modules/... Launch Env: NODE_PATH=/layers/<bp2>/modules /package.json /package-lock.json /app.js Node.js 10.3.1 Linux x86_64 sha256(package-lock.json) (empty)
  42. Node.js Files: /layers/<bp1>/nodejs/{bin,lib}/… Launch + Build Env: NODE_HOME=/layers/<bp1>/nodejs [nodejs] version

    = “10.3.1” Node Modules Files: /layers/<bp2>/modules/... Launch Env: NODE_PATH=/layers/<bp2>/modules /package.json /package-lock.json /app.js Node.js 10.3.1 Linux x86_64 sha256(package-lock.json) Application Layers Runtime Base Image Metadata OCI Image
  43. Node.js Files: /layers/<bp1>/nodejs/{bin,lib}/… Launch + Build Env: NODE_HOME=/layers/<bp1>/nodejs [nodejs] version

    = “10.3.1” Node Modules Files: /layers/<bp2>/modules/... Launch Env: NODE_PATH=/layers/<bp2>/modules /package.json /package-lock.json /app.js Node.js 10.3.1 Linux x86_64 sha256(package-lock.json) Application Layers Runtime Base Image Metadata OCI Image #2
  44. Doesn’t decomposition make distribution a challenge?

  45. Python Pipenv Pip Python Pip Conda Python Poetry

  46. Enter: Distribution Specification (RFC out, upcoming in v0.3.0)

  47. Python Pipenv Pip Python Pip Conda Python Poetry

  48. Conda Buildpack Python Runtime Buildpack Python Package Managers Buildpack

  49. Python Ecosystem Buildpack

  50. Node.js Engine Buildpack Yarn Buildpack NPM Buildpack Node.js Engine Buildpack

  51. Node.js Engine Buildpack Node.js Package Managers Buildpack

  52. Node.js Ecosystem Buildpack

  53. [[buildpacks]] id = "io.buildpacks.npm" name = "NPM Buildpack" version =

    "0.0.7" path = "./npm-cnb/" [buildpacks.metadata] # … [[buildpacks]] id = "io.buildpacks.yarn" name = "Yarn Buildpack" version = "0.0.6" path = "./yarn-cnb/" [buildpacks.metadata] # ... [[buildpacks]] id = "io.buildpacks.node" name = "Node Engine Buildpack" version = "0.0.5" path = "./node-cnb/" [buildpacks.metadata] # ... [[buildpacks]] id = "io.buildpacks.nodejs-pkg" name = "Node.js Package Manager Buildpack" version = "0.0.9" [[buildpacks.order]] [[buildpacks.order.groups]] id = "io.buildpacks.npm" version = "0.0.7" [[buildpacks.order]] [[buildpacks.order.groups]] id = "io.buildpacks.npm" version = "0.0.7" [[buildpacks]] id = "io.buildpacks.nodejs" name = "Node.js Ecosystem Buildpack" version = "0.1.0" [[buildpacks.order]] group = [ { id = "io.buildpacks.node", version = "0.0.5" }, { id = "io.buildpacks.nodejs-pkg", version = "0.0.9" } ] buildpack.toml
  54. /cnb/blobs/<sha256>/ /cnb/by-id/io.buildpacks.npm/0.0.7 -> /cnb/blobs/<sha256>/ /cnb/by-id/io.buildpacks.yarn/0.0.6 -> /cnb/blobs/<sha256>/ /cnb/blobs/<sha256>/ /cnb/by-id/io.buildpacks.nodejs/0.1.0 ->

    /cnb/blobs/<sha256>/ /cnb/by-id/io.buildpacks.nodejs-pkg/0.0.9 -> /cnb/blobs/<sha256>/ Stack Non-runnable OCI image, either in a Docker registry, or saved as .CNB file.
  55. /cnb/blobs/<sha256>/ /cnb/by-id/io.buildpacks.npm/0.0.7 -> /cnb/blobs/<sha256>/ /cnb/by-id/io.buildpacks.yarn/0.0.6 -> /cnb/blobs/<sha256>/ /cnb/blobs/<sha256>/ /cnb/by-id/io.buildpacks.nodejs/0.1.0 ->

    /cnb/blobs/<sha256>/ /cnb/by-id/io.buildpacks.nodejs-pkg/0.0.9 -> /cnb/blobs/<sha256>/ /usr/ /lib/ /etc/ … Stack build image
  56. Roadmap • Distribution RFC • Buildpack Registry • Support for

    scratch images • Spec for Windows Containers • Mixins (contract to support additional OS packages) • Inline Buildpacks
  57. Platform Integrations • Cloud Foundry • Heroku • Tekton •

    Riff • Dokku • Draft (potentially) • You?
  58. Try it out! • Second Public Beta – v0.2.0 ◦

    pack CLI ◦ Cloud Foundry and Heroku Buildpacks ◦ Ubuntu-based Operating System Image • Join us on Slack: ◦ slack.buildpacks.io • Join our Mailing List: ◦ lists.cncf.io/g/cncf-buildpacks