No content

Terence Lee @hone02

No content

New Buildpack API Build Detect Analysis Export where metadata about OCI layers generated during a previous build are made available to buildpacks where the remote layers are replaced by the generated layers where an optimal selection of compatible buildpacks is chosen and a build plan is created where buildpacks use that metadata to generate only the OCI layers that need to be replaced

Components ● Platform ○ pack - Local CLI for CNB ○ knative-integration - template for using CNB with knative/tekton ● Implementation ○ lifecycle - Implementation of the Buildpack API v3 ○ libbuildpack - Go language binding for the CNB API ● Core ○ spec - Buildpack API v3 specification ○ rfcs - RFCs for changes to CNB

Deep Dive into Real Buildpacks

No content

Creating a Builder Image

builder.toml (stack) [stack] id = "heroku-18" build-image = "heroku/pack:18-build" run-image = "heroku/pack:18"

Heroku 18 Stack ● Built on top of Ubuntu 18.04 LTS ● github.com/heroku/stack-images ● devcenter.heroku.com/articles/stack-packages ● 2 Docker Images ○ heroku/heroku:18 ○ heroku/heroku:18-build

Heroku-18 CNB Build Image (heroku/pack:18-build) FROM heroku/heroku:18-build # setup user ARG pack_uid=1000 ARG pack_gid=1000 ENV CNB_USER_ID=${pack_uid} ENV CNB_GROUP_ID=${pack_gid} RUN groupadd pack --gid ${pack_gid} && \ useradd -u ${pack_uid} -g ${pack_gid} -ms /bin/bash heroku # Set stack info ENV STACK "heroku-18" ENV CNB_STACK_ID "heroku-18" LABEL io.buildpacks.stack.id="heroku-18"

Heroku-18 CNB Run Image (heroku/pack:18) FROM heroku/heroku:18 # setup user ARG pack_uid=1000 ARG pack_gid=1000 ENV CNB_USER_ID=${pack_uid} ENV CNB_GROUP_ID=${pack_gid} RUN groupadd pack --gid ${pack_gid} && \ useradd -u ${pack_uid} -g ${pack_gid} -ms /bin/bash heroku # Set stack info ENV STACK "heroku-18" ENV CNB_STACK_ID "heroku-18" LABEL io.buildpacks.stack.id="heroku-18"

builder.toml (buildpacks) [[buildpacks]] id = "heroku/ruby" uri = "https://github.com/heroku/heroku-buildpack-ruby/..." [[buildpacks]] id = "heroku/procfile" uri = "https://github.com/heroku/procfile-cnb/releases/…"

builder.toml (buildpack groups) [[groups]] [[groups.buildpacks]] id = "heroku/ruby" version = "0.0.1" [[groups.buildpacks]] id = "heroku/procfile" version = "0.2" optional = true Ruby Procfile

Builder Image (heroku/buildpacks:18) $ pack create-builder builder.toml ● adds buildpacks to /buildpacks ● injects buildpack lifecycle binaries ● validates user id ● validates that buildpacks support stack ● creates directories: /workspace /layers /platform/env ● chown folders ● sets working dir

Docker Images ● Builder ○ heroku/buildpacks:18 ● Stack ○ Build: heroku/pack:18 ○ Run: heroku/pack:18-build

Heroku Buildpacks ● Cloud Native Buildpacks ○ Ruby ○ Java/JVM ● v2a Buildpacks + Shim ○ Node.js ○ Python ○ PHP ○ Go Ruby Procfile Java/JVM Node.js + CNB Shim Python + CNB Shim

Heroku CNB Ruby Buildpack Goals: ● Migration path from v2a buildpack ● Maintenance of both v2a + v3 buildpack ● Buildpacks represent a composable unit of work

Heroku Ruby Buildpack Source Code /Gemfile /Gemfile.lock /app Checks for Gemfile PASS

Detect Compatibility if [ -z "$CNB_STACK_ID" ]; then # v2 API APP_DIR=$1 else PLATFORM_DIR=$1 PLAN=$2 # working is the cwd now # v3 API APP_DIR=$(pwd) fi

Heroku Ruby Buildpack Ruby Files: /layers//ruby/{bin,lib}/… Launch + Build Env: PATH=/layers//ruby/bin Rubygems Files: /layers//gems/... Launch Env + Build Env: GEM_HOME=/layers//gems/ GEM_PATH=/layers//gems/ /Gemfile /Gemfile.lock /app Ruby 2.6.3 Linux x86_64 heroku-18 sha256(Gemfile.lock) Ruby 2.6.3 Linux x86_64 heroku-18

builder.toml (buildpack groups) [[groups]] [[groups.buildpacks]] id = "heroku/ruby" version = "0.0.1" [[groups.buildpacks]] id = "heroku/procfile" version = "0.2" optional = true Ruby Procfile

Heroku CNB Procfile Buildpack Compatibility with existing Heroku apps

Procfile format :

Ruby Procfile web: bin/rails servers -p $PORT -e $RAILS_ENV

launch.toml [[processes]] type = "" command = ""

launch.toml [[processes]] type = "web" command = "bin/rails servers -p $PORT -e $RAILS_ENV"

No content

Decomposition Cloud Foundry Buildpacks as modular, transparent, source code processors

Cloud Foundry Node.js Buildpack (v2)

Cloud Foundry Node.js Engine Buildpack Cloud Foundry Yarn Buildpack Cloud Foundry NPM Buildpack Cloud Foundry Node.js Engine Buildpack Group #1 Group #2 Build Configuration (for detection step)

Buildpack Detection A mechanism for automated buildpack, tool, and dependency selection

Cloud Foundry Node.js Engine Buildpack Cloud Foundry Yarn Buildpack Source Code /package.json /package-lock.json /app.js One job: provide Node.js if the app or subsequent buildpacks need it. PASS

Cloud Foundry Node.js Engine Buildpack Cloud Foundry Yarn Buildpack Source Code /package.json /package-lock.json /app.js PASS FAIL One job: use Yarn to install and validate node modules. Fails: Missing yarn.lock

Cloud Foundry Node.js Engine Buildpack Cloud Foundry Yarn Buildpack Cloud Foundry NPM Buildpack Cloud Foundry Node.js Engine Buildpack Group #1 Group #2 Build Configuration (for detection step)

Cloud Foundry Node.js Engine Buildpack Cloud Foundry NPM Buildpack Source Code /package.json /package-lock.json /app.js One job: provide Node.js if the app or subsequent buildpacks need it. PASS

Cloud Foundry Node.js Engine Buildpack Cloud Foundry NPM Buildpack Source Code /package.json /package-lock.json /app.js PASS One job: use NPM to install and validate node modules. package.json has: “engine”: “10.3.1”, so we add [nodejs] version = “10.3.1” to the build plan. PASS

Cloud Foundry Node.js Engine Buildpack Cloud Foundry Yarn Buildpack Cloud Foundry NPM Buildpack Cloud Foundry Node.js Engine Buildpack Group #1 Group #2 Build Configuration (for detection step)

Buildpack Build An unprivileged, reproducible mechanism for incremental construction of OCI images.

Cloud Foundry Node.js Engine Buildpack Cloud Foundry NPM Buildpack /package.json /package-lock.json /app.js Node.js Files: /layers//nodejs/{bin,lib}/… Launch + Build Env: NODE_HOME=/layers//nodejs [nodejs] version = “10.3.1” Node.js 10.3.1 Linux x86_64

Cloud Foundry Node.js Engine Buildpack Cloud Foundry NPM Buildpack Node.js Files: /layers//nodejs/{bin,lib}/… Launch + Build Env: NODE_HOME=/layers//nodejs [nodejs] version = “10.3.1” Node Modules Files: /layers//modules/... Launch Env: NODE_PATH=/layers//modules /package.json /package-lock.json /app.js Node.js 10.3.1 Linux x86_64 sha256(package-lock.json) (empty)

Node.js Files: /layers//nodejs/{bin,lib}/… Launch + Build Env: NODE_HOME=/layers//nodejs [nodejs] version = “10.3.1” Node Modules Files: /layers//modules/... Launch Env: NODE_PATH=/layers//modules /package.json /package-lock.json /app.js Node.js 10.3.1 Linux x86_64 sha256(package-lock.json) Application Layers Runtime Base Image Metadata OCI Image

Node.js Files: /layers//nodejs/{bin,lib}/… Launch + Build Env: NODE_HOME=/layers//nodejs [nodejs] version = “10.3.1” Node Modules Files: /layers//modules/... Launch Env: NODE_PATH=/layers//modules /package.json /package-lock.json /app.js Node.js 10.3.1 Linux x86_64 sha256(package-lock.json) Application Layers Runtime Base Image Metadata OCI Image #2

Doesn’t decomposition make distribution a challenge?

Python Pipenv Pip Python Pip Conda Python Poetry

Enter: Distribution Specification (RFC out, upcoming in v0.3.0)

Python Pipenv Pip Python Pip Conda Python Poetry

Conda Buildpack Python Runtime Buildpack Python Package Managers Buildpack

Python Ecosystem Buildpack

Node.js Engine Buildpack Yarn Buildpack NPM Buildpack Node.js Engine Buildpack

Node.js Engine Buildpack Node.js Package Managers Buildpack

Node.js Ecosystem Buildpack

[[buildpacks]] id = "io.buildpacks.npm" name = "NPM Buildpack" version = "0.0.7" path = "./npm-cnb/" [buildpacks.metadata] # … [[buildpacks]] id = "io.buildpacks.yarn" name = "Yarn Buildpack" version = "0.0.6" path = "./yarn-cnb/" [buildpacks.metadata] # ... [[buildpacks]] id = "io.buildpacks.node" name = "Node Engine Buildpack" version = "0.0.5" path = "./node-cnb/" [buildpacks.metadata] # ... [[buildpacks]] id = "io.buildpacks.nodejs-pkg" name = "Node.js Package Manager Buildpack" version = "0.0.9" [[buildpacks.order]] [[buildpacks.order.groups]] id = "io.buildpacks.npm" version = "0.0.7" [[buildpacks.order]] [[buildpacks.order.groups]] id = "io.buildpacks.npm" version = "0.0.7" [[buildpacks]] id = "io.buildpacks.nodejs" name = "Node.js Ecosystem Buildpack" version = "0.1.0" [[buildpacks.order]] group = [ { id = "io.buildpacks.node", version = "0.0.5" }, { id = "io.buildpacks.nodejs-pkg", version = "0.0.9" } ] buildpack.toml

/cnb/blobs// /cnb/by-id/io.buildpacks.npm/0.0.7 -> /cnb/blobs// /cnb/by-id/io.buildpacks.yarn/0.0.6 -> /cnb/blobs// /cnb/blobs// /cnb/by-id/io.buildpacks.nodejs/0.1.0 -> /cnb/blobs// /cnb/by-id/io.buildpacks.nodejs-pkg/0.0.9 -> /cnb/blobs// Stack Non-runnable OCI image, either in a Docker registry, or saved as .CNB file.

/cnb/blobs// /cnb/by-id/io.buildpacks.npm/0.0.7 -> /cnb/blobs// /cnb/by-id/io.buildpacks.yarn/0.0.6 -> /cnb/blobs// /cnb/blobs// /cnb/by-id/io.buildpacks.nodejs/0.1.0 -> /cnb/blobs// /cnb/by-id/io.buildpacks.nodejs-pkg/0.0.9 -> /cnb/blobs// /usr/ /lib/ /etc/ … Stack build image

Roadmap ● Distribution RFC ● Buildpack Registry ● Support for scratch images ● Spec for Windows Containers ● Mixins (contract to support additional OS packages) ● Inline Buildpacks

Platform Integrations ● Cloud Foundry ● Heroku ● Tekton ● Riff ● Dokku ● Draft (potentially) ● You?

Try it out! ● Second Public Beta – v0.2.0 ○ pack CLI ○ Cloud Foundry and Heroku Buildpacks ○ Ubuntu-based Operating System Image ● Join us on Slack: ○ slack.buildpacks.io ● Join our Mailing List: ○ lists.cncf.io/g/cncf-buildpacks