Cloud Native Buildpacks Intro

Joe Kutner @codefinger

Terence Lee @hone02

No content

k8s is a great platform to run images

What's the best way to build images?

No content

Containerize a Ruby App

Dockerfile (Ruby) FROM ruby COPY . /app WORKDIR /app RUN bundle install EXPOSE 5000 CMD bundle exec ruby ./app.rb

Building Docker Images $ docker build .

Building Docker Images $ docker build . Step 1/6 : FROM ruby ---> 88666731c3e1 Step 2/6 : COPY . /app ---> 173624d82900 Step 3/6 : WORKDIR /app ---> Running in 0649f4408d91 ---> 850a46b3ec29 Step 4/6 : RUN bundle install ---> Running in ed644f258949 ---> 499852b8c318 Step 5/6 : EXPOSE 5000 ---> Running in fc0958926d74 ---> 4d4fbc35cde5 Step 6/6 : CMD bundle exec ruby ./app.rb ---> Running in 0e901e0910db ---> 881870f15126 Successfully built 881870f15126

Containerize Rails?

Dockerfile (Rails) FROM ruby COPY . /app WORKDIR /app RUN bundle install EXPOSE 5000 CMD bin/rails

Dockerfile (Rails) FROM ruby RUN apt-get update -qq \ && apt-get install -y nodejs libpq-dev build-essential COPY . /app WORKDIR /app RUN bundle install RUN bundle exec rake assets:precompile EXPOSE 5000 CMD bin/rails

Build image best practices? ● Reduce image size ● Speed up incremental builds ● Pick base image

Dockerfile Layers Step 1/6 : FROM ruby ---> 88666731c3e1 Step 2/6 : COPY . /app ---> 173624d82900 Step 3/6 : WORKDIR /app ---> Running in 0649f4408d91 ---> 850a46b3ec29 Step 4/6 : RUN bundle install ---> Running in ed644f258949 ---> 499852b8c318 Step 5/6 : EXPOSE 5000 ---> Running in fc0958926d74 ---> 4d4fbc35cde5 Step 6/6 : CMD bundle exec ruby ./app.rb ---> Running in 0e901e0910db ---> 881870f15126 Successfully built 881870f15126 FROM ruby:latest COPY . /app WORKDIR /app RUN bundle install EXPOSE 5000 CMD bundle exec ruby ./app.rb

Reduce image size FROM ruby RUN apt-get update -qq \ && apt-get install -y nodejs libpq-dev build-essential \ && apt-get clean autoclean && apt-get autoremove -y \ && rm -rf /var/lib/apt /var/lib/dpkg /var/lib/cache /var/lib/log COPY . /app WORKDIR /app RUN bundle install RUN bundle exec rake assets:precompile \ && rm -rf /app/tmp/cache/assets/ EXPOSE 5000 CMD bin/rails

Speed Up Incremental Builds FROM ruby RUN apt-get update -qq \ && apt-get install -y nodejs libpq-dev build-essential \ && apt-get clean autoclean && apt-get autoremove -y \ && rm -rf /var/lib/apt /var/lib/dpkg /var/lib/cache /var/lib/log ADD Gemfile /app/ ADD Gemfile.lock /app/ RUN bundle install COPY . /app WORKDIR /app RUN bundle exec rake assets:precompile \ && rm -rf /app/tmp/cache/assets/ EXPOSE 5000 CMD bin/rails

Pick Base Image FROM ruby RUN apt-get update -qq \ && apt-get install -y nodejs libpq-dev build-essential COPY . /app WORKDIR /app RUN bundle install RUN bundle exec rake assets:precompile EXPOSE 5000 CMD bin/rails

hub.docker.com/_/ruby/

“Writing a quality Dockerfile is still my users' biggest point of friction” - David Dollar, CEO, Convox

Dockerfile Shortcomings ● Maintenance / Day 2 Operations (Security) ● Not App Aware ● Composability ● Leaky Abstraction

Maintenance / Day 2 Operations FROM ruby:latest RUN mkdir /usr/src/app ADD . /usr/src/app/ WORKDIR /usr/src/app/ RUN bundle install CMD ["/usr/src/app/main.rb"]

Composability How do we combine two Docker images?

Composability FROM openjdk:11-jdk as jdk COPY . /app WORKDIR /app RUN ./mvnw clean install FROM ruby COPY --from=jdk /docker-java-home /docker-java-home COPY . /app

Composability FROM openjdk:11-jdk as jdk COPY . /app WORKDIR /app RUN ./mvnw clean install FROM ruby COPY --from=jdk /docker-java-home /docker-java-home COPY --from=jdk /usr/lib/jvm/ /usr/lib/jvm/ COPY --from=jdk /usr/share/java/ /usr/share/java/ COPY --from=jdk /usr/share/ca-certificates-java/ /usr/share/ca-certificates-java/ COPY --from=jdk /etc/java-11-openjdk/ /etc/java-11-openjdk/ COPY --from=jdk /usr/bin/java /usr/bin/java COPY --from=jdk /usr/bin/jps /usr/bin/jps COPY --from=jdk /usr/bin/jshell /usr/bin/jshell COPY --from=jdk /usr/bin/jcmd /usr/bin/jcmd COPY --from=jdk /usr/bin/jar /usr/bin/jar ENV JAVA_HOME /docker-java-home ENV JAVA_VERSION 11.0.1 ENV JAVA_DEBIAN_VERSION 11.0.1+13-3 COPY . /app

Composability FROM openjdk:11-jdk as jdk COPY . /app WORKDIR /app RUN ./mvnw clean install FROM ruby COPY --from=jdk /docker-java-home /docker-java-home COPY --from=jdk /usr/lib/jvm/ /usr/lib/jvm/ COPY --from=jdk /usr/share/java/ /usr/share/java/ COPY --from=jdk /usr/share/ca-certificates-java/ /usr/share/ca-certificates-java/ COPY --from=jdk /etc/java-11-openjdk/ /etc/java-11-openjdk/ COPY --from=jdk /usr/bin/java /usr/bin/java COPY --from=jdk /usr/bin/jps /usr/bin/jps COPY --from=jdk /usr/bin/jshell /usr/bin/jshell COPY --from=jdk /usr/bin/jcmd /usr/bin/jcmd COPY --from=jdk /usr/bin/jar /usr/bin/jar ENV JAVA_HOME /docker-java-home ENV JAVA_VERSION 11.0.1 ENV JAVA_DEBIAN_VERSION 11.0.1+13-3 COPY . /app COPY --from=java /app/target /app/target

Composability FROM openjdk:11-jdk as jdk COPY . /app WORKDIR /app RUN ./mvnw clean install FROM openjdk:11-jre as jre FROM ruby COPY --from=jre /docker-java-home /docker-java-home COPY --from=jre /usr/lib/jvm/ /usr/lib/jvm/ COPY --from=jre /usr/share/java/ /usr/share/java/ COPY --from=jre /usr/share/ca-certificates-java/ /usr/share/ca-certificates-java/ COPY --from=jre /etc/java-11-openjdk/ /etc/java-11-openjdk/ COPY --from=jre /usr/bin/java /usr/bin/java COPY --from=jre /usr/bin/jps /usr/bin/jps COPY --from=jre /usr/bin/jshell /usr/bin/jshell COPY --from=jre /usr/bin/jcmd /usr/bin/jcmd COPY --from=jre /usr/bin/jar /usr/bin/jar ENV JAVA_HOME /docker-java-home ENV JAVA_VERSION 11.0.1 ENV JAVA_DEBIAN_VERSION 11.0.1+13-3 COPY . /app COPY --from=jdk /app/target /app/target

Composability (Multi-stage Builds) ● No environment variables ● Doesn’t follow symlinks ● Only copying FS layers manually ○ Can’t copy arbitrary layers/files/dirs ■ COPY --from=0 /n1 /n1 ■ COPY --from=0 /n2 /n2 ■ COPY --from=0 /n3 /n3

Leaky Abstraction Poor tool for app developers who just want to write code. Authoring a good Dockerfile requires too much knowledge of the underlying mechanisms. Mix of operation and app developer concerns.

Dockerfile Alternatives?

No content

$ git push heroku master

Buildpacks: Heroku for Everything Opinionated, app aware, source-centric way to build your apps.

Buildpack Overview ● bin/detect ● bin/compile ● bin/release Slug Tarball Stack Image ABI Compatibility Guarantee

Ruby Buildpack ● Steps ○ installing Ruby ○ installing and running Bundler to manage gem dependencies ○ injecting database configuration ○ compiling Rails assets ● Comprehensive Support ○ 7 years of battle hardened usage ○ Used in production by millions of apps ○ supported MRI as old as 1.8.7 to 2.6.0-rc1 (on release day) ○ Rails 2.x-5.2 ○ Minimize buildpack upgrade pain/burden

Buildpack Ecosystem ● Languages ○ .NET Core ○ Elixir ○ R ● Frontend ○ create-react-app ○ Meteor ○ Jekyll ● Tools ○ NGINX ○ OpenCV ● Off the Shelf Software ○ Metabase ○ Spree ○ Minecraft

Buildpack Ecosystem (Providers)

Cloud Native Buildpacks (2018) Combine the power of buildpacks with the benefits of containers. Cloud Native Sandbox Project (CNCF) Incorporate learnings from Pivotal and Heroku. Let developers focus on their app and not piecing together a build pipeline.

Pack - Buildpack CLI application developers to use buildpacks to convert code into runnable images ● Building app images with build ● Updating app images using rebase

Build $ pack build

No content

How does it work? 1. Detect 2. Analyze 3. Build 4. Export

Detect

Analyze

Build

Export

Image Rebasing $ pack rebase

Day 2 Operations / Security Patching

Docker Image Manifest ● tarballs of layers ● configuration for container ○ Command ○ Entrypoint ○ Env Vars

Docker Image Manifest { "schemaVersion": 2, "mediaType": "application/vnd.docker.distribution.manifest.v2+json", "config": { "mediaType": "application/vnd.docker.container.image.v1+json", "size": 7023, "digest": "sha256:b5b2b2c507a0944348e0303114d8d93aaaa081732b86451d9bce1f432a537bc7" }, "layers": [ { "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", "size": 32654, "digest": "sha256:e692418e4cbaf90ca69d05a66403747baa33ee08806650b51fab815ad7fc331f" }, { "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", "size": 16724, "digest": "sha256:3c3a4604a545cdc127456d94e421cd355bca5b528f4a9c1905b15da2eb4a4c6b" }, { "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", "size": 73109, "digest": "sha256:ec4b8955958665577945c89419d1af06b5f7636b4ac3da7f12184802ad867736" } ] }

Docker Image Manifest { "schemaVersion": 2, "mediaType": "application/vnd.docker.distribution.manifest.v2+json", "config": { "mediaType": "application/vnd.docker.container.image.v1+json", "size": 7023, "digest": "sha256:b5b2b2c507a0944348e0303114d8d93aaaa081732b86451d9bce1f432a537bc7" }, "layers": [ { "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", "size": 32654, "digest": "sha256:e692418e4cbaf90ca69d05a66403747baa33ee08806650b51fab815ad7fc331f" }, { "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", "size": 16724, "digest": "sha256:3c3a4604a545cdc127456d94e421cd355bca5b528f4a9c1905b15da2eb4a4c6b" }, { "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", "size": 73109, "digest": "sha256:ec4b8955958665577945c89419d1af06b5f7636b4ac3da7f12184802ad867736" } ] }

Buildpack Image Building ● Day 2 Operations ● App Aware Image Builder ● Composability ● Higher Level Abstraction

Day 2 Operations ● Fast as a feature ● Stack image updates ● Unified build pipelines

App Aware Image Builder ● Intentional about docker layers ● Intelligent about caching ● Smart defaults: memory, concurrency, commands

Composability Buildpacks are decoupled and cohesive units

Higher Level Abstraction

meet developers where they are, their app source code

Try Buildpacks today! ● Github repo ○ Specification (WIP): https://github.com/buildpack/spec ○ Lifecycle: https://github.com/buildpack/lifecycle ○ Pack CLI: https://github.com/buildpack/pack ● Slack ○ https://slack.buildpacks.io/ ● Samples ○ https://github.com/buildpack/samples ○ https://github.com/heroku/java-buildpack ○ https://github.com/cloudfoundry/nodejs-cnb ○ https://github.com/cloudfoundry/npm-cnb ○ https://github.com/jkutner/python-buildpack

Deep Dive ● Thursday @ 2:35pm ● Tahoma 5 @ TCC Joe Kutner @codefiner Terence Lee @hone02