Intro: Cloud Native Buildpacks - KubeCon EU 2019

https://youtu.be/WTE8iu_vnbk

Transcript

2. Terence Lee @hone02
4. $ git push heroku master
5. Buildpacks: Heroku for Everything (2011) Opinionated, app aware, source-centric way to build your apps.
6. Buildpack Overview • bin/detect • bin/compile ◦ (CF) bin/supply ◦ (CF) bin/finalize • bin/release Slug Tarball Stack Image ABI Compatibility Guarantee
7. Ruby Buildpack Steps • installing Ruby • installing and running Bundler to manage gem dependencies • injecting database configuration • compiling Rails assets Comprehensive Support ◦ 7 years of battle hardened usage ◦ Used in production by millions of apps ◦ supported MRI as old as 1.8.7 to 2.6.3 (on release day) ◦ Rails 2.x-5.2 ◦ Minimize buildpack upgrade pain/burden
8. Buildpack Ecosystem (Buildpacks) • Languages ◦ .NET Core ◦ Elixir ◦ R • Frontend ◦ create-react-app ◦ Meteor ◦ Jekyll • Tools ◦ NGINX ◦ OpenCV • Off the Shelf Software ◦ Metabase ◦ Spree ◦ Minecraft
9. Buildpack Ecosystem (Providers)
11. “Writing a quality Dockerfile is still my users' biggest point of friction” - David Dollar, CEO, Convox
12. “Top ten most popular docker images each contain at least 30 vulnerabilities” “'Mystery meat' OpenJDK builds strike again" in *official* openjdk Docker images on Docker Hub
13. Leaky Abstraction FROM python:3 WORKDIR /usr/src/app COPY requirements.txt ./ RUN pip install --no-cache-dir -r requirements.txt COPY . . CMD [ "python", "./your-daemon-or-script.py" ]
14. Least Privileged User FROM python:3 WORKDIR /usr/src/app COPY requirements.txt ./ RUN pip install --no-cache-dir -r requirements.txt COPY . . RUN useradd pythonista USER pythonista CMD [ "python", "./your-daemon-or-script.py" ]
15. Reducing Image Size
16. Composability How do we combine two Docker images?
17. Composability FROM openjdk:11-jdk as jdk COPY . /app WORKDIR /app RUN ./mvnw clean install FROM ruby COPY --from=jdk /docker-java-home /docker-java-home COPY . /app
18. Composability FROM openjdk:11-jdk as jdk COPY . /app WORKDIR /app RUN ./mvnw clean install FROM ruby COPY --from=jdk /docker-java-home /docker-java-home COPY --from=jdk /usr/lib/jvm/ /usr/lib/jvm/ COPY --from=jdk /usr/share/java/ /usr/share/java/ COPY --from=jdk /usr/share/ca-certificates-java/ /usr/share/ca-certificates-java/ COPY --from=jdk /etc/java-11-openjdk/ /etc/java-11-openjdk/ COPY --from=jdk /usr/bin/java /usr/bin/java COPY --from=jdk /usr/bin/jps /usr/bin/jps COPY --from=jdk /usr/bin/jshell /usr/bin/jshell COPY --from=jdk /usr/bin/jcmd /usr/bin/jcmd COPY --from=jdk /usr/bin/jar /usr/bin/jar ENV JAVA_HOME /docker-java-home ENV JAVA_VERSION 11.0.1 ENV JAVA_DEBIAN_VERSION 11.0.1+13-3 COPY . /app
19. Composability FROM openjdk:11-jdk as jdk COPY . /app WORKDIR /app RUN ./mvnw clean install FROM ruby COPY --from=jdk /docker-java-home /docker-java-home COPY --from=jdk /usr/lib/jvm/ /usr/lib/jvm/ COPY --from=jdk /usr/share/java/ /usr/share/java/ COPY --from=jdk /usr/share/ca-certificates-java/ /usr/share/ca-certificates-java/ COPY --from=jdk /etc/java-11-openjdk/ /etc/java-11-openjdk/ COPY --from=jdk /usr/bin/java /usr/bin/java COPY --from=jdk /usr/bin/jps /usr/bin/jps COPY --from=jdk /usr/bin/jshell /usr/bin/jshell COPY --from=jdk /usr/bin/jcmd /usr/bin/jcmd COPY --from=jdk /usr/bin/jar /usr/bin/jar ENV JAVA_HOME /docker-java-home ENV JAVA_VERSION 11.0.1 ENV JAVA_DEBIAN_VERSION 11.0.1+13-3 COPY . /app COPY --from=java /app/target /app/target
20. Composability FROM openjdk:11-jdk as jdk COPY . /app WORKDIR /app RUN ./mvnw clean install FROM openjdk:11-jre as jre FROM ruby COPY --from=jre /docker-java-home /docker-java-home COPY --from=jre /usr/lib/jvm/ /usr/lib/jvm/ COPY --from=jre /usr/share/java/ /usr/share/java/ COPY --from=jre /usr/share/ca-certificates-java/ /usr/share/ca-certificates-java/ COPY --from=jre /etc/java-11-openjdk/ /etc/java-11-openjdk/ COPY --from=jre /usr/bin/java /usr/bin/java COPY --from=jre /usr/bin/jps /usr/bin/jps COPY --from=jre /usr/bin/jshell /usr/bin/jshell COPY --from=jre /usr/bin/jcmd /usr/bin/jcmd COPY --from=jre /usr/bin/jar /usr/bin/jar ENV JAVA_HOME /docker-java-home ENV JAVA_VERSION 11.0.1 ENV JAVA_DEBIAN_VERSION 11.0.1+13-3 COPY . /app COPY --from=jdk /app/target /app/target
21. Composability (Multi-stage Builds) • No environment variables • Doesn’t follow symlinks • Limited interface for copying ◦ No support for globs, often need many copy statements ▪ COPY --from=0 /n1 /n1 ▪ COPY --from=0 /n2 /n2 ▪ COPY --from=0 /n3 /n3
22. Don’t leak sensitive information to Docker images FROM ubuntu as intermediate WORKDIR /app COPY secret/key /tmp/ RUN scp -i /tmp/key build@acme/files . FROM ubuntu WORKDIR /app COPY --from=intermediate /app .
23. meet developers where they are: their app source code
25. Day-2
50. Optimized Builds – How it Works • Only re-builds and uploads layers when necessary • OCI image specification: content addressable layers • Docker Registry v2: cross repository blob mounting Result: Fast builds, minimal data transfer, layer “rebasing” directly on the registry!
51. New Buildpack API bin/detect bin/supply bin/finalize bin/build bin/detect Old Buildpack Interface New Buildpack Interface plan TOML bin/release
52. New Buildpack API Build Detect Analysis Export where metadata about OCI layers generated during a previous build are made available to buildpacks where the remote layers are replaced by the generated layers where an optimal selection of compatible buildpacks is chosen and a build plan is created where buildpacks use that metadata to generate only the OCI layers that need to be replaced
53. bin/detect ruby_buildpack Ruby + Node.js App Gemfile package.json app/ ✓ plan.toml bin/detect nodejs_buildpack ✓ bin/build bin/build [ruby] version = "2.5.1" [ruby.metadata] launch = true [node] version = "8.12.0" [node.metadata] launch = true Node.js layer Ruby layer Node modules layer Multiple Buildpack Support Buildpack Group Gems layer
54. Build Steps – New API Build Install ruby bundle install Install node npm install OCI Image Ruby + Node.js App First Build Analysis First build, nothing to do Export Create nodejs layer Create node_modules layer Create app layer Create mri layer Create gem layer Create OS layer mri layer modules layer app layer configuration layer gems layer nodejs layer ubuntu:18.04
55. Build (w/ cache + metadata) Read metadata from disk bundle install (with cached gems) npm install (with cached modules) Ruby + Node.js App node modules updated gems updated Second Build Analysis Read metadata about layers Write metadata to disk for build Export Update app layer Update gems layer Update modules layer OCI Image mri layer modules layer app layer configuration layer gems layer nodejs layer ubuntu:18.04 app layer modules layer gems layer
56. • ◦ ◦ ◦ • ◦ • ◦