KubeCon 2018 - Intro: Cloud Native Buildpacks

Presented by Joe Kutner and myself.
Video here: https://videos.videoken.com/index.php/videos/intro-cloud-native-buildpacks-terence-lee-joe-kutner-salesforce-heroku/

You're great at running containers but you shouldn't have to be great at building them. In this talk, you'll learn about Cloud Native Buildpacks, a higher-level abstraction for building apps compared to Dockerfiles.

Buildpacks are a standardized tool for creating images in a secure, reproducible, and efficient manner. As an app developer, you don't need to know best practices around ordering commands for layer reuse. As an operator, you don't need to worry about exposing developers to the responsibilities that come with Dockerfile.

Come learn how buildpacks meet developers at their source code, automate the delivery of both OS-level and application-level dependency upgrades, and help you efficiently handle day-2 app operations.

Transcript

1. Cloud Native Buildpacks Intro
2. Joe Kutner @codefinger
3. Terence Lee @hone02
5. k8s is a great platform to run images
6. What's the best way to build images?
8. Containerize a Ruby App
9. Dockerfile (Ruby) FROM ruby COPY . /app WORKDIR /app RUN bundle install EXPOSE 5000 CMD bundle exec ruby ./app.rb
10. Building Docker Images $ docker build .
11. Building Docker Images $ docker build . Step 1/6 : FROM ruby ---> 88666731c3e1 Step 2/6 : COPY . /app ---> 173624d82900 Step 3/6 : WORKDIR /app ---> Running in 0649f4408d91 ---> 850a46b3ec29 Step 4/6 : RUN bundle install ---> Running in ed644f258949 ---> 499852b8c318 Step 5/6 : EXPOSE 5000 ---> Running in fc0958926d74 ---> 4d4fbc35cde5 Step 6/6 : CMD bundle exec ruby ./app.rb ---> Running in 0e901e0910db ---> 881870f15126 Successfully built 881870f15126
12. Containerize Rails?
13. Dockerfile (Rails) FROM ruby COPY . /app WORKDIR /app RUN bundle install EXPOSE 5000 CMD bin/rails
14. Dockerfile (Rails) FROM ruby RUN apt-get update -qq \ && apt-get install -y nodejs libpq-dev build-essential COPY . /app WORKDIR /app RUN bundle install RUN bundle exec rake assets:precompile EXPOSE 5000 CMD bin/rails
15. Build image best practices? • Reduce image size • Speed up incremental builds • Pick base image
16. Dockerfile Layers Step 1/6 : FROM ruby ---> 88666731c3e1 Step 2/6 : COPY . /app ---> 173624d82900 Step 3/6 : WORKDIR /app ---> Running in 0649f4408d91 ---> 850a46b3ec29 Step 4/6 : RUN bundle install ---> Running in ed644f258949 ---> 499852b8c318 Step 5/6 : EXPOSE 5000 ---> Running in fc0958926d74 ---> 4d4fbc35cde5 Step 6/6 : CMD bundle exec ruby ./app.rb ---> Running in 0e901e0910db ---> 881870f15126 Successfully built 881870f15126 FROM ruby:latest COPY . /app WORKDIR /app RUN bundle install EXPOSE 5000 CMD bundle exec ruby ./app.rb
17. Reduce image size FROM ruby RUN apt-get update -qq \ && apt-get install -y nodejs libpq-dev build-essential \ && apt-get clean autoclean && apt-get autoremove -y \ && rm -rf /var/lib/apt /var/lib/dpkg /var/lib/cache /var/lib/log COPY . /app WORKDIR /app RUN bundle install RUN bundle exec rake assets:precompile \ && rm -rf /app/tmp/cache/assets/ EXPOSE 5000 CMD bin/rails
18. Speed Up Incremental Builds FROM ruby RUN apt-get update -qq \ && apt-get install -y nodejs libpq-dev build-essential \ && apt-get clean autoclean && apt-get autoremove -y \ && rm -rf /var/lib/apt /var/lib/dpkg /var/lib/cache /var/lib/log ADD Gemfile /app/ ADD Gemfile.lock /app/ RUN bundle install COPY . /app WORKDIR /app RUN bundle exec rake assets:precompile \ && rm -rf /app/tmp/cache/assets/ EXPOSE 5000 CMD bin/rails
19. Pick Base Image FROM ruby RUN apt-get update -qq \ && apt-get install -y nodejs libpq-dev build-essential COPY . /app WORKDIR /app RUN bundle install RUN bundle exec rake assets:precompile EXPOSE 5000 CMD bin/rails
20. hub.docker.com/_/ruby/
21. “Writing a quality Dockerfile is still my users' biggest point of friction” - David Dollar, CEO, Convox
22. Dockerfile Shortcomings • Maintenance / Day 2 Operations (Security) • Not App Aware • Composability • Leaky Abstraction
23. Maintenance / Day 2 Operations FROM ruby:latest RUN mkdir /usr/src/app ADD . /usr/src/app/ WORKDIR /usr/src/app/ RUN bundle install CMD ["/usr/src/app/main.rb"]
24. Composability How do we combine two Docker images?
25. Composability FROM openjdk:11-jdk as jdk COPY . /app WORKDIR /app RUN ./mvnw clean install FROM ruby COPY --from=jdk /docker-java-home /docker-java-home COPY . /app
26. Composability FROM openjdk:11-jdk as jdk COPY . /app WORKDIR /app RUN ./mvnw clean install FROM ruby COPY --from=jdk /docker-java-home /docker-java-home COPY --from=jdk /usr/lib/jvm/ /usr/lib/jvm/ COPY --from=jdk /usr/share/java/ /usr/share/java/ COPY --from=jdk /usr/share/ca-certificates-java/ /usr/share/ca-certificates-java/ COPY --from=jdk /etc/java-11-openjdk/ /etc/java-11-openjdk/ COPY --from=jdk /usr/bin/java /usr/bin/java COPY --from=jdk /usr/bin/jps /usr/bin/jps COPY --from=jdk /usr/bin/jshell /usr/bin/jshell COPY --from=jdk /usr/bin/jcmd /usr/bin/jcmd COPY --from=jdk /usr/bin/jar /usr/bin/jar ENV JAVA_HOME /docker-java-home ENV JAVA_VERSION 11.0.1 ENV JAVA_DEBIAN_VERSION 11.0.1+13-3 COPY . /app
27. Composability FROM openjdk:11-jdk as jdk COPY . /app WORKDIR /app RUN ./mvnw clean install FROM ruby COPY --from=jdk /docker-java-home /docker-java-home COPY --from=jdk /usr/lib/jvm/ /usr/lib/jvm/ COPY --from=jdk /usr/share/java/ /usr/share/java/ COPY --from=jdk /usr/share/ca-certificates-java/ /usr/share/ca-certificates-java/ COPY --from=jdk /etc/java-11-openjdk/ /etc/java-11-openjdk/ COPY --from=jdk /usr/bin/java /usr/bin/java COPY --from=jdk /usr/bin/jps /usr/bin/jps COPY --from=jdk /usr/bin/jshell /usr/bin/jshell COPY --from=jdk /usr/bin/jcmd /usr/bin/jcmd COPY --from=jdk /usr/bin/jar /usr/bin/jar ENV JAVA_HOME /docker-java-home ENV JAVA_VERSION 11.0.1 ENV JAVA_DEBIAN_VERSION 11.0.1+13-3 COPY . /app COPY --from=java /app/target /app/target
28. Composability FROM openjdk:11-jdk as jdk COPY . /app WORKDIR /app RUN ./mvnw clean install FROM openjdk:11-jre as jre FROM ruby COPY --from=jre /docker-java-home /docker-java-home COPY --from=jre /usr/lib/jvm/ /usr/lib/jvm/ COPY --from=jre /usr/share/java/ /usr/share/java/ COPY --from=jre /usr/share/ca-certificates-java/ /usr/share/ca-certificates-java/ COPY --from=jre /etc/java-11-openjdk/ /etc/java-11-openjdk/ COPY --from=jre /usr/bin/java /usr/bin/java COPY --from=jre /usr/bin/jps /usr/bin/jps COPY --from=jre /usr/bin/jshell /usr/bin/jshell COPY --from=jre /usr/bin/jcmd /usr/bin/jcmd COPY --from=jre /usr/bin/jar /usr/bin/jar ENV JAVA_HOME /docker-java-home ENV JAVA_VERSION 11.0.1 ENV JAVA_DEBIAN_VERSION 11.0.1+13-3 COPY . /app COPY --from=jdk /app/target /app/target
29. Composability (Multi-stage Builds) • No environment variables • Doesn’t follow symlinks • Only copying FS layers manually ◦ Can’t copy arbitrary layers/files/dirs ▪ COPY --from=0 /n1 /n1 ▪ COPY --from=0 /n2 /n2 ▪ COPY --from=0 /n3 /n3
30. Leaky Abstraction Poor tool for app developers who just want to write code. Authoring a good Dockerfile requires too much knowledge of the underlying mechanisms. Mix of operation and app developer concerns.
31. Dockerfile Alternatives?
33. $ git push heroku master
34. Buildpacks: Heroku for Everything Opinionated, app aware, source-centric way to build your apps.
35. Buildpack Overview • bin/detect • bin/compile • bin/release Slug Tarball Stack Image ABI Compatibility Guarantee
36. Ruby Buildpack • Steps ◦ installing Ruby ◦ installing and running Bundler to manage gem dependencies ◦ injecting database configuration ◦ compiling Rails assets • Comprehensive Support ◦ 7 years of battle hardened usage ◦ Used in production by millions of apps ◦ supported MRI as old as 1.8.7 to 2.6.0-rc1 (on release day) ◦ Rails 2.x-5.2 ◦ Minimize buildpack upgrade pain/burden
37. Buildpack Ecosystem • Languages ◦ .NET Core ◦ Elixir ◦ R • Frontend ◦ create-react-app ◦ Meteor ◦ Jekyll • Tools ◦ NGINX ◦ OpenCV • Off the Shelf Software ◦ Metabase ◦ Spree ◦ Minecraft
38. Buildpack Ecosystem (Providers)
39. Cloud Native Buildpacks (2018) Combine the power of buildpacks with the benefits of containers. Cloud Native Sandbox Project (CNCF) Incorporate learnings from Pivotal and Heroku. Let developers focus on their app and not piecing together a build pipeline.
40. Pack - Buildpack CLI application developers to use buildpacks to convert code into runnable images • Building app images with build • Updating app images using rebase
41. Build $ pack build <image name>
43. How does it work? 1. Detect 2. Analyze 3. Build 4. Export
44. Detect
45. Analyze
46. Build
47. Export
48. Image Rebasing $ pack rebase <image name>
49. Day 2 Operations / Security Patching
50. Docker Image Manifest • tarballs of layers • configuration for container ◦ Command ◦ Entrypoint ◦ Env Vars
51. Docker Image Manifest { "schemaVersion": 2, "mediaType": "application/vnd.docker.distribution.manifest.v2+json", "config": { "mediaType": "application/vnd.docker.container.image.v1+json", "size": 7023, "digest": "sha256:b5b2b2c507a0944348e0303114d8d93aaaa081732b86451d9bce1f432a537bc7" }, "layers": [ { "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", "size": 32654, "digest": "sha256:e692418e4cbaf90ca69d05a66403747baa33ee08806650b51fab815ad7fc331f" }, { "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", "size": 16724, "digest": "sha256:3c3a4604a545cdc127456d94e421cd355bca5b528f4a9c1905b15da2eb4a4c6b" }, { "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", "size": 73109, "digest": "sha256:ec4b8955958665577945c89419d1af06b5f7636b4ac3da7f12184802ad867736" } ] }
52. Docker Image Manifest { "schemaVersion": 2, "mediaType": "application/vnd.docker.distribution.manifest.v2+json", "config": { "mediaType": "application/vnd.docker.container.image.v1+json", "size": 7023, "digest": "sha256:b5b2b2c507a0944348e0303114d8d93aaaa081732b86451d9bce1f432a537bc7" }, "layers": [ { "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", "size": 32654, "digest": "sha256:e692418e4cbaf90ca69d05a66403747baa33ee08806650b51fab815ad7fc331f" }, { "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", "size": 16724, "digest": "sha256:3c3a4604a545cdc127456d94e421cd355bca5b528f4a9c1905b15da2eb4a4c6b" }, { "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", "size": 73109, "digest": "sha256:ec4b8955958665577945c89419d1af06b5f7636b4ac3da7f12184802ad867736" } ] }
53. Buildpack Image Building • Day 2 Operations • App Aware Image Builder • Composability • Higher Level Abstraction
54. Day 2 Operations • Fast as a feature • Stack image updates • Unified build pipelines
55. App Aware Image Builder • Intentional about docker layers • Intelligent about caching • Smart defaults: memory, concurrency, commands
56. Composability Buildpacks are decoupled and cohesive units
57. Higher Level Abstraction
58. meet developers where they are, their app source code
59. Try Buildpacks today! • Github repo ◦ Specification (WIP): https://github.com/buildpack/spec ◦ Lifecycle: https://github.com/buildpack/lifecycle ◦ Pack CLI: https://github.com/buildpack/pack • Slack ◦ https://slack.buildpacks.io/ • Samples ◦ https://github.com/buildpack/samples ◦ https://github.com/heroku/java-buildpack ◦ https://github.com/cloudfoundry/nodejs-cnb ◦ https://github.com/cloudfoundry/npm-cnb ◦ https://github.com/jkutner/python-buildpack
60. Deep Dive • Thursday @ 2:35pm • Tahoma 5 @ TCC Joe Kutner @codefiner Terence Lee @hone02