Building a Docker Image Packaging Pipeline Using GitHub Actions

Building a Docker Image Packaging Pipeline Using GitHub Actions

DockerCon 2020 talk on creating reusable Dockerfiles, and building a packaging pipeline using GitHub Actions.

98234c645fe8c935edc0fec0186d28b8?s=128

Gareth Rushgrove

May 28, 2020
Tweet

Transcript

  1. Building a Docker Image Packaging Pipeline Using GitHub Actions Gareth

    Rushgrove
  2. Gareth Rushgrove Director, Product Management, Snyk Devops Weekly curator Open

    Source contributor @garethr
  3. Agenda Reusable Dockerfiles 01 Hello build-push-action 02 A real world

    example 03 Conclusions 04
  4. Reusable Dockerfiles Avoiding the copy and paste trap

  5. Dockerfile The defacto standard for building container images Dockerfile

  6. What does Dockerfile do? Create a filesystem, add metadata #

    Use the official image as a parent image. FROM node:current-slim # Set the working directory. WORKDIR /usr/src/app # Copy the file from your host to your current location. COPY package.json . # Run the command inside your image filesystem. RUN npm install # Inform Docker that the container is listening on the specified port at runtime. EXPOSE 8080 # Run the specified command within the container. CMD [ "npm", "start" ] # Copy the rest of your app's source code from your host to your image filesystem. COPY . .
  7. In the beginning We have one Dockerfile building one image

    Dockerfile
  8. Copy and paste Everyone’s first approach to reuse Dockerfile Dockerfile

    Dockerfile Dockerfile Dockerfile Dockerfile
  9. Time passes... Something has gone terrible wrong Dockerfile Dockerfile Dockerfile

    Dockerfile Dockerfile Dockerfile Dockerfile Dockerfile Dockerfile Dockerfile Dockerfile Dockerfile Dockerfile Dockerfile Dockerfile Dockerfile Dockerfile Dockerfile Dockerfile Dockerfile Dockerfile Dockerfile Dockerfile Dockerfile
  10. What can we do? Let’s start with a very simple

    Dockerfile FROM alpine COPY --from=open-policy-agent/conftest:v0.18.2 /conftest /conftest
  11. Docker build args Build-time variables with default values FROM alpine

    COPY --from=open-policy-agent/conftest:v0.18.2 /conftest /conftest FROM alpine ARG VERSION=v0.18.2 COPY --from=open-policy-agent/conftest:$VERSION /conftest /conftest
  12. Pass values at build time Can pass from local ENV

    as well FROM alpine COPY --from=open-policy-agent/conftest:v0.18.2 /conftest /conftest FROM alpine ARG VERSION=v0.18.2 COPY --from=open-policy-agent/conftest:$VERSION /conftest /conftest $ docker build --build-arg VERSION=v0.19.0
  13. Just keep adding ARGs! Possible to build very generic Dockerfiles

    FROM alpine ARG VERSION=v0.18.2 COPY --from=open-policy-agent/conftest:$VERSION /conftest /conftest FROM alpine ARG LABEL=v0.18.2 ARG IMAGE=open-policy-agent/conftest ARG PATH=/conftest COPY --from=$IMAGE:$LABEL $PATH $PATH
  14. One Dockerfile, many images Reduced duplication and maintenance overhead Dockerfile

  15. Lots of other Dockerfile patterns Aliases, inheritance, logic and more

  16. Introducing build-push-action Docker’s new official GitHub Action

  17. GitHub Actions Run a workflow on any GitHub event

  18. Define workflows in YAML Stored in your repository at .github/workflows

    name: .NET Core on: push jobs: build: runs-on: ubuntu-latest steps: - uses: actions/checkout@v2 - name: Setup .NET Core uses: actions/setup-dotnet@v1 with: dotnet-version: 3.1.101 - name: Install dependencies run: dotnet restore - name: Build run: dotnet build --configuration Release --no-restore - name: Test run: dotnet test --no-restore --verbosity normal
  19. GitHub Actions uses Docker Standard docker commands just work name:

    Docker on: push: branches: [ master ] pull_request: branches: [ master ] jobs: build: runs-on: ubuntu-latest steps: - uses: actions/checkout@v2 - name: Build the Docker image run: docker build . --file Dockerfile --tag my-image-name:$(date +%s)
  20. build-push-action github.com/docker/build-push-action

  21. build-push-action basics Build and push an image to Docker Hub

    uses: docker/build-push-action@v1 with: username: ${{ secrets.DOCKER_USERNAME }} password: ${{ secrets.DOCKER_PASSWORD }} repository: myorg/myrepository tags: latest
  22. GitHub Actions supports logical operations Push only on tags, rather

    than any commit uses: docker/build-push-action@v1 with: username: ${{ secrets.DOCKER_USERNAME }} password: ${{ secrets.DOCKER_PASSWORD }} repository: myorg/myrepository tag_with_ref: true push: ${{ startsWith(github.ref, 'refs/tags/') }}
  23. Lots of built-in goodies Lower the barrier to entry for

    good practice uses: docker/build-push-action@v1 with: username: ${{ secrets.DOCKER_USERNAME }} password: ${{ secrets.DOCKER_PASSWORD }} repository: myorg/myrepository tag_with_ref: true tag_with_sha: true add_git_labels: true always_pull: true labels: label_name_1=label_value_1 target: mytarget
  24. Let’s bring everything together Real world example time

  25. Snyk Docker images github.com/snyk/snyk-images

  26. Find security vulnerabilities in your applications Note the :golang tag

    on snyk/snyk $ git clone git@github.com:puppetlabs/wash.git $ docker run --rm -it --env SNYK_TOKEN -v $(PWD):/app snyk/snyk:golang Testing /app... Organization: garethr Package manager: gomodules Target file: go.mod Open source: no Project path: /app Licenses: enabled ✓ Tested 426 dependencies for known issues, no vulnerable paths found. Next steps: - Run `snyk monitor` to be notified about new related vulnerabilities. - Run `snyk test` as part of your CI/test.
  27. Snyk supports lots of languages And frameworks and versions

  28. Currently 49 images For different development environments

  29. 49 GitHub Actions jobs Build and push in parallel in

    <5 minutes
  30. Runs on push and on a regular schedule Nice way

    of making sure we rebuild regularly name: Build and push images on: push: branches: - master paths: - "*" - "!README.md" - "!build.rb" schedule: # As well as running when we make changes we should run at least # every week in order to pick up new parent images and new versions of Snyk - cron: "0 0 * * 0"
  31. Runs on push and on a regular schedule Nice way

    of making sure we rebuild regularly name: Build and push images on: push: branches: - master paths: - "*" - "!README.md" - "!build.rb" schedule: # As well as running when we make changes we should run at least # every week in order to pick up new parent images and new versions of Snyk - cron: "0 0 * * 0" Don’t rebuild on documentation changes
  32. 1 Dockerfile Take an image as an argument and set

    some defaults ARG IMAGE FROM ${IMAGE} as parent WORKDIR /app COPY docker-entrypoint.sh /usr/local/bin/ ENTRYPOINT ["/usr/local/bin/docker-entrypoint.sh"] CMD ["snyk", "test"] FROM ubuntu as snyk RUN apt-get update && apt-get install -y curl wget RUN curl -s https://api.github.com/repos/snyk/snyk/releases/latest | grep "browser_download_url" | grep linux | cut -d '"' -f 4 | wget -i - && \ sha256sum -c snyk-linux.sha256 && \ mv snyk-linux /usr/local/bin/snyk && \ chmod +x /usr/local/bin/snyk
  33. 1 Dockerfile Download precompiled Snyk for each platform FROM ubuntu

    as snyk RUN apt-get update && apt-get install -y curl wget RUN curl -s https://api.github.com/repos/snyk/snyk/releases/latest | grep "browser_download_url" | grep linux | cut -d '"' -f 4 | wget -i - && \ sha256sum -c snyk-linux.sha256 && \ mv snyk-linux /usr/local/bin/snyk && \ chmod +x /usr/local/bin/snyk FROM alpine as snyk-alpine RUN apk add --no-cache curl wget RUN curl -s https://api.github.com/repos/snyk/snyk/releases/latest | grep "browser_download_url" | grep alpine | cut -d '"' -f 4 | wget -i - && \ sha256sum -c snyk-alpine.sha256 && \ mv snyk-alpine /usr/local/bin/snyk && \
  34. 1 Dockerfile Output targets for each platform FROM parent as

    alpine RUN apk add --no-cache libstdc++ COPY --from=snyk-alpine /usr/local/bin/snyk /usr/local/bin/snyk FROM parent as linux COPY --from=snyk /usr/local/bin/snyk /usr/local/bin/snyk
  35. 1 Action generating all the other Actions Did I mention

    I like metaprogramming?
  36. Runs on push Run only when build files change name:

    "Generate Actions to build Snyk images" on: push: branches: - master paths: - build.rb - linux - alpine
  37. Enter the matrix Generated matrix for each combination matrix: include:

    - base: clojure:boot tag: clojure-boot target: linux - base: clojure:lein tag: clojure-lein target: linux - base: clojure:tools-deps tag: clojure-tools-deps target: linux - base: golang tag: golang target: linux - base: golang:1.12
  38. build-push-image With the inputs from the matrix steps: - uses:

    actions/checkout@v1 - uses: docker/build-push-action@v1 env: DOCKER_BUILDKIT: "1" with: username: ${{ secrets.DOCKER_USERNAME }} password: ${{ secrets.DOCKER_PASSWORD }} add_git_labels: true target: ${{ matrix.target }} repository: snyk/snyk tags: ${{ matrix.tag }} build_args: IMAGE=${{ matrix.base }}
  39. Example OCI labels Great for understanding provenance $ docker inspect

    snyk/snyk:python --format="{{json .Config.Labels}}" | jq { "org.opencontainers.image.created": "2020-05-17T00:14:15Z", "org.opencontainers.image.revision": "fa19b4d7fda74be0f342926c3c7feeb368f88b17", "org.opencontainers.image.source": "https://github.com/snyk/snyk-images" }
  40. Demo

  41. Conclusions If all you remember is...

  42. Learn Dockerfile Dockerfile is incredible simple to get started with,

    but hides some powerful features that not enough folks learn to use
  43. Bring automation closer to developers Identify programmatic solutions to repetitive

    problems
  44. Focus on maintenance The value of reducing the overhead of

    ongoing work needed to do the job is often underestimated
  45. Thanks for listening Sign up for free at snyk.io/signup