Building a Docker Image Packaging Pipeline Using GitHub Actions

DockerCon 2020 talk on creating reusable Dockerfiles, and building a packaging pipeline using GitHub Actions.

Transcript

1. Building a Docker Image Packaging
   Pipeline Using GitHub Actions
   Gareth Rushgrove
   

   View Slide

2. Gareth Rushgrove
   Director, Product Management, Snyk
   Devops Weekly curator
   Open Source contributor
   @garethr
   

   View Slide

3. Agenda Reusable Dockerfiles
   01
   Hello build-push-action
   02
   A real world example
   03
   Conclusions
   04
   

   View Slide

4. Reusable Dockerfiles
   Avoiding the copy and paste trap
   

   View Slide

5. Dockerfile
   The defacto standard for building container images
   Dockerfile
   

   View Slide

6. What does Dockerfile do?
   Create a filesystem, add metadata
   # Use the official image as a parent image.
   FROM node:current-slim
   # Set the working directory.
   WORKDIR /usr/src/app
   # Copy the file from your host to your current location.
   COPY package.json .
   # Run the command inside your image filesystem.
   RUN npm install
   # Inform Docker that the container is listening on the specified port at runtime.
   EXPOSE 8080
   # Run the specified command within the container.
   CMD [ "npm", "start" ]
   # Copy the rest of your app's source code from your host to your image filesystem.
   COPY . .
   

   View Slide

7. In the beginning
   We have one Dockerfile building one image
   Dockerfile
   

   View Slide

8. Copy and paste
   Everyone’s first approach to reuse
   Dockerfile
   Dockerfile
   Dockerfile
   Dockerfile
   Dockerfile
   Dockerfile
   

   View Slide

9. Time passes...
   Something has gone terrible wrong
   Dockerfile
   Dockerfile
   Dockerfile
   Dockerfile
   Dockerfile
   Dockerfile
   Dockerfile
   Dockerfile
   Dockerfile
   Dockerfile
   Dockerfile
   Dockerfile
   Dockerfile
   Dockerfile
   Dockerfile
   Dockerfile
   Dockerfile
   Dockerfile
   Dockerfile
   Dockerfile
   Dockerfile
   Dockerfile
   Dockerfile
   Dockerfile
   

   View Slide

10. What can we do?
    Let’s start with a very simple Dockerfile
    FROM alpine
    COPY --from=open-policy-agent/conftest:v0.18.2 /conftest /conftest
    

    View Slide

11. Docker build args
    Build-time variables with default values
    FROM alpine
    COPY --from=open-policy-agent/conftest:v0.18.2 /conftest /conftest
    FROM alpine
    ARG VERSION=v0.18.2
    COPY --from=open-policy-agent/conftest:$VERSION /conftest /conftest
    

    View Slide

12. Pass values at build time
    Can pass from local ENV as well
    FROM alpine
    COPY --from=open-policy-agent/conftest:v0.18.2 /conftest /conftest
    FROM alpine
    ARG VERSION=v0.18.2
    COPY --from=open-policy-agent/conftest:$VERSION /conftest /conftest
    $ docker build --build-arg VERSION=v0.19.0
    

    View Slide

13. Just keep adding ARGs!
    Possible to build very generic Dockerfiles
    FROM alpine
    ARG VERSION=v0.18.2
    COPY --from=open-policy-agent/conftest:$VERSION /conftest /conftest
    FROM alpine
    ARG LABEL=v0.18.2
    ARG IMAGE=open-policy-agent/conftest
    ARG PATH=/conftest
    COPY --from=$IMAGE:$LABEL $PATH $PATH
    

    View Slide

14. One Dockerfile, many images
    Reduced duplication and maintenance overhead
    Dockerfile
    

    View Slide

15. Lots of other Dockerfile patterns
    Aliases, inheritance, logic and more
    

    View Slide

16. Introducing build-push-action
    Docker’s new official GitHub Action
    

    View Slide

17. GitHub Actions
    Run a workflow on any GitHub event
    

    View Slide

18. Define workflows in YAML
    Stored in your repository at .github/workflows
    name: .NET Core
    on: push
    jobs:
    build:
    runs-on: ubuntu-latest
    steps:
    - uses: actions/checkout@v2
    - name: Setup .NET Core
    uses: actions/setup-dotnet@v1
    with:
    dotnet-version: 3.1.101
    - name: Install dependencies
    run: dotnet restore
    - name: Build
    run: dotnet build --configuration Release --no-restore
    - name: Test
    run: dotnet test --no-restore --verbosity normal
    

    View Slide

19. GitHub Actions uses Docker
    Standard docker commands just work
    name: Docker
    on:
    push:
    branches: [ master ]
    pull_request:
    branches: [ master ]
    jobs:
    build:
    runs-on: ubuntu-latest
    steps:
    - uses: actions/checkout@v2
    - name: Build the Docker image
    run: docker build . --file Dockerfile --tag my-image-name:$(date +%s)
    

    View Slide

20. build-push-action
    github.com/docker/build-push-action
    

    View Slide

21. build-push-action basics
    Build and push an image to Docker Hub
    uses: docker/build-push-action@v1
    with:
    username: ${{ secrets.DOCKER_USERNAME }}
    password: ${{ secrets.DOCKER_PASSWORD }}
    repository: myorg/myrepository
    tags: latest
    

    View Slide

22. GitHub Actions supports logical operations
    Push only on tags, rather than any commit
    uses: docker/build-push-action@v1
    with:
    username: ${{ secrets.DOCKER_USERNAME }}
    password: ${{ secrets.DOCKER_PASSWORD }}
    repository: myorg/myrepository
    tag_with_ref: true
    push: ${{ startsWith(github.ref, 'refs/tags/') }}
    

    View Slide

23. Lots of built-in goodies
    Lower the barrier to entry for good practice
    uses: docker/build-push-action@v1
    with:
    username: ${{ secrets.DOCKER_USERNAME }}
    password: ${{ secrets.DOCKER_PASSWORD }}
    repository: myorg/myrepository
    tag_with_ref: true
    tag_with_sha: true
    add_git_labels: true
    always_pull: true
    labels: label_name_1=label_value_1
    target: mytarget
    

    View Slide

24. Let’s bring everything together
    Real world example time
    

    View Slide

25. Snyk Docker images
    github.com/snyk/snyk-images
    

    View Slide

26. Find security vulnerabilities in your applications
    Note the :golang tag on snyk/snyk
    $ git clone git@github.com:puppetlabs/wash.git
    $ docker run --rm -it --env SNYK_TOKEN -v $(PWD):/app snyk/snyk:golang
    Testing /app...
    Organization: garethr
    Package manager: gomodules
    Target file: go.mod
    Open source: no
    Project path: /app
    Licenses: enabled
    ✓ Tested 426 dependencies for known issues, no vulnerable paths found.
    Next steps:
    - Run `snyk monitor` to be notified about new related vulnerabilities.
    - Run `snyk test` as part of your CI/test.
    

    View Slide

27. Snyk supports lots of languages
    And frameworks and versions
    

    View Slide

28. Currently 49 images
    For different development environments
    

    View Slide

29. 49 GitHub Actions jobs
    Build and push in parallel in <5 minutes
    

    View Slide

30. Runs on push and on a regular schedule
    Nice way of making sure we rebuild regularly
    name: Build and push images
    on:
    push:
    branches:
    - master
    paths:
    - "*"
    - "!README.md"
    - "!build.rb"
    schedule:
    # As well as running when we make changes we should run at least
    # every week in order to pick up new parent images and new versions of Snyk
    - cron: "0 0 * * 0"
    

    View Slide

31. Runs on push and on a regular schedule
    Nice way of making sure we rebuild regularly
    name: Build and push images
    on:
    push:
    branches:
    - master
    paths:
    - "*"
    - "!README.md"
    - "!build.rb"
    schedule:
    # As well as running when we make changes we should run at least
    # every week in order to pick up new parent images and new versions of Snyk
    - cron: "0 0 * * 0"
    Don’t rebuild on
    documentation changes
    

    View Slide

32. 1 Dockerfile
    Take an image as an argument and set some defaults
    ARG IMAGE
    FROM ${IMAGE} as parent
    WORKDIR /app
    COPY docker-entrypoint.sh /usr/local/bin/
    ENTRYPOINT ["/usr/local/bin/docker-entrypoint.sh"]
    CMD ["snyk", "test"]
    FROM ubuntu as snyk
    RUN apt-get update && apt-get install -y curl wget
    RUN curl -s https://api.github.com/repos/snyk/snyk/releases/latest | grep "browser_download_url" |
    grep linux | cut -d '"' -f 4 | wget -i - && \
    sha256sum -c snyk-linux.sha256 && \
    mv snyk-linux /usr/local/bin/snyk && \
    chmod +x /usr/local/bin/snyk
    

    View Slide

33. 1 Dockerfile
    Download precompiled Snyk for each platform
    FROM ubuntu as snyk
    RUN apt-get update && apt-get install -y curl wget
    RUN curl -s https://api.github.com/repos/snyk/snyk/releases/latest | grep
    "browser_download_url" | grep linux | cut -d '"' -f 4 | wget -i - && \
    sha256sum -c snyk-linux.sha256 && \
    mv snyk-linux /usr/local/bin/snyk && \
    chmod +x /usr/local/bin/snyk
    FROM alpine as snyk-alpine
    RUN apk add --no-cache curl wget
    RUN curl -s https://api.github.com/repos/snyk/snyk/releases/latest | grep
    "browser_download_url" | grep alpine | cut -d '"' -f 4 | wget -i - && \
    sha256sum -c snyk-alpine.sha256 && \
    mv snyk-alpine /usr/local/bin/snyk && \
    

    View Slide

34. 1 Dockerfile
    Output targets for each platform
    FROM parent as alpine
    RUN apk add --no-cache libstdc++
    COPY --from=snyk-alpine /usr/local/bin/snyk /usr/local/bin/snyk
    FROM parent as linux
    COPY --from=snyk /usr/local/bin/snyk /usr/local/bin/snyk
    

    View Slide

35. 1 Action generating all the other Actions
    Did I mention I like metaprogramming?
    

    View Slide

36. Runs on push
    Run only when build files change
    name: "Generate Actions to build Snyk images"
    on:
    push:
    branches:
    - master
    paths:
    - build.rb
    - linux
    - alpine
    

    View Slide

37. Enter the matrix
    Generated matrix for each combination
    matrix:
    include:
    - base: clojure:boot
    tag: clojure-boot
    target: linux
    - base: clojure:lein
    tag: clojure-lein
    target: linux
    - base: clojure:tools-deps
    tag: clojure-tools-deps
    target: linux
    - base: golang
    tag: golang
    target: linux
    - base: golang:1.12
    

    View Slide

38. build-push-image
    With the inputs from the matrix
    steps:
    - uses: actions/checkout@v1
    - uses: docker/build-push-action@v1
    env:
    DOCKER_BUILDKIT: "1"
    with:
    username: ${{ secrets.DOCKER_USERNAME }}
    password: ${{ secrets.DOCKER_PASSWORD }}
    add_git_labels: true
    target: ${{ matrix.target }}
    repository: snyk/snyk
    tags: ${{ matrix.tag }}
    build_args: IMAGE=${{ matrix.base }}
    

    View Slide

39. Example OCI labels
    Great for understanding provenance
    $ docker inspect snyk/snyk:python --format="{{json .Config.Labels}}" | jq
    {
    "org.opencontainers.image.created": "2020-05-17T00:14:15Z",
    "org.opencontainers.image.revision": "fa19b4d7fda74be0f342926c3c7feeb368f88b17",
    "org.opencontainers.image.source": "https://github.com/snyk/snyk-images"
    }
    

    View Slide

40. Demo
    

    View Slide

41. Conclusions
    If all you remember is...
    

    View Slide

42. Learn Dockerfile
    Dockerfile is incredible simple to
    get started with, but hides some
    powerful features that not enough
    folks learn to use
    

    View Slide

43. Bring automation closer
    to developers
    Identify programmatic solutions to
    repetitive problems
    

    View Slide

44. Focus on maintenance
    The value of reducing the overhead
    of ongoing work needed to do the
    job is often underestimated
    

    View Slide

45. Thanks for listening
    Sign up for free at snyk.io/signup
    

    View Slide