Upgrade to Pro — share decks privately, control downloads, hide ads and more …

DevOps Ruhr Meetup Container Software Supply Chains

DevOps Ruhr Meetup Container Software Supply Chains

Create faster value to automate your Container Software Supply Chain.

Talk at the 1th DevOps Ruhr Meetup at Essen ruhr:HUB from Niclas Mietz and Peter Rossbach from bee42 solutions gmbh.

* https://www.meetup.com/de-DE/DevOps-Ruhr/
* https://bee42.com

Peter Rossbach

February 07, 2018

More Decks by Peter Rossbach

Other Decks in Programming


  1. Customer Journey to CSSC Automatic build, test and deploy your

    container images Reuse base images Use the newest stable container images Don`t ship images with security risk inside Use a catalog of images with stable components or stacks Fail Fast Validate correct software functionality Continuous Delivery or Deployment your software at different stages Allow incremental change 2 . 2
  2. Example: Gitlab pipeline config # This file is a template,

    and might need editing # before it works on your project. image: docker:latest services: - docker:dind before_script: - docker login -u "$CI_REGISTRY_USER" \ -p "$CI_REGISTRY_PASSWORD" $CI_REGISTRY build-master: stage: build script: - docker build --pull -t "$CI_REGISTRY_IMAGE" . - docker push "$CI_REGISTRY_IMAGE" only: - master build: stage: build script: - docker build --pull -t "$CI_REGISTRY_IMAGE:$CI_COMMIT_REF_SLUG" . - docker push "$CI_REGISTRY_IMAGE:$CI_COMMIT_REF_SLUG" except: - master 3 . 6
  3. Sample Dockerfile FROM golang:alpine as builder MAINTAINER Jessica Frazelle <[email protected]>

    ENV PATH /go/bin:/usr/local/go/bin:$PATH ENV GOPATH /go RUN apk add --no-cache \ ca-certificates COPY . /go/src/github.com/jessfraz/dockfmt RUN set -x \ && apk add --no-cache --virtual .build-deps \ git \ gcc \ libc-dev \ libgcc \ make \ && cd /go/src/github.com/jessfraz/dockfmt \ && make static \ && mv dockfmt /usr/bin/dockfmt \ && apk del .build-deps \ && rm -rf /go \ && echo "Build complete." FROM scratch COPY --from=builder /usr/bin/dockfmt /usr/bin/dockfmt COPY --from=builder /etc/ssl/certs/ /etc/ssl/certs ENTRYPOINT [ "dockfmt" ] CMD [ "--help" ] https://github.com/jessfraz/dockfmt 3 . 7
  4. What's wrong? Gitlab/CI Runner need priviledges to run docker:dind DinD

    has no default docker image cache Slow down build performance This Build isn`t repeatable Use Image Digest Use a Package Mirror or Package Image Container (Linuxkit) Support concurrent runner builds Parallel build from different commits and branches! Control Gitlab-Runner Host Docker Engine access rights Docker CE Engine allow all commands Docker Sock Proxy Twistlock Auth Plugin Use Auth Plugin - Habormaster Use Docker EE Engine ACL's https://github.com/Tecnativa/docker-socket-proxy https://github.com/twistlock/authz http://harbormaster.io/ 3 . 8
  5. Reduce allowed images at your GitLab Runner You can use

    any images for your jobs and services! Use only trusted images Use only trusted registries Sign your images https://docs.gitlab.com/runner/executors/docker.html#the- image-keyword 3 . 9
  6. Example Runner Config Check your GitLab Runner con guration config.toml

    in the runner.docker section By default is ["*/*:*"] enabled concurrent = 1 check_interval = 0 [[runners]] name = "wormhole runner for everything" url = "<gitlab-instance-url>" token = "<my-magic-token>" executor = "docker" [runners.docker] tls_verify = false image = "docker:latest" privileged = false disable_cache = false allowed_images = ["myregistry.com/*:*","go:*"] allowed_services = ["postgres:9.6","mysql:5.7","mongodb:*"] volumes = ["/cache","/var/run/docker.sock:/var/run/docker.sock"] [runners.cache] 3 . 10
  7. Reduce allowed images in a GitLab Runner (cont.) By default

    is ["*/*:*"] enabled concurrent = 1 check_interval = 0 [[runners]] name = "docker dind" url = "<gitlab-instance-url>" token = "<my-magic-token>" executor = "docker" [runners.docker] tls_verify = false image = "docker:latest" privileged = true disable_cache = false volumes = ["/cache"] allowed_images = ["docker:*"] allowed_services = ["docker:*"] [runners.cache] 3 . 11
  8. Tipps Deploy isolated stacks docker-compose -p <project name>-<branch name>-<revision> <command>

    Clean up your runners Drop the waste: unused container, volumes, images, networks, services Use docker-gc inside runner host Replace your runner host every day Use Image-Digest at Docker le FROM Trigger to use newer version Use a spezial runner to create newer base images How to promote the images? Create a merge request at your main developer branch Rebase your feature branches before merge (developer driven) 3 . 12
  9. How organize container projects? Build a single image per project

    Build complex software with Docker multi stage builds Build complex software with multiple Docker les Sometimes you must build a family of images Multi Arch Java Build (JDK) and Runtime Support multi main versions (like Apache Tomcat 7/8/9) Matrix Builds Multi image projects 3 . 13
  10. How deploy container so ware? Use simple docker-compose (version 2)

    Deploy stacks to your docker swarm (docker-compose version 3) Use kubernetes and deploy with helm 3 . 14
  11. Build your container environment with security Secure your runtime Build

    your container images secure Management your secrets and con gs Monitor your system 4 . 1
  12. Secure your Container Runtime Limited access to kubernetes, docker engine

    and containerd. Control your user access Use Client Certs for remote access Use Docker Engine Auth Use Kubernetes policies and RBAC rules https://docs.docker.com/engine/extend/plugins_authorization/ https://kubernetes.io/docs/admin/authorization/rbac/ https://github.com/kubernetes/examples/blob/master/staging/podsecuritypolicy/rbac https://speakerdeck.com/ianlewis/kubernetes-security-best-practices https://github.com/ahmetb/kubernetes-network-policy-recipes/ 4 . 2
  13. Check Image Provenance Enable DOCKER_CONTENT_TRUST environment variable, so that only

    signed images can be pulled. Store root key of ine Backup signing keys; rotate & expire old keys Check image digest at deployment 4 . 3
  14. Monitor Containers Capture host logs Capture logs from Docker infrastructure

    Capture container logs Ensure adequate log information at the containers Ensure --log-level is set to INFO (default) Tag your container log messages Check source of images that use at production 4 . 4
  15. Do Not Run Container Processes as Root Container processes run

    as non-privileged USER If process must run as root, use Docker User Namespace feature to e-map to non-privileged user on host Limit access to run Docker Engine and container Don`t share host namespaces between containers Remove setUID/setGID binaries 4 . 5
  16. Don`t store secrets in Container Images Secrets are not stored

    in the Docker les/included in the image Secrets are not stored in environment variables Secrets are not stored in volume mounts on the host Secrets are stored in a dedicated secrets management system Docker Swarm Kubernetes Hashicorp Vault Secrets are rotated frequently 4 . 6
  17. Add Base Image Security Specify package versions and hash in

    FROM tag Understand the contents of running images Scan images regularly for vulnerabilities Update base image regularly and re-deploy containers 4 . 7
  18. Control your Base Image creation Build from trusted source Mirror

    all external images and dependencies Build without internet at your pipelines Use a secure up to date kernel Control your base os image Control your base container images Use only restricted internale package mirrors with snapshots Limit your linux distribution Use Linux Hardining Tipps inside your image build No fun! 4 . 8
  19. Limit container resources Mount lesystems read-only (prevent writing malicious code)

    Place limits on system resources (limit denial of service) Memory CPU Limit kernel calls (limit container breakout) and Linux capabilities Restrict network access (prevent attack pivot, data egress, - icc=false) 4 . 9
  20. Control your network access Isolate your network traf c Use

    rewalls between containers and pods 4 . 11
  21. CVE Checking Push nothing that has over Trash Harbor (Receive

    all images) Set correctly policy for replication Build active Pipeline to check the image before it will be pushed Add a daily job to scan used images 5 . 3
  22. Active Images Detect vulnerability at your systems Don’t redeploy vulnerability

    images Create issue with the affected images name in the Docker le Watcher on Registry/Harbor for vulnabilities Active handlin Check if new images CVE exists Check only an exclusive list of images Rebuild all images nightly when vulnerability exist 5 . 4
  23. Image Promotion strategies Promote Images with tagging Promote Images to

    another registry account Promote Images to another registry (Mirror) 5 . 5
  24. Promote Images with tagging docker: stage: docker image: docker:17.12.0-ce script:

    - docker build -t $CI_REGISTRY_IMAGE:${CI_COMMIT_SHA} . - docker login -u gitlab-ci-token -p $CI_JOB_TOKEN $CI_REGISTRY - docker push $CI_REGISTRY_IMAGE:${CI_COMMIT_SHA} pushimage: stage: registry image: docker:17.12.0-ce only: - master script: - docker login -u gitlab-ci-token -p $CI_JOB_TOKEN $CI_REGISTRY - docker image pull $CI_REGISTRY_IMAGE:${CI_COMMIT_SHA} - docker image tag $CI_REGISTRY_IMAGE:${CI_COMMIT_SHA} $CI_REGISTRY_IMA - docker push $CI_REGISTRY_IMAGE:latest 5 . 6
  25. Promote Images to another registry account REMOTE_REGISTRY_PASSWORD will be set

    through a secret variable docker_deploy_harbor: stage: docker_deploy_harbor image: docker variables: REMOTE_REGISTRY_URL: "harbor.bee42.com" REMOTE_REGISTRY_IMAGE: "${REMOTE_REGISTRY_URL}/${CI_PROJECT_NAME}:${C REMOTE_REGISTRY_USER: "bee-habor" script: - docker login -u gitlab-ci-token -p $CI_BUILD_TOKEN $CI_REGISTRY - docker pull $CI_REGISTRY_IMAGE:${CI_COMMIT_SHA} - docker image tag $CI_REGISTRY_IMAGE:${CI_COMMIT_SHA} $REMOTE_REGI - docker login -u $REMOTE_REGISTRY_USER -p $REMOTE_REGISTRY_PASSWOR - docker push $REMOTE_REGISTRY_IMAGE 5 . 7
  26. Why use multiple registries Multi-Region Replication Easier CleanUp Different Types

    for Audit, Archive, Distribution Can only use images are available inside the stage env It isn`t allowed to use external network traf c 5 . 8
  27. Add Application Security (App Sec) to pipeline Static Application Security

    Testing Dynamic Application Security Testing OWASP Security rules 5 . 9
  28. Create a Trust Chain of Images CVE Check before push

    to registry Create a issue per project Check after CVE DB change Which repo must be checked Daily build Sign your images Use package mirror Use only your trusted container images Build your images without internet access Only use automated pipelines to deploy to your systems Don`t patch, create from scratch! 5 . 10
  29. bee42 Trainings Container DevOps Camp Kubernetes DevOps Lab Container Java

    Lab Container GitLab/CI Lab https://bee42.com 6