DevOps Ruhr Meetup Container Software Supply Chains

Create faster value to automate your Container Software Supply Chain.

Talk at the 1th DevOps Ruhr Meetup at Essen ruhr:HUB from Niclas Mietz and Peter Rossbach from bee42 solutions gmbh.

* https://www.meetup.com/de-DE/DevOps-Ruhr/
* https://bee42.com

Transcript

1. Container So ware Supply Chain 1
2. Peter Rossbach    & Niclas Mietz    peter.rossbach@bee42.com prossbach prossbach niclas.mietz@bee42.com solidnerd solidnerd 2 . 1
3. Customer Journey to CSSC Automatic build, test and deploy your container images Reuse base images Use the newest stable container images Don`t ship images with security risk inside Use a catalog of images with stable components or stacks Fail Fast Validate correct software functionality Continuous Delivery or Deployment your software at different stages Allow incremental change 2 . 2
4. Simple Container So ware Supply Chain 3 . 1
5. Example: Build with Gitlab/CI 3 . 2
6. Docker Command Flow 3 . 3
7. Container image promotion 3 . 4
8. Build Container with a container toolchain 3 . 5
9. Example: Gitlab pipeline config # This file is a template, and might need editing # before it works on your project. image: docker:latest services: - docker:dind before_script: - docker login -u "$CI_REGISTRY_USER" \ -p "$CI_REGISTRY_PASSWORD" $CI_REGISTRY build-master: stage: build script: - docker build --pull -t "$CI_REGISTRY_IMAGE" . - docker push "$CI_REGISTRY_IMAGE" only: - master build: stage: build script: - docker build --pull -t "$CI_REGISTRY_IMAGE:$CI_COMMIT_REF_SLUG" . - docker push "$CI_REGISTRY_IMAGE:$CI_COMMIT_REF_SLUG" except: - master 3 . 6
10. Sample Dockerfile FROM golang:alpine as builder MAINTAINER Jessica Frazelle <jess@linux.com> ENV PATH /go/bin:/usr/local/go/bin:$PATH ENV GOPATH /go RUN apk add --no-cache \ ca-certificates COPY . /go/src/github.com/jessfraz/dockfmt RUN set -x \ && apk add --no-cache --virtual .build-deps \ git \ gcc \ libc-dev \ libgcc \ make \ && cd /go/src/github.com/jessfraz/dockfmt \ && make static \ && mv dockfmt /usr/bin/dockfmt \ && apk del .build-deps \ && rm -rf /go \ && echo "Build complete." FROM scratch COPY --from=builder /usr/bin/dockfmt /usr/bin/dockfmt COPY --from=builder /etc/ssl/certs/ /etc/ssl/certs ENTRYPOINT [ "dockfmt" ] CMD [ "--help" ] https://github.com/jessfraz/dockfmt 3 . 7
11. What's wrong? Gitlab/CI Runner need priviledges to run docker:dind DinD has no default docker image cache Slow down build performance This Build isn`t repeatable Use Image Digest Use a Package Mirror or Package Image Container (Linuxkit) Support concurrent runner builds Parallel build from different commits and branches! Control Gitlab-Runner Host Docker Engine access rights Docker CE Engine allow all commands Docker Sock Proxy Twistlock Auth Plugin Use Auth Plugin - Habormaster Use Docker EE Engine ACL's https://github.com/Tecnativa/docker-socket-proxy https://github.com/twistlock/authz http://harbormaster.io/ 3 . 8
12. Reduce allowed images at your GitLab Runner You can use any images for your jobs and services! Use only trusted images Use only trusted registries Sign your images https://docs.gitlab.com/runner/executors/docker.html#the- image-keyword 3 . 9
13. Example Runner Config Check your GitLab Runner con guration config.toml in the runner.docker section By default is ["*/*:*"] enabled concurrent = 1 check_interval = 0 [[runners]] name = "wormhole runner for everything" url = "<gitlab-instance-url>" token = "<my-magic-token>" executor = "docker" [runners.docker] tls_verify = false image = "docker:latest" privileged = false disable_cache = false allowed_images = ["myregistry.com/*:*","go:*"] allowed_services = ["postgres:9.6","mysql:5.7","mongodb:*"] volumes = ["/cache","/var/run/docker.sock:/var/run/docker.sock"] [runners.cache] 3 . 10
14. Reduce allowed images in a GitLab Runner (cont.) By default is ["*/*:*"] enabled concurrent = 1 check_interval = 0 [[runners]] name = "docker dind" url = "<gitlab-instance-url>" token = "<my-magic-token>" executor = "docker" [runners.docker] tls_verify = false image = "docker:latest" privileged = true disable_cache = false volumes = ["/cache"] allowed_images = ["docker:*"] allowed_services = ["docker:*"] [runners.cache] 3 . 11
15. Tipps Deploy isolated stacks docker-compose -p <project name>-<branch name>-<revision> <command> Clean up your runners Drop the waste: unused container, volumes, images, networks, services Use docker-gc inside runner host Replace your runner host every day Use Image-Digest at Docker le FROM Trigger to use newer version Use a spezial runner to create newer base images How to promote the images? Create a merge request at your main developer branch Rebase your feature branches before merge (developer driven) 3 . 12
16. How organize container projects? Build a single image per project Build complex software with Docker multi stage builds Build complex software with multiple Docker les Sometimes you must build a family of images Multi Arch Java Build (JDK) and Runtime Support multi main versions (like Apache Tomcat 7/8/9) Matrix Builds Multi image projects 3 . 13
17. How deploy container so ware? Use simple docker-compose (version 2) Deploy stacks to your docker swarm (docker-compose version 3) Use kubernetes and deploy with helm 3 . 14
18. Build your container environment with security Secure your runtime Build your container images secure Management your secrets and con gs Monitor your system 4 . 1
19. Secure your Container Runtime Limited access to kubernetes, docker engine and containerd. Control your user access Use Client Certs for remote access Use Docker Engine Auth Use Kubernetes policies and RBAC rules https://docs.docker.com/engine/extend/plugins_authorization/ https://kubernetes.io/docs/admin/authorization/rbac/ https://github.com/kubernetes/examples/blob/master/staging/podsecuritypolicy/rbac https://speakerdeck.com/ianlewis/kubernetes-security-best-practices https://github.com/ahmetb/kubernetes-network-policy-recipes/ 4 . 2
20. Check Image Provenance Enable DOCKER_CONTENT_TRUST environment variable, so that only signed images can be pulled. Store root key of ine Backup signing keys; rotate & expire old keys Check image digest at deployment 4 . 3
21. Monitor Containers Capture host logs Capture logs from Docker infrastructure Capture container logs Ensure adequate log information at the containers Ensure --log-level is set to INFO (default) Tag your container log messages Check source of images that use at production 4 . 4
22. Do Not Run Container Processes as Root Container processes run as non-privileged USER If process must run as root, use Docker User Namespace feature to e-map to non-privileged user on host Limit access to run Docker Engine and container Don`t share host namespaces between containers Remove setUID/setGID binaries 4 . 5
23. Don`t store secrets in Container Images Secrets are not stored in the Docker les/included in the image Secrets are not stored in environment variables Secrets are not stored in volume mounts on the host Secrets are stored in a dedicated secrets management system Docker Swarm Kubernetes Hashicorp Vault Secrets are rotated frequently 4 . 6
24. Add Base Image Security Specify package versions and hash in FROM tag Understand the contents of running images Scan images regularly for vulnerabilities Update base image regularly and re-deploy containers 4 . 7
25. Control your Base Image creation Build from trusted source Mirror all external images and dependencies Build without internet at your pipelines Use a secure up to date kernel Control your base os image Control your base container images Use only restricted internale package mirrors with snapshots Limit your linux distribution Use Linux Hardining Tipps inside your image build No fun! 4 . 8
26. Limit container resources Mount lesystems read-only (prevent writing malicious code) Place limits on system resources (limit denial of service) Memory CPU Limit kernel calls (limit container breakout) and Linux capabilities Restrict network access (prevent attack pivot, data egress, - icc=false) 4 . 9
27. Don`t start container without Host Security is active SELinux AppArmor Seccomp 4 . 10
28. Control your network access Isolate your network traf c Use rewalls between containers and pods 4 . 11
29. Vulnerability scanning Add CVE Scanning to your build pipeline 5 . 1
30. Harbor Registry http://vmware.github.io/harbor/ 5 . 2
31. CVE Checking Push nothing that has over Trash Harbor (Receive all images) Set correctly policy for replication Build active Pipeline to check the image before it will be pushed Add a daily job to scan used images 5 . 3
32. Active Images Detect vulnerability at your systems Don’t redeploy vulnerability images Create issue with the affected images name in the Docker le Watcher on Registry/Harbor for vulnabilities Active handlin Check if new images CVE exists Check only an exclusive list of images Rebuild all images nightly when vulnerability exist 5 . 4
33. Image Promotion strategies Promote Images with tagging Promote Images to another registry account Promote Images to another registry (Mirror) 5 . 5
34. Promote Images with tagging docker: stage: docker image: docker:17.12.0-ce script: - docker build -t $CI_REGISTRY_IMAGE:${CI_COMMIT_SHA} . - docker login -u gitlab-ci-token -p $CI_JOB_TOKEN $CI_REGISTRY - docker push $CI_REGISTRY_IMAGE:${CI_COMMIT_SHA} pushimage: stage: registry image: docker:17.12.0-ce only: - master script: - docker login -u gitlab-ci-token -p $CI_JOB_TOKEN $CI_REGISTRY - docker image pull $CI_REGISTRY_IMAGE:${CI_COMMIT_SHA} - docker image tag $CI_REGISTRY_IMAGE:${CI_COMMIT_SHA} $CI_REGISTRY_IMA - docker push $CI_REGISTRY_IMAGE:latest 5 . 6
35. Promote Images to another registry account REMOTE_REGISTRY_PASSWORD will be set through a secret variable docker_deploy_harbor: stage: docker_deploy_harbor image: docker variables: REMOTE_REGISTRY_URL: "harbor.bee42.com" REMOTE_REGISTRY_IMAGE: "${REMOTE_REGISTRY_URL}/${CI_PROJECT_NAME}:${C REMOTE_REGISTRY_USER: "bee-habor" script: - docker login -u gitlab-ci-token -p $CI_BUILD_TOKEN $CI_REGISTRY - docker pull $CI_REGISTRY_IMAGE:${CI_COMMIT_SHA} - docker image tag $CI_REGISTRY_IMAGE:${CI_COMMIT_SHA} $REMOTE_REGI - docker login -u $REMOTE_REGISTRY_USER -p $REMOTE_REGISTRY_PASSWOR - docker push $REMOTE_REGISTRY_IMAGE 5 . 7
36. Why use multiple registries Multi-Region Replication Easier CleanUp Different Types for Audit, Archive, Distribution Can only use images are available inside the stage env It isn`t allowed to use external network traf c 5 . 8
37. Add Application Security (App Sec) to pipeline Static Application Security Testing Dynamic Application Security Testing OWASP Security rules 5 . 9
38. Create a Trust Chain of Images CVE Check before push to registry Create a issue per project Check after CVE DB change Which repo must be checked Daily build Sign your images Use package mirror Use only your trusted container images Build your images without internet access Only use automated pipelines to deploy to your systems Don`t patch, create from scratch! 5 . 10
39. bee42 Trainings Container DevOps Camp Kubernetes DevOps Lab Container Java Lab Container GitLab/CI Lab https://bee42.com 6
40. RETHINK IT: DevOps Gathering 2018 Conference Date: 7
41. Open the doors for Q&A 8
42. Links https://www.sans.org/reading- room/whitepapers/auditing/checklist-audit-docker- containers-37437 https://www.nirmata.com/2016/01/22/continuous-delivery- for-containerized-applications/ http://martinfowler.com/bliki/BlueGreenDeployment.html 9