$30 off During Our Annual Pro Sale. View Details »

DevOps Ruhr Meetup Container Software Supply Chains

DevOps Ruhr Meetup Container Software Supply Chains

Create faster value to automate your Container Software Supply Chain.

Talk at the 1th DevOps Ruhr Meetup at Essen ruhr:HUB from Niclas Mietz and Peter Rossbach from bee42 solutions gmbh.

* https://www.meetup.com/de-DE/DevOps-Ruhr/
* https://bee42.com

Peter Rossbach

February 07, 2018
Tweet

More Decks by Peter Rossbach

Other Decks in Programming

Transcript

  1. Container
    So ware Supply
    Chain
    1

    View Slide

  2. Peter Rossbach



    &
    Niclas Mietz



    [email protected]
    prossbach
    prossbach
    [email protected]
    solidnerd
    solidnerd
    2 . 1

    View Slide

  3. Customer Journey to CSSC
    Automatic build, test and deploy your container images
    Reuse base images
    Use the newest stable container images
    Don`t ship images with security risk inside
    Use a catalog of images with stable components or stacks
    Fail Fast
    Validate correct software functionality
    Continuous Delivery or Deployment your software at
    different stages
    Allow incremental change
    2 . 2

    View Slide

  4. Simple Container So ware
    Supply Chain
    3 . 1

    View Slide

  5. Example: Build with Gitlab/CI
    3 . 2

    View Slide

  6. Docker Command Flow
    3 . 3

    View Slide

  7. Container image promotion
    3 . 4

    View Slide

  8. Build Container with a container toolchain
    3 . 5

    View Slide

  9. Example: Gitlab pipeline config
    # This file is a template, and might need editing
    # before it works on your project.
    image: docker:latest
    services:
    - docker:dind
    before_script:
    - docker login -u "$CI_REGISTRY_USER" \
    -p "$CI_REGISTRY_PASSWORD" $CI_REGISTRY
    build-master:
    stage: build
    script:
    - docker build --pull -t "$CI_REGISTRY_IMAGE" .
    - docker push "$CI_REGISTRY_IMAGE"
    only:
    - master
    build:
    stage: build
    script:
    - docker build --pull -t "$CI_REGISTRY_IMAGE:$CI_COMMIT_REF_SLUG" .
    - docker push "$CI_REGISTRY_IMAGE:$CI_COMMIT_REF_SLUG"
    except:
    - master
    3 . 6

    View Slide

  10. Sample Dockerfile
    FROM golang:alpine as builder
    MAINTAINER Jessica Frazelle
    ENV PATH /go/bin:/usr/local/go/bin:$PATH
    ENV GOPATH /go
    RUN apk add --no-cache \
    ca-certificates
    COPY . /go/src/github.com/jessfraz/dockfmt
    RUN set -x \
    && apk add --no-cache --virtual .build-deps \
    git \
    gcc \
    libc-dev \
    libgcc \
    make \
    && cd /go/src/github.com/jessfraz/dockfmt \
    && make static \
    && mv dockfmt /usr/bin/dockfmt \
    && apk del .build-deps \
    && rm -rf /go \
    && echo "Build complete."
    FROM scratch
    COPY --from=builder /usr/bin/dockfmt /usr/bin/dockfmt
    COPY --from=builder /etc/ssl/certs/ /etc/ssl/certs
    ENTRYPOINT [ "dockfmt" ]
    CMD [ "--help" ]
    https://github.com/jessfraz/dockfmt
    3 . 7

    View Slide

  11. What's wrong?
    Gitlab/CI Runner need priviledges to run docker:dind
    DinD has no default docker image cache
    Slow down build performance
    This Build isn`t repeatable
    Use Image Digest
    Use a Package Mirror or Package Image Container
    (Linuxkit)
    Support concurrent runner builds
    Parallel build from different commits and branches!
    Control Gitlab-Runner Host Docker Engine access rights
    Docker CE Engine allow all commands
    Docker Sock Proxy
    Twistlock Auth Plugin
    Use Auth Plugin - Habormaster
    Use Docker EE Engine ACL's
    https://github.com/Tecnativa/docker-socket-proxy
    https://github.com/twistlock/authz
    http://harbormaster.io/
    3 . 8

    View Slide

  12. Reduce allowed images at your GitLab
    Runner
    You can use any images for your jobs and services!
    Use only trusted images
    Use only trusted registries
    Sign your images
    https://docs.gitlab.com/runner/executors/docker.html#the-
    image-keyword
    3 . 9

    View Slide

  13. Example Runner Config
    Check your GitLab Runner con guration config.toml in the
    runner.docker section
    By default is ["*/*:*"] enabled
    concurrent = 1
    check_interval = 0
    [[runners]]
    name = "wormhole runner for everything"
    url = ""
    token = ""
    executor = "docker"
    [runners.docker]
    tls_verify = false
    image = "docker:latest"
    privileged = false
    disable_cache = false
    allowed_images = ["myregistry.com/*:*","go:*"]
    allowed_services = ["postgres:9.6","mysql:5.7","mongodb:*"]
    volumes = ["/cache","/var/run/docker.sock:/var/run/docker.sock"]
    [runners.cache]
    3 . 10

    View Slide

  14. Reduce allowed images in a GitLab Runner
    (cont.)
    By default is ["*/*:*"] enabled
    concurrent = 1
    check_interval = 0
    [[runners]]
    name = "docker dind"
    url = ""
    token = ""
    executor = "docker"
    [runners.docker]
    tls_verify = false
    image = "docker:latest"
    privileged = true
    disable_cache = false
    volumes = ["/cache"]
    allowed_images = ["docker:*"]
    allowed_services = ["docker:*"]
    [runners.cache]
    3 . 11

    View Slide

  15. Tipps
    Deploy isolated stacks
    docker-compose -p -name>-
    Clean up your runners
    Drop the waste: unused container, volumes, images,
    networks, services
    Use docker-gc inside runner host
    Replace your runner host every day
    Use Image-Digest at Docker le FROM
    Trigger to use newer version
    Use a spezial runner to create newer base images
    How to promote the images?
    Create a merge request at your main developer branch
    Rebase your feature branches before merge (developer
    driven)
    3 . 12

    View Slide

  16. How organize container projects?
    Build a single image per project
    Build complex software with Docker multi stage builds
    Build complex software with multiple Docker les
    Sometimes you must build a family of images
    Multi Arch
    Java Build (JDK) and Runtime
    Support multi main versions (like Apache Tomcat 7/8/9)
    Matrix Builds
    Multi image projects
    3 . 13

    View Slide

  17. How deploy container so ware?
    Use simple docker-compose (version 2)
    Deploy stacks to your docker swarm (docker-compose
    version 3)
    Use kubernetes and deploy with helm
    3 . 14

    View Slide

  18. Build your container
    environment with security
    Secure your runtime
    Build your container images secure
    Management your secrets and con gs
    Monitor your system
    4 . 1

    View Slide

  19. Secure your Container Runtime
    Limited access to kubernetes, docker engine and containerd.
    Control your user access
    Use Client Certs for remote access
    Use Docker Engine Auth
    Use Kubernetes policies and RBAC rules
    https://docs.docker.com/engine/extend/plugins_authorization/
    https://kubernetes.io/docs/admin/authorization/rbac/
    https://github.com/kubernetes/examples/blob/master/staging/podsecuritypolicy/rbac
    https://speakerdeck.com/ianlewis/kubernetes-security-best-practices
    https://github.com/ahmetb/kubernetes-network-policy-recipes/
    4 . 2

    View Slide

  20. Check Image Provenance
    Enable DOCKER_CONTENT_TRUST environment variable, so
    that only signed images can be pulled.
    Store root key of ine
    Backup signing keys; rotate & expire old keys
    Check image digest at deployment
    4 . 3

    View Slide

  21. Monitor Containers
    Capture host logs
    Capture logs from Docker infrastructure
    Capture container logs
    Ensure adequate log information at the containers
    Ensure --log-level is set to INFO (default)
    Tag your container log messages
    Check source of images that use at production
    4 . 4

    View Slide

  22. Do Not Run Container Processes as Root
    Container processes run as non-privileged USER
    If process must run as root, use Docker User Namespace
    feature to e-map to non-privileged user on host
    Limit access to run Docker Engine and container
    Don`t share host namespaces between containers
    Remove setUID/setGID binaries
    4 . 5

    View Slide

  23. Don`t store secrets in Container Images
    Secrets are not stored in the Docker les/included in the
    image
    Secrets are not stored in environment variables
    Secrets are not stored in volume mounts on the host
    Secrets are stored in a dedicated secrets management
    system
    Docker Swarm
    Kubernetes
    Hashicorp Vault
    Secrets are rotated frequently
    4 . 6

    View Slide

  24. Add Base Image Security
    Specify package versions and hash in FROM tag
    Understand the contents of running images
    Scan images regularly for vulnerabilities
    Update base image regularly and re-deploy containers
    4 . 7

    View Slide

  25. Control your Base Image creation
    Build from trusted source
    Mirror all external images and dependencies
    Build without internet at your pipelines
    Use a secure up to date kernel
    Control your base os image
    Control your base container images
    Use only restricted internale package mirrors with
    snapshots
    Limit your linux distribution
    Use Linux Hardining Tipps inside your image build
    No fun!
    4 . 8

    View Slide

  26. Limit container resources
    Mount lesystems read-only (prevent writing malicious
    code)
    Place limits on system resources (limit denial of service)
    Memory
    CPU
    Limit kernel calls (limit container breakout) and Linux
    capabilities
    Restrict network access (prevent attack pivot, data egress, -
    icc=false)
    4 . 9

    View Slide

  27. Don`t start container without Host
    Security is active
    SELinux
    AppArmor
    Seccomp
    4 . 10

    View Slide

  28. Control your network access
    Isolate your network traf c
    Use rewalls between containers and pods
    4 . 11

    View Slide

  29. Vulnerability scanning
    Add CVE Scanning to your build pipeline
    5 . 1

    View Slide

  30. Harbor Registry
    http://vmware.github.io/harbor/
    5 . 2

    View Slide

  31. CVE Checking
    Push nothing that has over
    Trash Harbor (Receive all images)
    Set correctly policy for replication
    Build active Pipeline to check the image before it will be
    pushed
    Add a daily job to scan used images
    5 . 3

    View Slide

  32. Active Images
    Detect vulnerability at your systems
    Don’t redeploy vulnerability images
    Create issue with the affected images name in the Docker le
    Watcher on Registry/Harbor for vulnabilities
    Active handlin
    Check if new images CVE exists
    Check only an exclusive list of images
    Rebuild all images nightly when vulnerability exist
    5 . 4

    View Slide

  33. Image Promotion strategies
    Promote Images with tagging
    Promote Images to another registry account
    Promote Images to another registry (Mirror)
    5 . 5

    View Slide

  34. Promote Images with tagging
    docker:
    stage: docker
    image: docker:17.12.0-ce
    script:
    - docker build -t $CI_REGISTRY_IMAGE:${CI_COMMIT_SHA} .
    - docker login -u gitlab-ci-token -p $CI_JOB_TOKEN $CI_REGISTRY
    - docker push $CI_REGISTRY_IMAGE:${CI_COMMIT_SHA}
    pushimage:
    stage: registry
    image: docker:17.12.0-ce
    only:
    - master
    script:
    - docker login -u gitlab-ci-token -p $CI_JOB_TOKEN $CI_REGISTRY
    - docker image pull $CI_REGISTRY_IMAGE:${CI_COMMIT_SHA}
    - docker image tag $CI_REGISTRY_IMAGE:${CI_COMMIT_SHA} $CI_REGISTRY_IMA
    - docker push $CI_REGISTRY_IMAGE:latest
    5 . 6

    View Slide

  35. Promote Images to another registry account
    REMOTE_REGISTRY_PASSWORD will be set through a secret
    variable
    docker_deploy_harbor:
    stage: docker_deploy_harbor
    image: docker
    variables:
    REMOTE_REGISTRY_URL: "harbor.bee42.com"
    REMOTE_REGISTRY_IMAGE: "${REMOTE_REGISTRY_URL}/${CI_PROJECT_NAME}:${C
    REMOTE_REGISTRY_USER: "bee-habor"
    script:
    - docker login -u gitlab-ci-token -p $CI_BUILD_TOKEN $CI_REGISTRY
    - docker pull $CI_REGISTRY_IMAGE:${CI_COMMIT_SHA}
    - docker image tag $CI_REGISTRY_IMAGE:${CI_COMMIT_SHA} $REMOTE_REGI
    - docker login -u $REMOTE_REGISTRY_USER -p $REMOTE_REGISTRY_PASSWOR
    - docker push $REMOTE_REGISTRY_IMAGE
    5 . 7

    View Slide

  36. Why use multiple registries
    Multi-Region
    Replication
    Easier CleanUp
    Different Types for Audit, Archive, Distribution
    Can only use images are available inside the stage env
    It isn`t allowed to use external network traf c
    5 . 8

    View Slide

  37. Add Application Security (App Sec) to
    pipeline
    Static Application Security Testing
    Dynamic Application Security Testing
    OWASP Security rules
    5 . 9

    View Slide

  38. Create a Trust Chain of Images
    CVE Check before push to registry
    Create a issue per project
    Check after CVE DB change
    Which repo must be checked
    Daily build
    Sign your images
    Use package mirror
    Use only your trusted container images
    Build your images without internet access
    Only use automated pipelines to deploy to your systems
    Don`t patch, create from scratch!
    5 . 10

    View Slide

  39. bee42 Trainings
    Container DevOps Camp
    Kubernetes DevOps Lab
    Container Java Lab
    Container GitLab/CI Lab
    https://bee42.com
    6

    View Slide

  40. RETHINK IT: DevOps Gathering
    2018
    Conference Date: 7

    View Slide

  41. Open the doors for Q&A
    8

    View Slide

  42. Links
    https://www.sans.org/reading-
    room/whitepapers/auditing/checklist-audit-docker-
    containers-37437
    https://www.nirmata.com/2016/01/22/continuous-delivery-
    for-containerized-applications/
    http://martinfowler.com/bliki/BlueGreenDeployment.html
    9

    View Slide