DevOps Ruhr Meetup Container Software Supply Chains

Create faster value to automate your Container Software Supply Chain.

Talk at the 1th DevOps Ruhr Meetup at Essen ruhr:HUB from Niclas Mietz and Peter Rossbach from bee42 solutions gmbh.

* https://www.meetup.com/de-DE/DevOps-Ruhr/
* https://bee42.com

Transcript

1. Container
   So ware Supply
   Chain
   1
   

   View Slide

2. Peter Rossbach
   
   
   
   &
   Niclas Mietz
   
   
   
   peter.rossbach@bee42.com
   prossbach
   prossbach
   niclas.mietz@bee42.com
   solidnerd
   solidnerd
   2 . 1
   

   View Slide

3. Customer Journey to CSSC
   Automatic build, test and deploy your container images
   Reuse base images
   Use the newest stable container images
   Don`t ship images with security risk inside
   Use a catalog of images with stable components or stacks
   Fail Fast
   Validate correct software functionality
   Continuous Delivery or Deployment your software at
   different stages
   Allow incremental change
   2 . 2
   

   View Slide

4. Simple Container So ware
   Supply Chain
   3 . 1
   

   View Slide

5. Example: Build with Gitlab/CI
   3 . 2
   

   View Slide

6. Docker Command Flow
   3 . 3
   

   View Slide

7. Container image promotion
   3 . 4
   

   View Slide

8. Build Container with a container toolchain
   3 . 5
   

   View Slide

9. Example: Gitlab pipeline config
   # This file is a template, and might need editing
   # before it works on your project.
   image: docker:latest
   services:
   - docker:dind
   before_script:
   - docker login -u "$CI_REGISTRY_USER" \
   -p "$CI_REGISTRY_PASSWORD" $CI_REGISTRY
   build-master:
   stage: build
   script:
   - docker build --pull -t "$CI_REGISTRY_IMAGE" .
   - docker push "$CI_REGISTRY_IMAGE"
   only:
   - master
   build:
   stage: build
   script:
   - docker build --pull -t "$CI_REGISTRY_IMAGE:$CI_COMMIT_REF_SLUG" .
   - docker push "$CI_REGISTRY_IMAGE:$CI_COMMIT_REF_SLUG"
   except:
   - master
   3 . 6
   

   View Slide

10. Sample Dockerfile
    FROM golang:alpine as builder
    MAINTAINER Jessica Frazelle
    ENV PATH /go/bin:/usr/local/go/bin:$PATH
    ENV GOPATH /go
    RUN apk add --no-cache \
    ca-certificates
    COPY . /go/src/github.com/jessfraz/dockfmt
    RUN set -x \
    && apk add --no-cache --virtual .build-deps \
    git \
    gcc \
    libc-dev \
    libgcc \
    make \
    && cd /go/src/github.com/jessfraz/dockfmt \
    && make static \
    && mv dockfmt /usr/bin/dockfmt \
    && apk del .build-deps \
    && rm -rf /go \
    && echo "Build complete."
    FROM scratch
    COPY --from=builder /usr/bin/dockfmt /usr/bin/dockfmt
    COPY --from=builder /etc/ssl/certs/ /etc/ssl/certs
    ENTRYPOINT [ "dockfmt" ]
    CMD [ "--help" ]
    https://github.com/jessfraz/dockfmt
    3 . 7
    

    View Slide

11. What's wrong?
    Gitlab/CI Runner need priviledges to run docker:dind
    DinD has no default docker image cache
    Slow down build performance
    This Build isn`t repeatable
    Use Image Digest
    Use a Package Mirror or Package Image Container
    (Linuxkit)
    Support concurrent runner builds
    Parallel build from different commits and branches!
    Control Gitlab-Runner Host Docker Engine access rights
    Docker CE Engine allow all commands
    Docker Sock Proxy
    Twistlock Auth Plugin
    Use Auth Plugin - Habormaster
    Use Docker EE Engine ACL's
    https://github.com/Tecnativa/docker-socket-proxy
    https://github.com/twistlock/authz
    http://harbormaster.io/
    3 . 8
    

    View Slide

12. Reduce allowed images at your GitLab
    Runner
    You can use any images for your jobs and services!
    Use only trusted images
    Use only trusted registries
    Sign your images
    https://docs.gitlab.com/runner/executors/docker.html#the-
    image-keyword
    3 . 9
    

    View Slide

13. Example Runner Config
    Check your GitLab Runner con guration config.toml in the
    runner.docker section
    By default is ["*/*:*"] enabled
    concurrent = 1
    check_interval = 0
    [[runners]]
    name = "wormhole runner for everything"
    url = ""
    token = ""
    executor = "docker"
    [runners.docker]
    tls_verify = false
    image = "docker:latest"
    privileged = false
    disable_cache = false
    allowed_images = ["myregistry.com/*:*","go:*"]
    allowed_services = ["postgres:9.6","mysql:5.7","mongodb:*"]
    volumes = ["/cache","/var/run/docker.sock:/var/run/docker.sock"]
    [runners.cache]
    3 . 10
    

    View Slide

14. Reduce allowed images in a GitLab Runner
    (cont.)
    By default is ["*/*:*"] enabled
    concurrent = 1
    check_interval = 0
    [[runners]]
    name = "docker dind"
    url = ""
    token = ""
    executor = "docker"
    [runners.docker]
    tls_verify = false
    image = "docker:latest"
    privileged = true
    disable_cache = false
    volumes = ["/cache"]
    allowed_images = ["docker:*"]
    allowed_services = ["docker:*"]
    [runners.cache]
    3 . 11
    

    View Slide

15. Tipps
    Deploy isolated stacks
    docker-compose -p -name>-
    Clean up your runners
    Drop the waste: unused container, volumes, images,
    networks, services
    Use docker-gc inside runner host
    Replace your runner host every day
    Use Image-Digest at Docker le FROM
    Trigger to use newer version
    Use a spezial runner to create newer base images
    How to promote the images?
    Create a merge request at your main developer branch
    Rebase your feature branches before merge (developer
    driven)
    3 . 12
    

    View Slide

16. How organize container projects?
    Build a single image per project
    Build complex software with Docker multi stage builds
    Build complex software with multiple Docker les
    Sometimes you must build a family of images
    Multi Arch
    Java Build (JDK) and Runtime
    Support multi main versions (like Apache Tomcat 7/8/9)
    Matrix Builds
    Multi image projects
    3 . 13
    

    View Slide

17. How deploy container so ware?
    Use simple docker-compose (version 2)
    Deploy stacks to your docker swarm (docker-compose
    version 3)
    Use kubernetes and deploy with helm
    3 . 14
    

    View Slide

18. Build your container
    environment with security
    Secure your runtime
    Build your container images secure
    Management your secrets and con gs
    Monitor your system
    4 . 1
    

    View Slide

19. Secure your Container Runtime
    Limited access to kubernetes, docker engine and containerd.
    Control your user access
    Use Client Certs for remote access
    Use Docker Engine Auth
    Use Kubernetes policies and RBAC rules
    https://docs.docker.com/engine/extend/plugins_authorization/
    https://kubernetes.io/docs/admin/authorization/rbac/
    https://github.com/kubernetes/examples/blob/master/staging/podsecuritypolicy/rbac
    https://speakerdeck.com/ianlewis/kubernetes-security-best-practices
    https://github.com/ahmetb/kubernetes-network-policy-recipes/
    4 . 2
    

    View Slide

20. Check Image Provenance
    Enable DOCKER_CONTENT_TRUST environment variable, so
    that only signed images can be pulled.
    Store root key of ine
    Backup signing keys; rotate & expire old keys
    Check image digest at deployment
    4 . 3
    

    View Slide

21. Monitor Containers
    Capture host logs
    Capture logs from Docker infrastructure
    Capture container logs
    Ensure adequate log information at the containers
    Ensure --log-level is set to INFO (default)
    Tag your container log messages
    Check source of images that use at production
    4 . 4
    

    View Slide

22. Do Not Run Container Processes as Root
    Container processes run as non-privileged USER
    If process must run as root, use Docker User Namespace
    feature to e-map to non-privileged user on host
    Limit access to run Docker Engine and container
    Don`t share host namespaces between containers
    Remove setUID/setGID binaries
    4 . 5
    

    View Slide

23. Don`t store secrets in Container Images
    Secrets are not stored in the Docker les/included in the
    image
    Secrets are not stored in environment variables
    Secrets are not stored in volume mounts on the host
    Secrets are stored in a dedicated secrets management
    system
    Docker Swarm
    Kubernetes
    Hashicorp Vault
    Secrets are rotated frequently
    4 . 6
    

    View Slide

24. Add Base Image Security
    Specify package versions and hash in FROM tag
    Understand the contents of running images
    Scan images regularly for vulnerabilities
    Update base image regularly and re-deploy containers
    4 . 7
    

    View Slide

25. Control your Base Image creation
    Build from trusted source
    Mirror all external images and dependencies
    Build without internet at your pipelines
    Use a secure up to date kernel
    Control your base os image
    Control your base container images
    Use only restricted internale package mirrors with
    snapshots
    Limit your linux distribution
    Use Linux Hardining Tipps inside your image build
    No fun!
    4 . 8
    

    View Slide

26. Limit container resources
    Mount lesystems read-only (prevent writing malicious
    code)
    Place limits on system resources (limit denial of service)
    Memory
    CPU
    Limit kernel calls (limit container breakout) and Linux
    capabilities
    Restrict network access (prevent attack pivot, data egress, -
    icc=false)
    4 . 9
    

    View Slide

27. Don`t start container without Host
    Security is active
    SELinux
    AppArmor
    Seccomp
    4 . 10
    

    View Slide

28. Control your network access
    Isolate your network traf c
    Use rewalls between containers and pods
    4 . 11
    

    View Slide

29. Vulnerability scanning
    Add CVE Scanning to your build pipeline
    5 . 1
    

    View Slide

30. Harbor Registry
    http://vmware.github.io/harbor/
    5 . 2
    

    View Slide

31. CVE Checking
    Push nothing that has over
    Trash Harbor (Receive all images)
    Set correctly policy for replication
    Build active Pipeline to check the image before it will be
    pushed
    Add a daily job to scan used images
    5 . 3
    

    View Slide

32. Active Images
    Detect vulnerability at your systems
    Don’t redeploy vulnerability images
    Create issue with the affected images name in the Docker le
    Watcher on Registry/Harbor for vulnabilities
    Active handlin
    Check if new images CVE exists
    Check only an exclusive list of images
    Rebuild all images nightly when vulnerability exist
    5 . 4
    

    View Slide

33. Image Promotion strategies
    Promote Images with tagging
    Promote Images to another registry account
    Promote Images to another registry (Mirror)
    5 . 5
    

    View Slide

34. Promote Images with tagging
    docker:
    stage: docker
    image: docker:17.12.0-ce
    script:
    - docker build -t $CI_REGISTRY_IMAGE:${CI_COMMIT_SHA} .
    - docker login -u gitlab-ci-token -p $CI_JOB_TOKEN $CI_REGISTRY
    - docker push $CI_REGISTRY_IMAGE:${CI_COMMIT_SHA}
    pushimage:
    stage: registry
    image: docker:17.12.0-ce
    only:
    - master
    script:
    - docker login -u gitlab-ci-token -p $CI_JOB_TOKEN $CI_REGISTRY
    - docker image pull $CI_REGISTRY_IMAGE:${CI_COMMIT_SHA}
    - docker image tag $CI_REGISTRY_IMAGE:${CI_COMMIT_SHA} $CI_REGISTRY_IMA
    - docker push $CI_REGISTRY_IMAGE:latest
    5 . 6
    

    View Slide

35. Promote Images to another registry account
    REMOTE_REGISTRY_PASSWORD will be set through a secret
    variable
    docker_deploy_harbor:
    stage: docker_deploy_harbor
    image: docker
    variables:
    REMOTE_REGISTRY_URL: "harbor.bee42.com"
    REMOTE_REGISTRY_IMAGE: "${REMOTE_REGISTRY_URL}/${CI_PROJECT_NAME}:${C
    REMOTE_REGISTRY_USER: "bee-habor"
    script:
    - docker login -u gitlab-ci-token -p $CI_BUILD_TOKEN $CI_REGISTRY
    - docker pull $CI_REGISTRY_IMAGE:${CI_COMMIT_SHA}
    - docker image tag $CI_REGISTRY_IMAGE:${CI_COMMIT_SHA} $REMOTE_REGI
    - docker login -u $REMOTE_REGISTRY_USER -p $REMOTE_REGISTRY_PASSWOR
    - docker push $REMOTE_REGISTRY_IMAGE
    5 . 7
    

    View Slide

36. Why use multiple registries
    Multi-Region
    Replication
    Easier CleanUp
    Different Types for Audit, Archive, Distribution
    Can only use images are available inside the stage env
    It isn`t allowed to use external network traf c
    5 . 8
    

    View Slide

37. Add Application Security (App Sec) to
    pipeline
    Static Application Security Testing
    Dynamic Application Security Testing
    OWASP Security rules
    5 . 9
    

    View Slide

38. Create a Trust Chain of Images
    CVE Check before push to registry
    Create a issue per project
    Check after CVE DB change
    Which repo must be checked
    Daily build
    Sign your images
    Use package mirror
    Use only your trusted container images
    Build your images without internet access
    Only use automated pipelines to deploy to your systems
    Don`t patch, create from scratch!
    5 . 10
    

    View Slide

39. bee42 Trainings
    Container DevOps Camp
    Kubernetes DevOps Lab
    Container Java Lab
    Container GitLab/CI Lab
    https://bee42.com
    6
    

    View Slide

40. RETHINK IT: DevOps Gathering
    2018
    Conference Date: 7
    

    View Slide

41. Open the doors for Q&A
    8
    

    View Slide

42. Links
    https://www.sans.org/reading-
    room/whitepapers/auditing/checklist-audit-docker-
    containers-37437
    https://www.nirmata.com/2016/01/22/continuous-delivery-
    for-containerized-applications/
    http://martinfowler.com/bliki/BlueGreenDeployment.html
    9
    

    View Slide