Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Container Security for RED and BLUE Teams! - All Day DevOps 2020 Spring Break Edition

Container Security for RED and BLUE Teams! - All Day DevOps 2020 Spring Break Edition

In this talk will focus on two different perspectives of container security. We will see, how attackers can break into docker container and kubernetes clusters to gain access, escalate privileges to infrastructure by using misconfigurations and application security vulnerabilities. Also, we can see how defenders can leverage the power of automation at different layers like Infrastructure Security, Supply chain Security and Run time Security to protect against the container security attacks.

By the end of this talk participants able to use this knowledge to identify and exploit vulnerabilities in applications running on containers inside kubernetes clusters. Also, it will be useful to apply pragmatic security using tools, techniques and procedures (TTPs) to secure the container infrastructure.

**About Madhu Akula**

Madhu Akula is a security ninja, published author and cloud native security researcher with an extensive experience. Also he is an active member of the international security, devops and cloud native communities (null, DevSecOps, AllDayDevOps, etc). Hold industry certifications like OSCP (Offensive Security Certified Professional), CKA (Certified Kubernetes Administrator), etc. His research has identified vulnerabilities in over 200+ companies and organisations including; Google, Microsoft, LinkedIn, eBay, AT&T, WordPress, NTOP and Adobe, etc and credited with multiple CVE’s, Acknowledgements and rewards. He is co-author of Security Automation with Ansible2 (ISBN-13: 978-1788394512), which is listed as a technical resource by Red Hat Ansible. Also won 1st prize for building Infrastructure Security Monitoring solution at InMobi flagship hackathon among 100+ engineering teams.

Madhu Akula

April 17, 2020
Tweet

More Decks by Madhu Akula

Other Decks in Technology

Transcript

  1. About - Madhu Akula
    • Cloud Native Security Specialist @ Xebia
    • Security (Cloud, Containers, Kubernetes & Automation)
    • Speaker & Trainer @ BlackHat, DEF CON, USENIX LISA, OWASP Appsec EU,
    All Day DevOps, DevSecCon, c0c0n, Nullcon, null, etc.
    • Co-Author of Security Automation with Ansible2 book
    • Discovered vulnerabilities in over 200+ organizations including; Google, Microsoft,
    LinkedIn, Coudflare, AT&T, Wordpress, NTOP and Adobe, etc.
    • Holds industry certifications like OSCP and CKA
    • Never Ending Learner!
    • https://madhuakula.com
    @madhuakula
    B
    R

    View full-size slide

  2. What you will learn in next 30 minutes!
    • This talk is NOT about what is Docker, Kubernetes, etc.
    • This talk is about
    • Why container security?
    • Red team overview of container security
    • Blue team overview of container security
    • Tools, Techniques and Procedures (TTP’s)
    • Real-world scenarios
    • Approach to attackers and defenders
    • Case studies and reference resources
    • Next steps for learning more and more
    @madhuakula
    B
    R

    View full-size slide

  3. Would you like to learn Docker & Kubernetes?
    • https://docs.docker.com
    • https://kubernetes.io/docs/home
    • https://container.training
    • https://training.play-with-docker.com
    • https://labs.play-with-k8s.com
    • https://training.play-with-kubernetes.com
    • https://www.katacoda.com/learn
    • Many more...
    @madhuakula
    B
    R

    View full-size slide

  4. Why container security?
    @madhuakula
    https://kromtech.com/blog/security-center/cryptojacking-invades-cloud-how-modern-containerization-trend-is-exploited-by-attackers

    View full-size slide

  5. Why container security?
    @madhuakula
    https://blog.madhuakula.com/some-tips-to-review-docker-hub-hack-of-190k-accounts-addcd602aade

    View full-size slide

  6. Why container security?
    @madhuakula
    https://hackerone.com/reports/341876

    View full-size slide

  7. Why container security?
    Many other vulnerabilities and real-world impacts...
    @madhuakula

    View full-size slide

  8. Container Security Overview
    • Container Attack Surface
    • Namespaces
    • Control Groups
    • Daemon
    • Configuration
    • Capabilities
    • Content Trust
    • Container Registry
    • Volumes
    • Networks
    • Many other...
    @madhuakula
    B
    R

    View full-size slide

  9. Kubernetes Security Overview
    @madhuakula
    https://www.microsoft.com/security/blog/2020/04/02/attack-matrix-kubernetes/
    B
    R

    View full-size slide

  10. amicontained - Container Introspection Tool
    https://github.com/genuinetools/amicontained
    It helps to find out what container runtime is being used as well as features available like
    capabilities, profiles applied, etc.
    amicontained - Container Introspection Tool B
    R

    View full-size slide

  11. • Committing the sensitive information
    to version control systems
    • Not including the sensitive files in the
    build process using .dockerignore file
    • This is one of the common mistake in
    modern era
    trufflehog - Hardcoded sensitive information B
    R

    View full-size slide

  12. Insecurely configured docker service
    R

    View full-size slide

  13. Insecure docker socket service
    Insecure docker socket service
    R

    View full-size slide

  14. Insecure docker socket service
    Securing Daemon Configuration B
    • By default, Docker runs through a non-networked UNIX socket. It can
    also optionally communicate using an HTTP socket.
    • If you need Docker to be reachable through the network in a safe
    manner, you can enable TLS by specifying the tlsverify flag and pointing
    Docker’s tlscacert flag to a trusted CA certificate.
    • If you want to secure your Docker client connections by default, you can
    move the files to the .docker directory in your home directory --- and set
    the DOCKER_HOST and DOCKER_TLS_VERIFY variables as well (instead of
    passing -H=tcp://$HOST:2376 and --tlsverify on every call).
    https://docs.docker.com/engine/security/https

    View full-size slide

  15. Analysing or Understanding unknown image
    Analyzing or Understanding unknown image B
    R

    View full-size slide

  16. Insecure docker socket service
    Content Trust B
    • Docker Content Trust (DCT) provides the ability to use digital signatures
    for data sent to and received from remote Docker registries. These
    signatures allow client-side or runtime verification of the integrity and
    publisher of specific image tags
    • We can sign and push a container image with the docker trust command
    • Content trust is disabled by default in the Docker Client. To enable it, set
    the DOCKER_CONTENT_TRUST environment variable to 1. This prevents
    users from working with tagged images unless they contain a signature.
    https://docs.docker.com/engine/security/trust/content_trust/

    View full-size slide

  17. dive - Exploring each layer in a docker image
    https://github.com/wagoodman/dive
    dive - Exploring each layer in a docker image B
    R

    View full-size slide

  18. Inspecting container volumes
    Inspecting container volumes B
    R

    View full-size slide

  19. Volume analysis for sensitive information
    Volume analysis for sensitive information
    R

    View full-size slide

  20. Inspecting container networking
    Inspecting container networking B
    R

    View full-size slide

  21. Always look for env variables
    ● This is one of the common places most
    developers and operations teams store
    secrets, API keys, etc.
    ● Also it contains other information like
    different service or cluster related
    information
    Always look for env variables B
    R

    View full-size slide

  22. docker diff - comparing with base image
    docker diff - comparing with base image B
    R

    View full-size slide

  23. container escape - extra capability and host pid
    container escape - extra capability and host pid
    R

    View full-size slide

  24. container escape - extra capability and host pid
    container escape - extra capability and host pid
    R

    View full-size slide

  25. Insecure docker socket service
    Security Profiles B
    • The Linux Security Module (LSM) framework provides a mechanism for
    various security checks to be hooked by new kernel extensions
    • AppArmor profile generator for docker containers using bane
    https://github.com/genuinetools/bane

    View full-size slide

  26. Kubernetes secrets are not encrypted!
    Kubernetes secrets are not encrypted! B
    R

    View full-size slide

  27. Insecure docker socket service
    Injecting Vault Secrets Into K8S Pods via a Sidecar B
    https://www.hashicorp.com/blog/injecting-vault-secrets-into-kubernetes-pods-via-a-sidecar/
    # patch-basic-annotations.yaml
    spec:
    template:
    metadata:
    annotations:
    vault.hashicorp.com/agent-inject: "true"
    vault.hashicorp.com/agent-inject-secret-helloworld: "secrets/helloworld"
    vault.hashicorp.com/role: "myapp"

    View full-size slide

  28. SSRF in the kubernetes world like a Cluster Pwn
    ● In the Google Cloud (GCP), we have to
    use Metadata-Flavor: Google to obtain
    the metadata
    ● Now GKE offers to protect kube-env
    using metadata concealment proxy
    and workload identity
    SSRF in the kubernetes world like a Cluster Pwn
    R

    View full-size slide

  29. SSRF in the kubernetes world like a Cluster Pwn
    SSRF in the kubernetes world like a Cluster Pwn
    R

    View full-size slide

  30. Insecure docker socket service
    Metadata Concealment B
    • Most of the cloud providers has fix for this in some way
    • GKE: Workload Identity, Metadata Concealment for Nodes
    https://cloud.google.com/kubernetes-engine/docs/how-to/workload-identity
    • AWS: IMDSv2 for SSRF
    https://aws.amazon.com/blogs/security/defense-in-depth-open-firewalls-reverse-proxies-ssrf-vulnerabilities-ec2-instance-
    metadata-service/
    https://github.com/genuinetools/bane

    View full-size slide

  31. Command Injection to node access (host)
    Command Injection to node access (host)
    R

    View full-size slide

  32. Command Injection to node access (host)
    Command Injection to node access (host)
    R

    View full-size slide

  33. Command Injection to node access (host)
    Command Injection to node access (host)
    R

    View full-size slide

  34. Kubernetes centralised logs in stack driver
    Runtime Security Detection – Sysdig Falco
    https://www.youtube.com/watch?v=zd0ksjZI5Vk
    https://falco.org/
    Falco, the open source cloud-native runtime security project, is the defacto Kubernetes threat detection engine.
    Falco detects unexpected application behaviour and alerts on threats at runtime.
    B

    View full-size slide

  35. No default security boundary in k8s namespaces
    No default security boundary in k8s namespaces B
    R

    View full-size slide

  36. Kubernetes centralised logs in stack driver
    Network Security Policies
    https://github.com/ahmetb/kubernetes-network-policy-recipes
    Provides isolation between Kubernetes resources (pods, namespaces, svc, etc.) using labels and
    selectors across the cluster.
    B

    View full-size slide

  37. Default service account in a Pod
    Default service account in a Pod
    R

    View full-size slide

  38. Default service account in a Pod
    https://hackernoon.com/capturing-all-the-flags-in-bsidessf-ctf-by-pwning-our-infrastructure-3570b99b4dd0
    Default service account in a Pod
    Default service account in a Pod
    https://hackernoon.com/capturing-all-the-flags-in-bsidessf-ctf-by-pwning-our-infrastructure-3570b99b4dd0
    Default service account in a Pod
    R

    View full-size slide

  39. Default misconfigured Helm Tiller = Cluster Pwn
    https://engineering.bitnami.com/articles/helm-security.html
    Default misconfigured Helm Tiller = Cluster Pwn
    R

    View full-size slide

  40. Default misconfigured Helm Tiller = Cluster Pwn
    Default misconfigured Helm Tiller = Cluster Pwn
    R

    View full-size slide

  41. Kubernetes centralised logs in stack driver
    RBAC with least privilege access possible
    https://kubernetes.io/docs/reference/access-authn-authz/rbac/
    Role-based access control (RBAC) is a method of regulating access to computer or network resources
    based on the roles of individual users within your organization.
    B
    Useful utilities to check out is
    https://github.com/liggitt/audit2rbac
    https://github.com/FairwindsOps/rbac
    -manager
    https://github.com/jtblin/kube2iam

    View full-size slide

  42. Trivy - Vulnerability Scanner for Containers
    https://github.com/aquasecurity/trivy
    Trivy - Vulnerability Scanner for Containers B
    R

    View full-size slide

  43. dockle - Container Image Linter for Security
    https://github.com/goodwithtech/dockle
    dockle - Container Image Linter for Security B
    R

    View full-size slide

  44. docker-bench-security
    https://github.com/docker/docker-bench-security
    ● A script that checks for dozens of
    common best-practices around
    deploying Docker containers in
    production
    ○ Host configuration
    ○ Docker daemon configuration and files
    ○ Docker container images
    ○ Docker runtime
    ○ Docker security operations
    ○ Docker swarm configuration
    docker-bench-security B
    R

    View full-size slide

  45. kube-bench - CIS Kubernetes Benchmark
    https://github.com/aquasecurity/kube-bench
    ● Master Node Security Configuration
    ○ API Server
    ○ Scheduler
    ○ Controller Manager
    ○ Configuration Files
    ○ etcd
    ○ General Security Primitives
    ○ PodSecurityPolicices
    ● Worker Node Security Configuration
    ○ Kubelet
    ○ Configuration Files
    kube-bench - CIS Kubernetes Benchmark B
    R

    View full-size slide

  46. kube-hunter
    ● Kube-hunter hunts for security
    weaknesses in Kubernetes clusters.
    The tool was developed to increase
    awareness and visibility for security
    issues in Kubernetes environments.
    You should NOT run kube-hunter on a
    Kubernetes cluster you don't own!
    kube-hunter B
    R

    View full-size slide

  47. kubeaudit - Audit your kubernetes clusters
    https://github.com/Shopify/kubeaudit
    kubeaudit - Audit your kubernetes clusters B
    R

    View full-size slide

  48. kubesec.io - Risk analysis for k8s resources
    https://kubesec.io/
    kubesec.io - Risk analysis for k8s resources B
    R

    View full-size slide

  49. CVE-2018-1002105
    https://www.youtube.com/watch?v=4CTK2aUXTHo
    Known Vulnerabilities (CVE-2018-1002105) B
    R

    View full-size slide

  50. https://www.youtube.com/watch?v=4CTK2aUXTHo
    CVE-2018-1002105
    Known Vulnerabilities (CVE-2018-1002105) B
    R

    View full-size slide

  51. https://github.com/Frichetten/CVE-2019-5736-PoC
    ● This is a Go implementation
    of CVE-2019-5736, a
    container escape for Docker.
    The exploit works by
    overwriting and executing
    the host systems runc binary
    from within the container
    CVE-2019-5736
    Known Vulnerabilities (CVE-2019-5736) B
    R

    View full-size slide

  52. https://github.com/eoftedal/writings/blob/master/published/CVE-2019-9901-path-traversal.md
    CVE-2019-9901 - Istio/Envoy Path traversal
    CVE-2019-9901 : Istio/Envoy Path traversal B
    R

    View full-size slide

  53. docker logs and events
    docker logs and events B
    R

    View full-size slide

  54. Kubernetes centralised logs in stack driver
    Centralized logging and Monitoring B

    View full-size slide

  55. Kubernetes centralised logs in stack driver
    Open Policy Agent – Policy Engine
    https://www.openpolicyagent.org/
    Policy-based control for cloud native environments
    Flexible, fine-grained control for administrators across the stack
    B

    View full-size slide

  56. Kubernetes centralised logs in stack driver
    Container Runtime Sandbox
    gVisor, Firecracker, etc.
    B
    • gVisor is a user-space kernel, written in Go, that implements a substantial
    portion of the Linux system call interface. It provides an additional layer
    of isolation between running applications and the host operating system.
    • Firecracker is an open source virtualization technology that is purpose-
    built for creating and managing secure, multi-tenant container and
    function-based services.
    • Many other…

    View full-size slide

  57. Kubernetes centralised logs in stack driver
    TLS – Let’s Encrypt with cert-manager
    https://cert-manager.io/
    Automate certificate management in cloud native environments. cert-manager builds on top of
    Kubernetes, introducing certificate authorities and certificates as first-class resource types in the
    Kubernetes API. This makes it possible to provide 'certificates as a service' to developers working within
    your Kubernetes cluster.
    B

    View full-size slide

  58. Kubernetes centralised logs in stack driver
    Pod Security Policies
    https://kubernetes.io/docs/concepts/policy/pod-security-policy
    A Pod Security Policy is a cluster-
    level resource that controls
    security sensitive aspects of the
    pod specification. The
    PodSecurityPolicy objects define
    a set of conditions that a pod
    must run with in order to be
    accepted into the system, as
    well as defaults for the related
    fields.
    Good utility to check out is
    https://github.com/sysdiglabs/k
    ube-psp-advisor
    B

    View full-size slide

  59. References & Resources
    • Docker Security Docs
    • Kubernetes Security Docs
    • Attack matrix for Kubernetes
    • Breaking & Pwning Docker Containers & Kubernetes Clusters
    • Advanced Persistence Threats: The Future of Kubernetes Attacks
    • Kubernetes Security Resources
    • 11 Ways (Not) to Get Hacked
    • Attacking & Auditing Docker Containers using Open Source @ DEFCON 26
    • Attacking and Auditing Docker Containers and Kubernetes Clusters @ DEFCON 27
    @madhuakula
    B
    R

    View full-size slide

  60. References & Resources
    • contained.af
    • CIS Benchmarks Docker
    • Understanding and Hardening Linux Containers
    • Abusing Privileged and Unprivileged Linux Containers
    • Container Security Notes
    • Linux Container Security
    • Docker Runtime Privileges and Capabilities
    • Apparmor Security Profiles on Docker
    • Seccomp Security Profiles on Docker
    • Docker Labs Capabilities
    • Practical SELinux and Containers
    • Containers and Operating systems morning paper gist
    • Kubernetes Network Policies
    • Kubernetes Webinar series
    @madhuakula
    B
    R

    View full-size slide

  61. Are you looking for more and more tools?
    https://tools.cloudnativesecurity.info
    @madhuakula
    B
    R

    View full-size slide

  62. Thank You!
    Madhu Akula
    @madhuakula
    https://madhuakula.com
    @madhuakula
    B
    R

    View full-size slide