Container Security for RED and BLUE Teams! - All Day DevOps 2020 Spring Break Edition

In this talk will focus on two different perspectives of container security. We will see, how attackers can break into docker container and kubernetes clusters to gain access, escalate privileges to infrastructure by using misconfigurations and application security vulnerabilities. Also, we can see how defenders can leverage the power of automation at different layers like Infrastructure Security, Supply chain Security and Run time Security to protect against the container security attacks.

By the end of this talk participants able to use this knowledge to identify and exploit vulnerabilities in applications running on containers inside kubernetes clusters. Also, it will be useful to apply pragmatic security using tools, techniques and procedures (TTPs) to secure the container infrastructure.

**About Madhu Akula**

Madhu Akula is a security ninja, published author and cloud native security researcher with an extensive experience. Also he is an active member of the international security, devops and cloud native communities (null, DevSecOps, AllDayDevOps, etc). Hold industry certifications like OSCP (Offensive Security Certified Professional), CKA (Certified Kubernetes Administrator), etc. His research has identified vulnerabilities in over 200+ companies and organisations including; Google, Microsoft, LinkedIn, eBay, AT&T, WordPress, NTOP and Adobe, etc and credited with multiple CVE’s, Acknowledgements and rewards. He is co-author of Security Automation with Ansible2 (ISBN-13: 978-1788394512), which is listed as a technical resource by Red Hat Ansible. Also won 1st prize for building Infrastructure Security Monitoring solution at InMobi flagship hackathon among 100+ engineering teams.

Transcript

1. B
   R
   

   View Slide

2. About - Madhu Akula
   • Cloud Native Security Specialist @ Xebia
   • Security (Cloud, Containers, Kubernetes & Automation)
   • Speaker & Trainer @ BlackHat, DEF CON, USENIX LISA, OWASP Appsec EU,
   All Day DevOps, DevSecCon, c0c0n, Nullcon, null, etc.
   • Co-Author of Security Automation with Ansible2 book
   • Discovered vulnerabilities in over 200+ organizations including; Google, Microsoft,
   LinkedIn, Coudflare, AT&T, Wordpress, NTOP and Adobe, etc.
   • Holds industry certifications like OSCP and CKA
   • Never Ending Learner!
   • https://madhuakula.com
   @madhuakula
   B
   R
   

   View Slide

3. What you will learn in next 30 minutes!
   • This talk is NOT about what is Docker, Kubernetes, etc.
   • This talk is about
   • Why container security?
   • Red team overview of container security
   • Blue team overview of container security
   • Tools, Techniques and Procedures (TTP’s)
   • Real-world scenarios
   • Approach to attackers and defenders
   • Case studies and reference resources
   • Next steps for learning more and more
   @madhuakula
   B
   R
   

   View Slide

4. Would you like to learn Docker & Kubernetes?
   • https://docs.docker.com
   • https://kubernetes.io/docs/home
   • https://container.training
   • https://training.play-with-docker.com
   • https://labs.play-with-k8s.com
   • https://training.play-with-kubernetes.com
   • https://www.katacoda.com/learn
   • Many more...
   @madhuakula
   B
   R
   

   View Slide

5. Why container security?
   @madhuakula
   https://kromtech.com/blog/security-center/cryptojacking-invades-cloud-how-modern-containerization-trend-is-exploited-by-attackers
   

   View Slide

6. Why container security?
   @madhuakula
   https://blog.madhuakula.com/some-tips-to-review-docker-hub-hack-of-190k-accounts-addcd602aade
   

   View Slide

7. Why container security?
   @madhuakula
   https://hackerone.com/reports/341876
   

   View Slide

8. Why container security?
   Many other vulnerabilities and real-world impacts...
   @madhuakula
   

   View Slide

9. Container Security Overview
   • Container Attack Surface
   • Namespaces
   • Control Groups
   • Daemon
   • Configuration
   • Capabilities
   • Content Trust
   • Container Registry
   • Volumes
   • Networks
   • Many other...
   @madhuakula
   B
   R
   

   View Slide

10. Kubernetes Security Overview
    @madhuakula
    https://www.microsoft.com/security/blog/2020/04/02/attack-matrix-kubernetes/
    B
    R
    

    View Slide

11. amicontained - Container Introspection Tool
    https://github.com/genuinetools/amicontained
    It helps to find out what container runtime is being used as well as features available like
    capabilities, profiles applied, etc.
    amicontained - Container Introspection Tool B
    R
    

    View Slide

12. • Committing the sensitive information
    to version control systems
    • Not including the sensitive files in the
    build process using .dockerignore file
    • This is one of the common mistake in
    modern era
    trufflehog - Hardcoded sensitive information B
    R
    

    View Slide

13. Insecurely configured docker service
    R
    

    View Slide

14. Insecure docker socket service
    Insecure docker socket service
    R
    

    View Slide

15. Insecure docker socket service
    Securing Daemon Configuration B
    • By default, Docker runs through a non-networked UNIX socket. It can
    also optionally communicate using an HTTP socket.
    • If you need Docker to be reachable through the network in a safe
    manner, you can enable TLS by specifying the tlsverify flag and pointing
    Docker’s tlscacert flag to a trusted CA certificate.
    • If you want to secure your Docker client connections by default, you can
    move the files to the .docker directory in your home directory --- and set
    the DOCKER_HOST and DOCKER_TLS_VERIFY variables as well (instead of
    passing -H=tcp://$HOST:2376 and --tlsverify on every call).
    https://docs.docker.com/engine/security/https
    

    View Slide

16. Analysing or Understanding unknown image
    Analyzing or Understanding unknown image B
    R
    

    View Slide

17. Insecure docker socket service
    Content Trust B
    • Docker Content Trust (DCT) provides the ability to use digital signatures
    for data sent to and received from remote Docker registries. These
    signatures allow client-side or runtime verification of the integrity and
    publisher of specific image tags
    • We can sign and push a container image with the docker trust command
    • Content trust is disabled by default in the Docker Client. To enable it, set
    the DOCKER_CONTENT_TRUST environment variable to 1. This prevents
    users from working with tagged images unless they contain a signature.
    https://docs.docker.com/engine/security/trust/content_trust/
    

    View Slide

18. dive - Exploring each layer in a docker image
    https://github.com/wagoodman/dive
    dive - Exploring each layer in a docker image B
    R
    

    View Slide

19. Inspecting container volumes
    Inspecting container volumes B
    R
    

    View Slide

20. Volume analysis for sensitive information
    Volume analysis for sensitive information
    R
    

    View Slide

21. Inspecting container networking
    Inspecting container networking B
    R
    

    View Slide

22. Always look for env variables
    ● This is one of the common places most
    developers and operations teams store
    secrets, API keys, etc.
    ● Also it contains other information like
    different service or cluster related
    information
    Always look for env variables B
    R
    

    View Slide

23. docker diff - comparing with base image
    docker diff - comparing with base image B
    R
    

    View Slide

24. container escape - extra capability and host pid
    container escape - extra capability and host pid
    R
    

    View Slide

25. container escape - extra capability and host pid
    container escape - extra capability and host pid
    R
    

    View Slide

26. Insecure docker socket service
    Security Profiles B
    • The Linux Security Module (LSM) framework provides a mechanism for
    various security checks to be hooked by new kernel extensions
    • AppArmor profile generator for docker containers using bane
    https://github.com/genuinetools/bane
    

    View Slide

27. Kubernetes secrets are not encrypted!
    Kubernetes secrets are not encrypted! B
    R
    

    View Slide

28. Insecure docker socket service
    Injecting Vault Secrets Into K8S Pods via a Sidecar B
    https://www.hashicorp.com/blog/injecting-vault-secrets-into-kubernetes-pods-via-a-sidecar/
    # patch-basic-annotations.yaml
    spec:
    template:
    metadata:
    annotations:
    vault.hashicorp.com/agent-inject: "true"
    vault.hashicorp.com/agent-inject-secret-helloworld: "secrets/helloworld"
    vault.hashicorp.com/role: "myapp"
    

    View Slide

29. SSRF in the kubernetes world like a Cluster Pwn
    ● In the Google Cloud (GCP), we have to
    use Metadata-Flavor: Google to obtain
    the metadata
    ● Now GKE offers to protect kube-env
    using metadata concealment proxy
    and workload identity
    SSRF in the kubernetes world like a Cluster Pwn
    R
    

    View Slide

30. SSRF in the kubernetes world like a Cluster Pwn
    SSRF in the kubernetes world like a Cluster Pwn
    R
    

    View Slide

31. Insecure docker socket service
    Metadata Concealment B
    • Most of the cloud providers has fix for this in some way
    • GKE: Workload Identity, Metadata Concealment for Nodes
    https://cloud.google.com/kubernetes-engine/docs/how-to/workload-identity
    • AWS: IMDSv2 for SSRF
    https://aws.amazon.com/blogs/security/defense-in-depth-open-firewalls-reverse-proxies-ssrf-vulnerabilities-ec2-instance-
    metadata-service/
    https://github.com/genuinetools/bane
    

    View Slide

32. Command Injection to node access (host)
    Command Injection to node access (host)
    R
    

    View Slide

33. Command Injection to node access (host)
    Command Injection to node access (host)
    R
    

    View Slide

34. Command Injection to node access (host)
    Command Injection to node access (host)
    R
    

    View Slide

35. Kubernetes centralised logs in stack driver
    Runtime Security Detection – Sysdig Falco
    https://www.youtube.com/watch?v=zd0ksjZI5Vk
    https://falco.org/
    Falco, the open source cloud-native runtime security project, is the defacto Kubernetes threat detection engine.
    Falco detects unexpected application behaviour and alerts on threats at runtime.
    B
    

    View Slide

36. No default security boundary in k8s namespaces
    No default security boundary in k8s namespaces B
    R
    

    View Slide

37. Kubernetes centralised logs in stack driver
    Network Security Policies
    https://github.com/ahmetb/kubernetes-network-policy-recipes
    Provides isolation between Kubernetes resources (pods, namespaces, svc, etc.) using labels and
    selectors across the cluster.
    B
    

    View Slide

38. Default service account in a Pod
    Default service account in a Pod
    R
    

    View Slide

39. Default service account in a Pod
    https://hackernoon.com/capturing-all-the-flags-in-bsidessf-ctf-by-pwning-our-infrastructure-3570b99b4dd0
    Default service account in a Pod
    Default service account in a Pod
    https://hackernoon.com/capturing-all-the-flags-in-bsidessf-ctf-by-pwning-our-infrastructure-3570b99b4dd0
    Default service account in a Pod
    R
    

    View Slide

40. Default misconfigured Helm Tiller = Cluster Pwn
    https://engineering.bitnami.com/articles/helm-security.html
    Default misconfigured Helm Tiller = Cluster Pwn
    R
    

    View Slide

41. Default misconfigured Helm Tiller = Cluster Pwn
    Default misconfigured Helm Tiller = Cluster Pwn
    R
    

    View Slide

42. Kubernetes centralised logs in stack driver
    RBAC with least privilege access possible
    https://kubernetes.io/docs/reference/access-authn-authz/rbac/
    Role-based access control (RBAC) is a method of regulating access to computer or network resources
    based on the roles of individual users within your organization.
    B
    Useful utilities to check out is
    https://github.com/liggitt/audit2rbac
    https://github.com/FairwindsOps/rbac
    -manager
    https://github.com/jtblin/kube2iam
    

    View Slide

43. Trivy - Vulnerability Scanner for Containers
    https://github.com/aquasecurity/trivy
    Trivy - Vulnerability Scanner for Containers B
    R
    

    View Slide

44. dockle - Container Image Linter for Security
    https://github.com/goodwithtech/dockle
    dockle - Container Image Linter for Security B
    R
    

    View Slide

45. docker-bench-security
    https://github.com/docker/docker-bench-security
    ● A script that checks for dozens of
    common best-practices around
    deploying Docker containers in
    production
    ○ Host configuration
    ○ Docker daemon configuration and files
    ○ Docker container images
    ○ Docker runtime
    ○ Docker security operations
    ○ Docker swarm configuration
    docker-bench-security B
    R
    

    View Slide

46. kube-bench - CIS Kubernetes Benchmark
    https://github.com/aquasecurity/kube-bench
    ● Master Node Security Configuration
    ○ API Server
    ○ Scheduler
    ○ Controller Manager
    ○ Configuration Files
    ○ etcd
    ○ General Security Primitives
    ○ PodSecurityPolicices
    ● Worker Node Security Configuration
    ○ Kubelet
    ○ Configuration Files
    kube-bench - CIS Kubernetes Benchmark B
    R
    

    View Slide

47. kube-hunter
    ● Kube-hunter hunts for security
    weaknesses in Kubernetes clusters.
    The tool was developed to increase
    awareness and visibility for security
    issues in Kubernetes environments.
    You should NOT run kube-hunter on a
    Kubernetes cluster you don't own!
    kube-hunter B
    R
    

    View Slide

48. kubeaudit - Audit your kubernetes clusters
    https://github.com/Shopify/kubeaudit
    kubeaudit - Audit your kubernetes clusters B
    R
    

    View Slide

49. kubesec.io - Risk analysis for k8s resources
    https://kubesec.io/
    kubesec.io - Risk analysis for k8s resources B
    R
    

    View Slide

50. CVE-2018-1002105
    https://www.youtube.com/watch?v=4CTK2aUXTHo
    Known Vulnerabilities (CVE-2018-1002105) B
    R
    

    View Slide

51. https://www.youtube.com/watch?v=4CTK2aUXTHo
    CVE-2018-1002105
    Known Vulnerabilities (CVE-2018-1002105) B
    R
    

    View Slide

52. https://github.com/Frichetten/CVE-2019-5736-PoC
    ● This is a Go implementation
    of CVE-2019-5736, a
    container escape for Docker.
    The exploit works by
    overwriting and executing
    the host systems runc binary
    from within the container
    CVE-2019-5736
    Known Vulnerabilities (CVE-2019-5736) B
    R
    

    View Slide

53. https://github.com/eoftedal/writings/blob/master/published/CVE-2019-9901-path-traversal.md
    CVE-2019-9901 - Istio/Envoy Path traversal
    CVE-2019-9901 : Istio/Envoy Path traversal B
    R
    

    View Slide

54. docker logs and events
    docker logs and events B
    R
    

    View Slide

55. Kubernetes centralised logs in stack driver
    Centralized logging and Monitoring B
    

    View Slide

56. Kubernetes centralised logs in stack driver
    Open Policy Agent – Policy Engine
    https://www.openpolicyagent.org/
    Policy-based control for cloud native environments
    Flexible, fine-grained control for administrators across the stack
    B
    

    View Slide

57. Kubernetes centralised logs in stack driver
    Container Runtime Sandbox
    gVisor, Firecracker, etc.
    B
    • gVisor is a user-space kernel, written in Go, that implements a substantial
    portion of the Linux system call interface. It provides an additional layer
    of isolation between running applications and the host operating system.
    • Firecracker is an open source virtualization technology that is purpose-
    built for creating and managing secure, multi-tenant container and
    function-based services.
    • Many other…
    

    View Slide

58. Kubernetes centralised logs in stack driver
    TLS – Let’s Encrypt with cert-manager
    https://cert-manager.io/
    Automate certificate management in cloud native environments. cert-manager builds on top of
    Kubernetes, introducing certificate authorities and certificates as first-class resource types in the
    Kubernetes API. This makes it possible to provide 'certificates as a service' to developers working within
    your Kubernetes cluster.
    B
    

    View Slide

59. Kubernetes centralised logs in stack driver
    Pod Security Policies
    https://kubernetes.io/docs/concepts/policy/pod-security-policy
    A Pod Security Policy is a cluster-
    level resource that controls
    security sensitive aspects of the
    pod specification. The
    PodSecurityPolicy objects define
    a set of conditions that a pod
    must run with in order to be
    accepted into the system, as
    well as defaults for the related
    fields.
    Good utility to check out is
    https://github.com/sysdiglabs/k
    ube-psp-advisor
    B
    

    View Slide

60. References & Resources
    • Docker Security Docs
    • Kubernetes Security Docs
    • Attack matrix for Kubernetes
    • Breaking & Pwning Docker Containers & Kubernetes Clusters
    • Advanced Persistence Threats: The Future of Kubernetes Attacks
    • Kubernetes Security Resources
    • 11 Ways (Not) to Get Hacked
    • Attacking & Auditing Docker Containers using Open Source @ DEFCON 26
    • Attacking and Auditing Docker Containers and Kubernetes Clusters @ DEFCON 27
    @madhuakula
    B
    R
    

    View Slide

61. References & Resources
    • contained.af
    • CIS Benchmarks Docker
    • Understanding and Hardening Linux Containers
    • Abusing Privileged and Unprivileged Linux Containers
    • Container Security Notes
    • Linux Container Security
    • Docker Runtime Privileges and Capabilities
    • Apparmor Security Profiles on Docker
    • Seccomp Security Profiles on Docker
    • Docker Labs Capabilities
    • Practical SELinux and Containers
    • Containers and Operating systems morning paper gist
    • Kubernetes Network Policies
    • Kubernetes Webinar series
    @madhuakula
    B
    R
    

    View Slide

62. Are you looking for more and more tools?
    https://tools.cloudnativesecurity.info
    @madhuakula
    B
    R
    

    View Slide

63. Thank You!
    Madhu Akula
    @madhuakula
    https://madhuakula.com
    @madhuakula
    B
    R
    

    View Slide