B R

About - Madhu Akula • Cloud Native Security Specialist @ Xebia • Security (Cloud, Containers, Kubernetes & Automation) • Speaker & Trainer @ BlackHat, DEF CON, USENIX LISA, OWASP Appsec EU, All Day DevOps, DevSecCon, c0c0n, Nullcon, null, etc. • Co-Author of Security Automation with Ansible2 book • Discovered vulnerabilities in over 200+ organizations including; Google, Microsoft, LinkedIn, Coudflare, AT&T, Wordpress, NTOP and Adobe, etc. • Holds industry certifications like OSCP and CKA • Never Ending Learner! • https://madhuakula.com @madhuakula B R

What you will learn in next 30 minutes! • This talk is NOT about what is Docker, Kubernetes, etc. • This talk is about • Why container security? • Red team overview of container security • Blue team overview of container security • Tools, Techniques and Procedures (TTP’s) • Real-world scenarios • Approach to attackers and defenders • Case studies and reference resources • Next steps for learning more and more @madhuakula B R

Would you like to learn Docker & Kubernetes? • https://docs.docker.com • https://kubernetes.io/docs/home • https://container.training • https://training.play-with-docker.com • https://labs.play-with-k8s.com • https://training.play-with-kubernetes.com • https://www.katacoda.com/learn • Many more... @madhuakula B R

Why container security? @madhuakula https://kromtech.com/blog/security-center/cryptojacking-invades-cloud-how-modern-containerization-trend-is-exploited-by-attackers

Why container security? @madhuakula https://blog.madhuakula.com/some-tips-to-review-docker-hub-hack-of-190k-accounts-addcd602aade

Why container security? @madhuakula https://hackerone.com/reports/341876

Why container security? Many other vulnerabilities and real-world impacts... @madhuakula

Container Security Overview • Container Attack Surface • Namespaces • Control Groups • Daemon • Configuration • Capabilities • Content Trust • Container Registry • Volumes • Networks • Many other... @madhuakula B R

Kubernetes Security Overview @madhuakula https://www.microsoft.com/security/blog/2020/04/02/attack-matrix-kubernetes/ B R

amicontained - Container Introspection Tool https://github.com/genuinetools/amicontained It helps to find out what container runtime is being used as well as features available like capabilities, profiles applied, etc. amicontained - Container Introspection Tool B R

• Committing the sensitive information to version control systems • Not including the sensitive files in the build process using .dockerignore file • This is one of the common mistake in modern era trufflehog - Hardcoded sensitive information B R

Insecurely configured docker service R

Insecure docker socket service Insecure docker socket service R

Insecure docker socket service Securing Daemon Configuration B • By default, Docker runs through a non-networked UNIX socket. It can also optionally communicate using an HTTP socket. • If you need Docker to be reachable through the network in a safe manner, you can enable TLS by specifying the tlsverify flag and pointing Docker’s tlscacert flag to a trusted CA certificate. • If you want to secure your Docker client connections by default, you can move the files to the .docker directory in your home directory --- and set the DOCKER_HOST and DOCKER_TLS_VERIFY variables as well (instead of passing -H=tcp://$HOST:2376 and --tlsverify on every call). https://docs.docker.com/engine/security/https

Analysing or Understanding unknown image Analyzing or Understanding unknown image B R

Insecure docker socket service Content Trust B • Docker Content Trust (DCT) provides the ability to use digital signatures for data sent to and received from remote Docker registries. These signatures allow client-side or runtime verification of the integrity and publisher of specific image tags • We can sign and push a container image with the docker trust command • Content trust is disabled by default in the Docker Client. To enable it, set the DOCKER_CONTENT_TRUST environment variable to 1. This prevents users from working with tagged images unless they contain a signature. https://docs.docker.com/engine/security/trust/content_trust/

dive - Exploring each layer in a docker image https://github.com/wagoodman/dive dive - Exploring each layer in a docker image B R

Inspecting container volumes Inspecting container volumes B R

Volume analysis for sensitive information Volume analysis for sensitive information R

Inspecting container networking Inspecting container networking B R

Always look for env variables ● This is one of the common places most developers and operations teams store secrets, API keys, etc. ● Also it contains other information like different service or cluster related information Always look for env variables B R

docker diff - comparing with base image docker diff - comparing with base image B R

container escape - extra capability and host pid container escape - extra capability and host pid R

container escape - extra capability and host pid container escape - extra capability and host pid R

Insecure docker socket service Security Profiles B • The Linux Security Module (LSM) framework provides a mechanism for various security checks to be hooked by new kernel extensions • AppArmor profile generator for docker containers using bane https://github.com/genuinetools/bane

Kubernetes secrets are not encrypted! Kubernetes secrets are not encrypted! B R

Insecure docker socket service Injecting Vault Secrets Into K8S Pods via a Sidecar B https://www.hashicorp.com/blog/injecting-vault-secrets-into-kubernetes-pods-via-a-sidecar/ # patch-basic-annotations.yaml spec: template: metadata: annotations: vault.hashicorp.com/agent-inject: "true" vault.hashicorp.com/agent-inject-secret-helloworld: "secrets/helloworld" vault.hashicorp.com/role: "myapp"

SSRF in the kubernetes world like a Cluster Pwn ● In the Google Cloud (GCP), we have to use Metadata-Flavor: Google to obtain the metadata ● Now GKE offers to protect kube-env using metadata concealment proxy and workload identity SSRF in the kubernetes world like a Cluster Pwn R

SSRF in the kubernetes world like a Cluster Pwn SSRF in the kubernetes world like a Cluster Pwn R

Insecure docker socket service Metadata Concealment B • Most of the cloud providers has fix for this in some way • GKE: Workload Identity, Metadata Concealment for Nodes https://cloud.google.com/kubernetes-engine/docs/how-to/workload-identity • AWS: IMDSv2 for SSRF https://aws.amazon.com/blogs/security/defense-in-depth-open-firewalls-reverse-proxies-ssrf-vulnerabilities-ec2-instance- metadata-service/ https://github.com/genuinetools/bane

Command Injection to node access (host) Command Injection to node access (host) R

Command Injection to node access (host) Command Injection to node access (host) R

Command Injection to node access (host) Command Injection to node access (host) R

Kubernetes centralised logs in stack driver Runtime Security Detection – Sysdig Falco https://www.youtube.com/watch?v=zd0ksjZI5Vk https://falco.org/ Falco, the open source cloud-native runtime security project, is the defacto Kubernetes threat detection engine. Falco detects unexpected application behaviour and alerts on threats at runtime. B

No default security boundary in k8s namespaces No default security boundary in k8s namespaces B R

Kubernetes centralised logs in stack driver Network Security Policies https://github.com/ahmetb/kubernetes-network-policy-recipes Provides isolation between Kubernetes resources (pods, namespaces, svc, etc.) using labels and selectors across the cluster. B

Default service account in a Pod Default service account in a Pod R

Default service account in a Pod https://hackernoon.com/capturing-all-the-flags-in-bsidessf-ctf-by-pwning-our-infrastructure-3570b99b4dd0 Default service account in a Pod Default service account in a Pod https://hackernoon.com/capturing-all-the-flags-in-bsidessf-ctf-by-pwning-our-infrastructure-3570b99b4dd0 Default service account in a Pod R

Default misconfigured Helm Tiller = Cluster Pwn https://engineering.bitnami.com/articles/helm-security.html Default misconfigured Helm Tiller = Cluster Pwn R

Default misconfigured Helm Tiller = Cluster Pwn Default misconfigured Helm Tiller = Cluster Pwn R

Kubernetes centralised logs in stack driver RBAC with least privilege access possible https://kubernetes.io/docs/reference/access-authn-authz/rbac/ Role-based access control (RBAC) is a method of regulating access to computer or network resources based on the roles of individual users within your organization. B Useful utilities to check out is https://github.com/liggitt/audit2rbac https://github.com/FairwindsOps/rbac -manager https://github.com/jtblin/kube2iam

Trivy - Vulnerability Scanner for Containers https://github.com/aquasecurity/trivy Trivy - Vulnerability Scanner for Containers B R

dockle - Container Image Linter for Security https://github.com/goodwithtech/dockle dockle - Container Image Linter for Security B R

docker-bench-security https://github.com/docker/docker-bench-security ● A script that checks for dozens of common best-practices around deploying Docker containers in production ○ Host configuration ○ Docker daemon configuration and files ○ Docker container images ○ Docker runtime ○ Docker security operations ○ Docker swarm configuration docker-bench-security B R

kube-bench - CIS Kubernetes Benchmark https://github.com/aquasecurity/kube-bench ● Master Node Security Configuration ○ API Server ○ Scheduler ○ Controller Manager ○ Configuration Files ○ etcd ○ General Security Primitives ○ PodSecurityPolicices ● Worker Node Security Configuration ○ Kubelet ○ Configuration Files kube-bench - CIS Kubernetes Benchmark B R

kube-hunter ● Kube-hunter hunts for security weaknesses in Kubernetes clusters. The tool was developed to increase awareness and visibility for security issues in Kubernetes environments. You should NOT run kube-hunter on a Kubernetes cluster you don't own! kube-hunter B R

kubeaudit - Audit your kubernetes clusters https://github.com/Shopify/kubeaudit kubeaudit - Audit your kubernetes clusters B R

kubesec.io - Risk analysis for k8s resources https://kubesec.io/ kubesec.io - Risk analysis for k8s resources B R

CVE-2018-1002105 https://www.youtube.com/watch?v=4CTK2aUXTHo Known Vulnerabilities (CVE-2018-1002105) B R

https://www.youtube.com/watch?v=4CTK2aUXTHo CVE-2018-1002105 Known Vulnerabilities (CVE-2018-1002105) B R

https://github.com/Frichetten/CVE-2019-5736-PoC ● This is a Go implementation of CVE-2019-5736, a container escape for Docker. The exploit works by overwriting and executing the host systems runc binary from within the container CVE-2019-5736 Known Vulnerabilities (CVE-2019-5736) B R

https://github.com/eoftedal/writings/blob/master/published/CVE-2019-9901-path-traversal.md CVE-2019-9901 - Istio/Envoy Path traversal CVE-2019-9901 : Istio/Envoy Path traversal B R

docker logs and events docker logs and events B R

Kubernetes centralised logs in stack driver Centralized logging and Monitoring B

Kubernetes centralised logs in stack driver Open Policy Agent – Policy Engine https://www.openpolicyagent.org/ Policy-based control for cloud native environments Flexible, fine-grained control for administrators across the stack B

Kubernetes centralised logs in stack driver Container Runtime Sandbox gVisor, Firecracker, etc. B • gVisor is a user-space kernel, written in Go, that implements a substantial portion of the Linux system call interface. It provides an additional layer of isolation between running applications and the host operating system. • Firecracker is an open source virtualization technology that is purpose- built for creating and managing secure, multi-tenant container and function-based services. • Many other…

Kubernetes centralised logs in stack driver TLS – Let’s Encrypt with cert-manager https://cert-manager.io/ Automate certificate management in cloud native environments. cert-manager builds on top of Kubernetes, introducing certificate authorities and certificates as first-class resource types in the Kubernetes API. This makes it possible to provide 'certificates as a service' to developers working within your Kubernetes cluster. B

Kubernetes centralised logs in stack driver Pod Security Policies https://kubernetes.io/docs/concepts/policy/pod-security-policy A Pod Security Policy is a cluster- level resource that controls security sensitive aspects of the pod specification. The PodSecurityPolicy objects define a set of conditions that a pod must run with in order to be accepted into the system, as well as defaults for the related fields. Good utility to check out is https://github.com/sysdiglabs/k ube-psp-advisor B

References & Resources • Docker Security Docs • Kubernetes Security Docs • Attack matrix for Kubernetes • Breaking & Pwning Docker Containers & Kubernetes Clusters • Advanced Persistence Threats: The Future of Kubernetes Attacks • Kubernetes Security Resources • 11 Ways (Not) to Get Hacked • Attacking & Auditing Docker Containers using Open Source @ DEFCON 26 • Attacking and Auditing Docker Containers and Kubernetes Clusters @ DEFCON 27 @madhuakula B R

References & Resources • contained.af • CIS Benchmarks Docker • Understanding and Hardening Linux Containers • Abusing Privileged and Unprivileged Linux Containers • Container Security Notes • Linux Container Security • Docker Runtime Privileges and Capabilities • Apparmor Security Profiles on Docker • Seccomp Security Profiles on Docker • Docker Labs Capabilities • Practical SELinux and Containers • Containers and Operating systems morning paper gist • Kubernetes Network Policies • Kubernetes Webinar series @madhuakula B R

Are you looking for more and more tools? https://tools.cloudnativesecurity.info @madhuakula B R

Thank You! Madhu Akula @madhuakula https://madhuakula.com @madhuakula B R