Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Kubernetes Security Detection Engineering – Mapping Back To MITRE ATT&CK Matrix | HITB 2023 AMS

Kubernetes Security Detection Engineering – Mapping Back To MITRE ATT&CK Matrix | HITB 2023 AMS

Kubernetes has become a de facto way of running containerized workloads from startups to enterprises and governments, however like most modern technology, it’s not mature, especially in regards to security. Given its nature of being immutable and things happening in a matter of seconds, it’s super hard to perform security detection and incident response.

In this talk we will be focusing on the MITRE ATT&CK matrix for Kubernetes with showcasing what things can go wrong in different phases of the running container workloads, then we map back to what we should observe, collect, analyze, monitor, alert, and respond. We will showcase all the possible mappings of the matrix to the detection engineering. We will also cover some interesting real-world examples of hacks, known vulnerabilities, and misconfiguration. We will also showcase how we simulate these attacks in a controlled environment using the Kubernetes Goat project.

Madhu Akula

April 21, 2023

More Decks by Madhu Akula

Other Decks in Technology


  1. About Me - Madhu Akula 👉 Pragmatic Security Leader, working

    on Cloud Native Infra, Security, and Startups 👉 Creator of Kubernetes Goat, Hacker Container, tools.tldr.run, many other OSS projects. 👉 Speaks & Trains at Black Hat (USA, EU, Asia), DEF CON, GitHub, USENIX, OWASP, All Day DevOps, SANS, DevSecCon, CNCF, c0c0n, Nullcon, null, many others around the globe. 👉 Author of Security Automation with Ansible2, OWASP KSTG, whitepapers, etc. 👉 Technical reviewer (multiple books) & Review board member of multiple conferences, organizations, communities, advisory, etc. 👉 Found security vulnerabilities in 200+ organizations and products including Google, Microsoft, AT&T, Adobe, eBay, WordPress, Ntop, Cloudflare, Yahoo, LocalBitcoins, etc. 👉 Certified Kubernetes(CKA/CKS), Offensive Security Certified Professional, etc. 👉 Never ending learner! @madhuakula https://madhuakula.com
  2. What is Docker? • Docker is an open source platform

    for building, deploying, and managing containerized applications • Docker became the de facto standard to build and share containerized apps - from desktop, to the cloud, even edge devices • Docker enables developers to easily pack, ship, and run any application as a lightweight, portable, self-sufficient container, which can run virtually anywhere https://docs.docker.com/get-started/overview/ @madhuakula
  3. What is Kubernetes? Kubernetes is a portable, extensible, open-source platform

    for managing containerized workloads and services, that facilitates both declarative configuration and automation. It has a large, rapidly growing ecosystem. Kubernetes services, support, and tools are widely available. https://kubernetes.io/docs/concepts/overview/what-is-kubernetes/ @madhuakula
  4. Defense In Depth - Layered Approach Some of the very

    high level abstraction layers, each layer contains many ways how we can secure and defend against attackers. • Application Security • Supply Chain Security • Infrastructure Security • Runtime Security • Continuous Security @madhuakula
  5. There are many things we need to think as you

    seen it’s a layered approach! 👉 Here are some of the things you can think of looking for detection from various perspectives of an attacker Detection - Container Escape / Privilege Escalation @madhuakula 🛡 Standard Linux Server logging (SSH, System, Services, etc. /var/log/*) 🛡 Container/Pod logs, and Node level system logging 🛡 Components (Kubelet, Runtime, Middlewares, Entrypoints, etc.) 🛡 Admission Controller/Audit Logging from Kubernetes 🛡 Runtime Security logging (Things like Falco, Tetragon, etc. - SYSCALLs, PS, FIM, Net, Flow, etc. ) 🛡 [Near] Real-time Proactive Monitoring, Detection / Prevention
  6. ✅ List of logs for Kubernetes Security Detection 👉 API

    Server logs: Kubernetes API server, including audit logs, requests and responses, and authentication logs. 👉 Kubernetes Audit Logs: Audit Kubernetes can provide insight into activities related to API server requests and resource changes. 👉 Controller Manager logs: Kubernetes Controller Manager, including events, component status, and leader election. 👉 etcd logs: Kubernetes etcd datastore, including request and response logs, snapshot creation, and cluster state changes. 👉 Kubelet logs: Kubernetes Kubelet, including container logs, node status, and pod events. 👉 kube-proxy logs: Kubernetes kube-proxy, including service proxying, health checks, and NAT operations. @madhuakula
  7. ✅ List of logs for Kubernetes Security Detection 👉 Network

    policy logs: Kubernetes Network Policies, including rule matches and denied connections. 👉 Pod logs: Kubernetes pods, including container logs, application logs, and any error or warning messages. 👉 RBAC logs: Kubernetes Role-Based Access Control, including user authentication and authorization events. 👉 Scheduler logs: Kubernetes scheduler, including scheduling decisions, pod status updates, and failed scheduling attempts. 👉 Service Mesh Logs: Service mesh components like Istio or Linkerd can provide insight into network activity within the mesh, including service-to-service communication and security policies. @madhuakula
  8. ✅ List of logs for Kubernetes Security Detection 👉 Container

    Runtime Logs: Container runtimes like Docker or containerd can provide insight into container activities such as process execution, file system operations, and network communication. 👉 Container Network Interface (CNI) Plugin Logs: CNI plugins can provide visibility into network activities, including network policies, network connectivity, and network segmentation. 👉 Host System Logs: The host system can provide insight into activities related to system-level events, including process creation, file system operations, and network communication. 👉 Application Logs: Applications running inside containers can provide insight into application-level activities, including user actions and application-level vulnerabilities. 👉 Many other context related Telemetry (Open Telemetry), Tracing data is a gold mine of information for the detection engineering to make the most out of the systems @madhuakula
  9. Kubernetes Goat is an interactive Kubernetes security learning playground. Intentionally

    vulnerable by design scenarios to showcase the common misconfigurations, real-world vulnerabilities, and security issues in Kubernetes clusters, containers, and cloud native environments. What is Kubernetes Goat 🐐 @madhuakula
  10. 🔥 Kubernetes Goat Audience 💥 Attackers & Red Teams 🛡

    Defenders & Blue Teams 🧰 Products & Vendors 🔐 Developers & DevOps Teams 💡 Interested in Kubernetes Security @madhuakula
  11. 🚀 Scenarios in Kubernetes Goat 1. Sensitive keys in codebases

    2. DIND (docker-in-docker) exploitation 3. SSRF in the Kubernetes (K8S) world 4. Container escape to the host system 5. Docker CIS benchmarks analysis 6. Kubernetes CIS benchmarks analysis 7. Attacking private registry 8. NodePort exposed services 9. Helm v2 tiller to PwN the cluster - [Deprecated] 10. Analyzing crypto miner container 11. Kubernetes namespaces bypass 12. Gaining environment information 13. DoS the Memory/CPU resources 14. Hacker container preview 15. Hidden in layers 16. RBAC least privileges misconfiguration 17. KubeAudit - Audit Kubernetes clusters 18. Falco - Runtime security monitoring & detection 19. Popeye - A Kubernetes cluster sanitizer 20. Secure network boundaries using NSP 21. Cilium Tetragon - eBPF-based Security Observability and Runtime Enforcement 22. Securing Kubernetes Clusters using Kyverno Policy Engine More scenarios releasing soon… ❤ @madhuakula
  12. 🧰 How can I setup Kubernetes Goat ☸ Vanilla Kubernetes

    Cluster ☁ AWS Kubernetes (EKS) ☁ GCP Kubernetes (GKE) ☁ Azure Kubernetes (AKS) ☸ Kubernetes IN Docker (KiND) ☸ Lightweight Kubernetes (K3S) ☸ Digital Ocean, Vagrant, Many others… @madhuakula
  13. ⎈ Setting up in your Kubernetes Cluster • Make sure

    you have Kubernetes cluster with cluster-admin privileges. Also kubectl and helm installed in your system before running the following commands to setup the Kubernetes Goat • Now you can access the Kubernetes Goat by navigating to @madhuakula
  14. Spread the ❤ #KubernetesGoat 🙌 Give it a try 🚀

    Contribute ideas & suggestions 🤝 Work with the project & improve 🙏 Share your valuable feedback 🌟 Star in GitHub 🎉 Spread word #KubernetesGoat @madhuakula