Kubernetes Security Detection Engineering – Mapping Back To MITRE ATT&CK Matrix | HITB 2023 AMS

Kubernetes has become a de facto way of running containerized workloads from startups to enterprises and governments, however like most modern technology, it’s not mature, especially in regards to security. Given its nature of being immutable and things happening in a matter of seconds, it’s super hard to perform security detection and incident response.

In this talk we will be focusing on the MITRE ATT&CK matrix for Kubernetes with showcasing what things can go wrong in different phases of the running container workloads, then we map back to what we should observe, collect, analyze, monitor, alert, and respond. We will showcase all the possible mappings of the matrix to the detection engineering. We will also cover some interesting real-world examples of hacks, known vulnerabilities, and misconfiguration. We will also showcase how we simulate these attacks in a controlled environment using the Kubernetes Goat project.

Transcript

1. Kubernetes Security Detection Engineering - Mapping Back to MITRE ATT&CK Matrix HITB 2023 AMS Madhu Akula
2. About Me - Madhu Akula 👉 Pragmatic Security Leader, working on Cloud Native Infra, Security, and Startups 👉 Creator of Kubernetes Goat, Hacker Container, tools.tldr.run, many other OSS projects. 👉 Speaks & Trains at Black Hat (USA, EU, Asia), DEF CON, GitHub, USENIX, OWASP, All Day DevOps, SANS, DevSecCon, CNCF, c0c0n, Nullcon, null, many others around the globe. 👉 Author of Security Automation with Ansible2, OWASP KSTG, whitepapers, etc. 👉 Technical reviewer (multiple books) & Review board member of multiple conferences, organizations, communities, advisory, etc. 👉 Found security vulnerabilities in 200+ organizations and products including Google, Microsoft, AT&T, Adobe, eBay, WordPress, Ntop, Cloudflare, Yahoo, LocalBitcoins, etc. 👉 Certified Kubernetes(CKA/CKS), Offensive Security Certified Professional, etc. 👉 Never ending learner! @madhuakula https://madhuakula.com
3. What is Docker? https://kubernetes.io/docs/concepts/overview/what-is-kubernetes/#going-back-in-time @madhuakula
4. What is Docker? • Docker is an open source platform for building, deploying, and managing containerized applications • Docker became the de facto standard to build and share containerized apps - from desktop, to the cloud, even edge devices • Docker enables developers to easily pack, ship, and run any application as a lightweight, portable, self-sufficient container, which can run virtually anywhere https://docs.docker.com/get-started/overview/ @madhuakula
5. What is Kubernetes? Kubernetes is a portable, extensible, open-source platform for managing containerized workloads and services, that facilitates both declarative configuration and automation. It has a large, rapidly growing ecosystem. Kubernetes services, support, and tools are widely available. https://kubernetes.io/docs/concepts/overview/what-is-kubernetes/ @madhuakula
6. What is Kubernetes? @madhuakula
7. The illustrated children's guide to Kubernetes https://www.youtube.com/watch?v=3I9PkvZ80BQ @madhuakula
8. Why Kubernetes Security? @madhuakula Lack of knowledge in security teams
9. Rapidly growing Cloud Native Landscape ecosystem Why Kubernetes Security? @madhuakula
10. Technology Gap Adoption & Maturity Why Kubernetes Security? @madhuakula
11. Why Kubernetes Security? @madhuakula
12. MITRE ATT&CK for Kubernetes https://attack.mitre.orghttps//microsoft.github.io/Threat-Matrix-for-Kubernetes/ @madhuakula
13. Kubernetes Attack Path / Kill Chain @madhuakula
14. Practical MITRE ATT&CK for Kubernetes - Attack Path https://youtu.be/7nc78ZrvP4Y @madhuakula
15. Let’s map back Attack Path to Detection Engineering @madhuakula
16. Defense In Depth - Layered Approach Some of the very high level abstraction layers, each layer contains many ways how we can secure and defend against attackers. • Application Security • Supply Chain Security • Infrastructure Security • Runtime Security • Continuous Security @madhuakula
17. Why Layered Approach? https://github.com/cncf/financial-user-group/blob/master/projects/k8s-threat-model/AttackTrees/AttackerOnTheNetwork.md Attackers have many ways! Defenders have many layers! @madhuakula
18. @madhuakula 🔥 Let’s focus on detecting Container Escape / Privilege Escalation
19. There are many things we need to think as you seen it’s a layered approach! 👉 Here are some of the things you can think of looking for detection from various perspectives of an attacker Detection - Container Escape / Privilege Escalation @madhuakula 🛡 Standard Linux Server logging (SSH, System, Services, etc. /var/log/*) 🛡 Container/Pod logs, and Node level system logging 🛡 Components (Kubelet, Runtime, Middlewares, Entrypoints, etc.) 🛡 Admission Controller/Audit Logging from Kubernetes 🛡 Runtime Security logging (Things like Falco, Tetragon, etc. - SYSCALLs, PS, FIM, Net, Flow, etc. ) 🛡 [Near] Real-time Proactive Monitoring, Detection / Prevention
20. Demo Time 🤞 @madhuakula
21. What just happened! @madhuakula https://madhuakula.com/kubernetes-goat/docs/scenarios/scenario-21/ebpf-runtime-security-monitoring-and-detection-i n-kubernetes-cluster-using-cilium-tetragon/welcome
22. ✅ List of logs for Kubernetes Security Detection 👉 API Server logs: Kubernetes API server, including audit logs, requests and responses, and authentication logs. 👉 Kubernetes Audit Logs: Audit Kubernetes can provide insight into activities related to API server requests and resource changes. 👉 Controller Manager logs: Kubernetes Controller Manager, including events, component status, and leader election. 👉 etcd logs: Kubernetes etcd datastore, including request and response logs, snapshot creation, and cluster state changes. 👉 Kubelet logs: Kubernetes Kubelet, including container logs, node status, and pod events. 👉 kube-proxy logs: Kubernetes kube-proxy, including service proxying, health checks, and NAT operations. @madhuakula
23. ✅ List of logs for Kubernetes Security Detection 👉 Network policy logs: Kubernetes Network Policies, including rule matches and denied connections. 👉 Pod logs: Kubernetes pods, including container logs, application logs, and any error or warning messages. 👉 RBAC logs: Kubernetes Role-Based Access Control, including user authentication and authorization events. 👉 Scheduler logs: Kubernetes scheduler, including scheduling decisions, pod status updates, and failed scheduling attempts. 👉 Service Mesh Logs: Service mesh components like Istio or Linkerd can provide insight into network activity within the mesh, including service-to-service communication and security policies. @madhuakula
24. ✅ List of logs for Kubernetes Security Detection 👉 Container Runtime Logs: Container runtimes like Docker or containerd can provide insight into container activities such as process execution, file system operations, and network communication. 👉 Container Network Interface (CNI) Plugin Logs: CNI plugins can provide visibility into network activities, including network policies, network connectivity, and network segmentation. 👉 Host System Logs: The host system can provide insight into activities related to system-level events, including process creation, file system operations, and network communication. 👉 Application Logs: Applications running inside containers can provide insight into application-level activities, including user actions and application-level vulnerabilities. 👉 Many other context related Telemetry (Open Telemetry), Tracing data is a gold mine of information for the detection engineering to make the most out of the systems @madhuakula
25. How can we learn and practice to defend against MITRE ATT&CK ? @madhuakula
26. Kubernetes Goat is an interactive Kubernetes security learning playground. Intentionally vulnerable by design scenarios to showcase the common misconfigurations, real-world vulnerabilities, and security issues in Kubernetes clusters, containers, and cloud native environments. What is Kubernetes Goat 🐐 @madhuakula
27. ⚡ Get Started with Kubernetes Goat 🐐 https://madhuakula.com/kubernetes-goat @madhuakula
28. 🔥 Kubernetes Goat Audience 💥 Attackers & Red Teams 🛡 Defenders & Blue Teams 🧰 Products & Vendors 🔐 Developers & DevOps Teams 💡 Interested in Kubernetes Security @madhuakula
29. 🚀 Scenarios in Kubernetes Goat 1. Sensitive keys in codebases 2. DIND (docker-in-docker) exploitation 3. SSRF in the Kubernetes (K8S) world 4. Container escape to the host system 5. Docker CIS benchmarks analysis 6. Kubernetes CIS benchmarks analysis 7. Attacking private registry 8. NodePort exposed services 9. Helm v2 tiller to PwN the cluster - [Deprecated] 10. Analyzing crypto miner container 11. Kubernetes namespaces bypass 12. Gaining environment information 13. DoS the Memory/CPU resources 14. Hacker container preview 15. Hidden in layers 16. RBAC least privileges misconfiguration 17. KubeAudit - Audit Kubernetes clusters 18. Falco - Runtime security monitoring & detection 19. Popeye - A Kubernetes cluster sanitizer 20. Secure network boundaries using NSP 21. Cilium Tetragon - eBPF-based Security Observability and Runtime Enforcement 22. Securing Kubernetes Clusters using Kyverno Policy Engine More scenarios releasing soon… ❤ @madhuakula
30. 🧰 How can I setup Kubernetes Goat ☸ Vanilla Kubernetes Cluster ☁ AWS Kubernetes (EKS) ☁ GCP Kubernetes (GKE) ☁ Azure Kubernetes (AKS) ☸ Kubernetes IN Docker (KiND) ☸ Lightweight Kubernetes (K3S) ☸ Digital Ocean, Vagrant, Many others… @madhuakula
31. ⎈ Setting up in your Kubernetes Cluster • Make sure you have Kubernetes cluster with cluster-admin privileges. Also kubectl and helm installed in your system before running the following commands to setup the Kubernetes Goat • Now you can access the Kubernetes Goat by navigating to http://127.0.0.1:1234 @madhuakula
32. ⚡ Get Started with Kubernetes Goat 🐐 @madhuakula
33. ⚡ Get Started with Kubernetes Goat 🐐 @madhuakula
34. 🔟 OWASP Kubernetes Top 10 https://owasp.org/www-project-kubernetes-top-ten/ @madhuakula
35. 🛡 MITRE ATT&CK for Kubernetes Goat https://madhuakula.com/kubernetes-goat/docs/mitre/mitre-attack @madhuakula
36. 🥳 Adoption of Kubernetes Goat https://youtu.be/62_Cj6yseno?t=352 @madhuakula
37. Spread the ❤ #KubernetesGoat 🙌 Give it a try 🚀 Contribute ideas & suggestions 🤝 Work with the project & improve 🙏 Share your valuable feedback 🌟 Star in GitHub 🎉 Spread word #KubernetesGoat @madhuakula
38. Thank you! https://madhuakula.com @madhuakula