Kubernetes Security Detection Engineering - Mapping Back to MITRE ATT&CK Matrix HITB 2023 AMS Madhu Akula

About Me - Madhu Akula 👉 Pragmatic Security Leader, working on Cloud Native Infra, Security, and Startups 👉 Creator of Kubernetes Goat, Hacker Container, tools.tldr.run, many other OSS projects. 👉 Speaks & Trains at Black Hat (USA, EU, Asia), DEF CON, GitHub, USENIX, OWASP, All Day DevOps, SANS, DevSecCon, CNCF, c0c0n, Nullcon, null, many others around the globe. 👉 Author of Security Automation with Ansible2, OWASP KSTG, whitepapers, etc. 👉 Technical reviewer (multiple books) & Review board member of multiple conferences, organizations, communities, advisory, etc. 👉 Found security vulnerabilities in 200+ organizations and products including Google, Microsoft, AT&T, Adobe, eBay, WordPress, Ntop, Cloudflare, Yahoo, LocalBitcoins, etc. 👉 Certified Kubernetes(CKA/CKS), Offensive Security Certified Professional, etc. 👉 Never ending learner! @madhuakula https://madhuakula.com

What is Docker? https://kubernetes.io/docs/concepts/overview/what-is-kubernetes/#going-back-in-time @madhuakula

What is Docker? ● Docker is an open source platform for building, deploying, and managing containerized applications ● Docker became the de facto standard to build and share containerized apps - from desktop, to the cloud, even edge devices ● Docker enables developers to easily pack, ship, and run any application as a lightweight, portable, self-sufficient container, which can run virtually anywhere https://docs.docker.com/get-started/overview/ @madhuakula

What is Kubernetes? Kubernetes is a portable, extensible, open-source platform for managing containerized workloads and services, that facilitates both declarative configuration and automation. It has a large, rapidly growing ecosystem. Kubernetes services, support, and tools are widely available. https://kubernetes.io/docs/concepts/overview/what-is-kubernetes/ @madhuakula

What is Kubernetes? @madhuakula

The illustrated children's guide to Kubernetes https://www.youtube.com/watch?v=3I9PkvZ80BQ @madhuakula

Why Kubernetes Security? @madhuakula Lack of knowledge in security teams

Rapidly growing Cloud Native Landscape ecosystem Why Kubernetes Security? @madhuakula

Technology Gap Adoption & Maturity Why Kubernetes Security? @madhuakula

Why Kubernetes Security? @madhuakula

MITRE ATT&CK for Kubernetes https://attack.mitre.orghttps//microsoft.github.io/Threat-Matrix-for-Kubernetes/ @madhuakula

Kubernetes Attack Path / Kill Chain @madhuakula

Practical MITRE ATT&CK for Kubernetes - Attack Path https://youtu.be/7nc78ZrvP4Y @madhuakula

Let’s map back Attack Path to Detection Engineering @madhuakula

Defense In Depth - Layered Approach Some of the very high level abstraction layers, each layer contains many ways how we can secure and defend against attackers. ● Application Security ● Supply Chain Security ● Infrastructure Security ● Runtime Security ● Continuous Security @madhuakula

Why Layered Approach? https://github.com/cncf/financial-user-group/blob/master/projects/k8s-threat-model/AttackTrees/AttackerOnTheNetwork.md Attackers have many ways! Defenders have many layers! @madhuakula

@madhuakula 🔥 Let’s focus on detecting Container Escape / Privilege Escalation

There are many things we need to think as you seen it’s a layered approach! 👉 Here are some of the things you can think of looking for detection from various perspectives of an attacker Detection - Container Escape / Privilege Escalation @madhuakula 🛡 Standard Linux Server logging (SSH, System, Services, etc. /var/log/*) 🛡 Container/Pod logs, and Node level system logging 🛡 Components (Kubelet, Runtime, Middlewares, Entrypoints, etc.) 🛡 Admission Controller/Audit Logging from Kubernetes 🛡 Runtime Security logging (Things like Falco, Tetragon, etc. - SYSCALLs, PS, FIM, Net, Flow, etc. ) 🛡 [Near] Real-time Proactive Monitoring, Detection / Prevention

Demo Time 🤞 @madhuakula

What just happened! @madhuakula https://madhuakula.com/kubernetes-goat/docs/scenarios/scenario-21/ebpf-runtime-security-monitoring-and-detection-i n-kubernetes-cluster-using-cilium-tetragon/welcome

✅ List of logs for Kubernetes Security Detection 👉 API Server logs: Kubernetes API server, including audit logs, requests and responses, and authentication logs. 👉 Kubernetes Audit Logs: Audit Kubernetes can provide insight into activities related to API server requests and resource changes. 👉 Controller Manager logs: Kubernetes Controller Manager, including events, component status, and leader election. 👉 etcd logs: Kubernetes etcd datastore, including request and response logs, snapshot creation, and cluster state changes. 👉 Kubelet logs: Kubernetes Kubelet, including container logs, node status, and pod events. 👉 kube-proxy logs: Kubernetes kube-proxy, including service proxying, health checks, and NAT operations. @madhuakula

✅ List of logs for Kubernetes Security Detection 👉 Network policy logs: Kubernetes Network Policies, including rule matches and denied connections. 👉 Pod logs: Kubernetes pods, including container logs, application logs, and any error or warning messages. 👉 RBAC logs: Kubernetes Role-Based Access Control, including user authentication and authorization events. 👉 Scheduler logs: Kubernetes scheduler, including scheduling decisions, pod status updates, and failed scheduling attempts. 👉 Service Mesh Logs: Service mesh components like Istio or Linkerd can provide insight into network activity within the mesh, including service-to-service communication and security policies. @madhuakula

✅ List of logs for Kubernetes Security Detection 👉 Container Runtime Logs: Container runtimes like Docker or containerd can provide insight into container activities such as process execution, file system operations, and network communication. 👉 Container Network Interface (CNI) Plugin Logs: CNI plugins can provide visibility into network activities, including network policies, network connectivity, and network segmentation. 👉 Host System Logs: The host system can provide insight into activities related to system-level events, including process creation, file system operations, and network communication. 👉 Application Logs: Applications running inside containers can provide insight into application-level activities, including user actions and application-level vulnerabilities. 👉 Many other context related Telemetry (Open Telemetry), Tracing data is a gold mine of information for the detection engineering to make the most out of the systems @madhuakula

How can we learn and practice to defend against MITRE ATT&CK ? @madhuakula

Kubernetes Goat is an interactive Kubernetes security learning playground. Intentionally vulnerable by design scenarios to showcase the common misconfigurations, real-world vulnerabilities, and security issues in Kubernetes clusters, containers, and cloud native environments. What is Kubernetes Goat 🐐 @madhuakula

⚡ Get Started with Kubernetes Goat 🐐 https://madhuakula.com/kubernetes-goat @madhuakula

🔥 Kubernetes Goat Audience 💥 Attackers & Red Teams 🛡 Defenders & Blue Teams 🧰 Products & Vendors 🔐 Developers & DevOps Teams 💡 Interested in Kubernetes Security @madhuakula

🚀 Scenarios in Kubernetes Goat 1. Sensitive keys in codebases 2. DIND (docker-in-docker) exploitation 3. SSRF in the Kubernetes (K8S) world 4. Container escape to the host system 5. Docker CIS benchmarks analysis 6. Kubernetes CIS benchmarks analysis 7. Attacking private registry 8. NodePort exposed services 9. Helm v2 tiller to PwN the cluster - [Deprecated] 10. Analyzing crypto miner container 11. Kubernetes namespaces bypass 12. Gaining environment information 13. DoS the Memory/CPU resources 14. Hacker container preview 15. Hidden in layers 16. RBAC least privileges misconfiguration 17. KubeAudit - Audit Kubernetes clusters 18. Falco - Runtime security monitoring & detection 19. Popeye - A Kubernetes cluster sanitizer 20. Secure network boundaries using NSP 21. Cilium Tetragon - eBPF-based Security Observability and Runtime Enforcement 22. Securing Kubernetes Clusters using Kyverno Policy Engine More scenarios releasing soon… ❤ @madhuakula

🧰 How can I setup Kubernetes Goat ☸ Vanilla Kubernetes Cluster ☁ AWS Kubernetes (EKS) ☁ GCP Kubernetes (GKE) ☁ Azure Kubernetes (AKS) ☸ Kubernetes IN Docker (KiND) ☸ Lightweight Kubernetes (K3S) ☸ Digital Ocean, Vagrant, Many others… @madhuakula

⎈ Setting up in your Kubernetes Cluster ● Make sure you have Kubernetes cluster with cluster-admin privileges. Also kubectl and helm installed in your system before running the following commands to setup the Kubernetes Goat ● Now you can access the Kubernetes Goat by navigating to http://127.0.0.1:1234 @madhuakula

⚡ Get Started with Kubernetes Goat 🐐 @madhuakula

⚡ Get Started with Kubernetes Goat 🐐 @madhuakula

🔟 OWASP Kubernetes Top 10 https://owasp.org/www-project-kubernetes-top-ten/ @madhuakula

🛡 MITRE ATT&CK for Kubernetes Goat https://madhuakula.com/kubernetes-goat/docs/mitre/mitre-attack @madhuakula

🥳 Adoption of Kubernetes Goat https://youtu.be/62_Cj6yseno?t=352 @madhuakula

Spread the ❤ #KubernetesGoat 🙌 Give it a try 🚀 Contribute ideas & suggestions 🤝 Work with the project & improve 🙏 Share your valuable feedback 🌟 Star in GitHub 🎉 Spread word #KubernetesGoat @madhuakula

Thank you! https://madhuakula.com @madhuakula