Kubernetes 101 for Penetration Testers - null Mumbai
This hands-on (demo) driven talk intends to get application / network security analysts get started with Kubernetes Cluster Penetration Testing. The slides are meant to be used for hands-on learning using locally setup cluster.
How to participate? ● Observe what I am doing during the session ● DO NOT do hands-on during the session - 1 hour is too less ● Use the slides as a reference to try out hands-on after the session ○ Slides are built specifically as a reference material ● Use additional reference material provided for further learning ● Ping me for doubts & questions (@abh1sek on Twitter)
What I am expecting from you (audience)? ● Curious and willing to learn new things ● Familiar with Linux err… I mean GNU/Linux ● Familiar with network or application security ● Familiar with basic vulnerability and exploit terminology ● Familiar with vulnerability assessment & penetration testing
What is a Container? Containers are a technology for packaging the (compiled) code for an application along with the dependencies it needs at run time. Each container that you run is repeatable; the standardization from having dependencies included means that you get the same behavior wherever you run it. Think of container as “Code + Config + Runtime” packaged in an archive stored locally or in a Git like remote repository, called Container Registry
What is Kubernetes? Kubernetes is a portable, extensible, open-source platform for managing containerized workloads and services, that facilitates both declarative configuration and automation. A container orchestrator really - Refer to Illustrated Children’s Guide to Kubernetes :) https://kubernetes.io/docs/concepts/overview/what-is-kubernetes/
Setup a local playground for learning Kubernetes minikube start --driver=docker -n 3 \ --enable-default-cni=false --network-plugin=cni kubectl cluster-info kubectl get nodes -o wide ❗ Multi-node clusters are currently experimental and might exhibit unintended behavior. To track progress on multi-node clusters, see https://github.com/kubernetes/minikube/issues/7538. Try out the online playground at Katacoda https://www.katacoda.com/courses/kubernetes/playground
Pods Pods are the smallest deployable units of computing that can be created and managed in Kubernetes kubectl run --restart=Never nginx-1 --image nginx kubectl get pods -o wide
Services An abstract way to expose an application running on a set of Pods as a network service. There are multiple service types such as ClusterIP, NodePort, LoadBalancer kubectl expose pod nginx-1 \ --port=8888 --target-port=80 --type=NodePort export NODE_PORT=$(kubectl get svc nginx-1 -o jsonpath='{.spec.ports[0].nodePort}') curl http://:$NODE_PORT/ NodePort may be risky to use
A Simple Threat Model Detailed Threat Model available from CNCF/TOB https://github.com/kubernetes/community/tree/master/wg-security-audit How can they attack? 03 ● Leverage configuration weaknesses ● Exploit vulnerabilities ● Exploit trust across components ● Lack of appropriate AuthZ controls ● Lack of security hardening of the cluster What can they attack? 02 ● Cluster state storage (etcd) ● Secrets ● Volumes (Data Breach) ● Container Image (Private Repository) ● Compute Resources (Example: Crypto Mining) Who are the attackers? 01 ● External (From internet) ● Internal (Attacker in a Pod) ● Developer (User with some access in the cluster) ● Malicious Administrator ● End User
Typical Attacker’s Workflow against a Kubernetes Cluster 1. Discovery (Recon) 2. Vulnerability Testing a. You must do a conventional VA/PT for the infrastructure (OS) running Master and Node components in additional to Kubernete specific testing 3. Exploitation a. Privilege Escalation b. Lateral Movement 4. Persistence
(External or Internal Attacker) API Server AuthZ Testing curl -sk https://$API_SERVER_ENDPOINT/api/v1/namespaces curl -sk https://$API_SERVER_ENDPOINT/api/v1/namespaces/default/pods kubectl auth can-i list namespaces kubectl auth can-i list pods kubectl auth can-i create pod Testing with unprivileged credential like Pod default service account
(External Attacker) What if etcd is exposed? etcd is exposed in Minikube cluster (as it should be) docker run -it --rm \ --network host \ --env ETCDCTL_API=3 \ --env ALLOW_NONE_AUTHENTICATION=yes \ bitnami/etcd:latest -- \ etcdctl --endpoints https://$ETCD_IP:2379 get / Should fail as client-cert auth is enabled by default, but you may be lucky :)
(Attacker in a Pod) Discovery (Recon) kubectl run -it attacker \ --image appsecco/k8s-security-tools \ -- bash Simulating an attacker in a Pod with required security tools printenv ifconfig host -v kubernetes.default kubectl auth can-i create pod ls -al /var/run/secrets/kubernetes.io/serviceaccount/
(Attacker in a Pod) Cluster Networking Kubernetes Networking Model https://kubernetes.io/docs/concepts/cluster-administration/networking/ Pods on a node can communicate with all pods on all nodes without NAT using the Pod Network i.e. anyone can talk to anyone by default Service Discovery through DNS assigns unique IP address to services in a dedicated Service Network CIDR All of this is facilitated by the CNI Plugin ifconfig ping kubernetes
Running a Vulnerability Scan docker run --rm -it \ appsecco/k8s-security-tools \ kube-hunter As external attacker to scan Master IP(s) for known issues kube-hunter --pod --cidr $POD_CIDR As internal attacker from attacker tools container https://github.com/aquasecurity/kube-hunter
Test for Cloud Instance Metadata Service (Example) export TOKEN=$(curl -H "Metadata-Flavor: Google" http://metadata.google.internal/computeMetadata/v1/instance/service -accounts/default/token) curl -H "Metadata-Flavor: Google" http://metadata.google.internal/computeMetadata/v1/in stance/service-accounts/default/scopes curl -u "oauth2accesstoken:$TOKEN" https://eu.gcr.io/v2/_catalog Private registry access using instance service account token on Google Cloud
(Attacker in a Pod) Exploitation ● Objective? ○ Move around and gain access to other Pods (and resources) ○ Finally gain access to the cluster as cluster-admin ● How? ○ Known vulnerable components in the control plane ○ Open or vulnerable service in Pod/Service network ■ Example: Helm Tiller Privilege Escalation ○ Abusing privilege ■ Example: Privilege Escalation Abusing hostPath Volume Mount
Privilege Escalation using hostPath Volume Mount A hostPath volume mounts a file or directory from the host node's filesystem into your Pod. This is not something that most Pods will need, but it offers a powerful escape hatch for some applications. https://blog.appsecco.com/kubernetes-names pace-breakout-using-insecure-host-path-volu me-part-1-b382f2a6e216
Helm Tiller Privilege Escalation 1. Tiller, the in-cluster deployer component of Helm is running inside the cluster without authentication (default in Helm 2, removed in Helm 3) 2. We connect to tiller on predictable service name, namespace and port a. Alternatively, we can scan Service CIDR and discover tiller as well 3. We connect to tiller and ask it to install a chart that binds cluster-admin like privilege to namespace default service account 4. Our Pod, or for that matter, any Pod in running in default namespace now owns the cluster https://engineering.bitnami.com/articles/helm-security.html https://v2.helm.sh/docs/securing_installation/
OWASP Kubernetes Security Testing Guide (KSTG) ● Early stage - Work in Progress ○ https://owasp.org/www-project-kubernetes-security-testing-guide/ ○ https://github.com/owasp/kstg ● Aims to be the reference guide for Kubernetes Cluster Penetration Testing ● Me (@abh1sek) and Madhu Akula (@madhuakula) working on it for now, looking for your contribution :)
Appsecco (Free) Training on Docker & Kubernetes Security ● Free and open source training material including hands-on lab for Docker & Kubernetes security for you to try out. https://github.com/appsecco/atta cking-and-auditing-docker-contai ners-and-kubernetes-clusters
Kubernetes Threat Model and Penetration Test Report ● Kubernetes Security Working Group ○ Threat Model ○ Penetration Test Report ○ Security White paper ● https://github.com/kubernetes/community/tree/master/wg-security-audit
Other Useful Resources ● Hacker Container for Kubernetes Security Assessments ● Hacking and Hardening Kubernetes Clusters by Example [I] - Brad Geesaman, Symantec ● Advanced Persistent Threats: The Future of Kubernetes Attacks ● Kubernetes From an Attacker's Perspective — OWASP Bay Area Meetup ● CIS Benchmark for Kubernetes ● aquasecurity/kube-hunter: Hunt for security weaknesses in Kubernetes clusters ● aquasecurity/kube-bench: Checks whether Kubernetes is deployed according to security best practices as defined in the CIS Kubernetes Benchmark ● kelseyhightower/kubernetes-the-hard-way: Bootstrap Kubernetes the hard way on Google Cloud Platform. No scripts.
abhisek
.png){kind=icon}
pharma_x_tech
nrryuya
hamadakoji
soracom
kohbis
rigolon
finengine
no24oka
andpad
linedevth
moriyuya
kenta_hi
jakevdp
kastner
notwaldorf
destraynor
zakiyama
malarkey
aki_iinuma
carmenintech
andyhume
holman
bluesmoon
marcelosomers