Who else is in your pod? An attacker’s approach to Kubernetes Security Riyaz Walikar / 18th March 2021 

Riyaz Walikar Co-founder & Chief Breaker of Things @ Kloudle Inc. Twitter: @riyazwalikar, Blog: https://ibreak.software • Over a decade of experience in breaking web & mobile apps, networks, wireless, cloud and most recently container and Kubernetes • Have led Product Security and Consulting teams at Citrix and PwC SDC in a past life and was the OffSec Lead at Appsecco before moving on to co-founding Kloudle • Have multiple certifications as a result of my learning curve over the years, including CKA, CKAD, OSCP, CREST etc. (also, because they look cool on LinkedIn ☺) • Speaker at multiple cons, co-author, trainer, security evangelist with a short attention span and attracted to all things shiny 

Who else is in your pod? • What does your organisation’s footprint look like on the Internet? • What are attackers doing to get into your infrastructure or attack your Kubernetes? What Tactics and Techniques are they using? • What are some things you can do to minimize spread if an attack occurs or to prevent one in the first place 

No content

What does your attack footprint look like? • Employee information via forums/LinkedIn • Internal technology details via job postings, StackOverflow/Dev.to responses, blogposts • DNS records, history, IP ranges • TCP/IP services that get detected by Internet wide port scans • App stack and OS/server version info via HTTP Headers/page responses • Keys/Secrets embedded in client side JS • Internal hostnames, software and usernames embedded in PDF file properties • Default credentials for 3rd party apps • Older and vulnerable software/packages • Hidden default files that reveal package information for your app • Configuration files containing app internals and credentials • Repositories on Github, Bitbucket etc. • Commits containing secrets that have been removed but which linger in git history • Credentials and staging/dev API endpoints in documentation • Data in docker images • Misconfigured cloud storage/block storage 

Do folks really expose stuff online? 

ALL 

THE 

TIME 

What do attackers do? • All attackers, irrespective of how unstructured they are, follow a set of Tactics to reach their end goal • Each of these Tactics (consider them as milestones) in an attack have various Techniques by which the Tactic is fulfilled • Depending on the technology being targeted the Tactic may differ but often align with • Initial Discovery • Code execution • Privilege Escalation • Persistence • Evasion / Evidence Removal • Abuse / Impact 

MITRE ATT&CK Threat Matrix • MITRE ATT&CK® is a globally-accessible knowledge base of adversary tactics and techniques based on real-world observations - https://attack.mitre.org/ • Matrices have been created for various technology verticals – Windows, Linux, AWS, GCP, Azure, Android, iOS etc. • The Threat Matrix describes Tactics, Techniques and Procedures (TTPs) for these technologies and can be used by both offensive security folks to plan test cases and defenders to identify attack paths that the bad guys will take • Microsoft created a similar Threat Matrix for Kubernetes 

Kubernetes ATT&CK like Threat Matrix 

Initial Access: Exposed Dashboard https://redlock.io/blog/cryptojacking-tesla 

Initial Access: Application Vulnerability https://hackerone.com/reports/341876 

Initial Access • Multiple large companies have in the past exposed their Kubernetes Dashboards without authentication to the Internet • Kubelet REST API ports 10250 (read/write) and 10255 from older Kubernetes are still exposed on the Internet • Application vulnerabilities that can be used to make network requests or read files like SSRF, XXEs, LFIs, Path Traversal, arbitrary file downloads, SQL Injections or plain RCEs can be used to gain access to environment variables or files containing credentials 

Kubernetes ATT&CK like Threat Matrix 

Execution: New container 

Execution • Once the ability to interact with the cluster is obtained - either via an exposed dashboard, a kubeconfig file taken off a developer laptop or secrets leaked via an application vulnerability, attackers will attempt to gain proper shell access or run custom code within the cluster • Sometimes this Tactic can lead to a privilege escalation as well, leading to a compromise of the underlying node via a container escape weakness 

Kubernetes ATT&CK like Threat Matrix 

Persistence: Kubernetes Cronjob 

Persistence • Attackers can persist within the Kubernetes infrastructure and maintain access in various different ways including but not limited to • Adding SSH keys to the underlying node • Creating jobs/cronjobs that cause a reverse shell to be connected • Create shadow admin clusterroles or reuse a cluster-admin role • Use a writable hostPath mount to add custom code to apps or config files • Read and reuse secrets and configMaps from the cluster • Add a backdoor binary to a running container after gaining exec • Attackers like to persist in most networks for the thrill of seeing how long it takes owners to boot them out, to use the network as a means of a further attack/abuse or as a symbolic trophy in their collection 

Kubernetes ATT&CK like Threat Matrix 

Privilege Escalation: Cluster-admin binding 

Privilege Escalation • Escalation of privileges is a key step to gain additional access or to ensure the attacker remains hidden within the infrastructure • Some ways by which an attacker can escalate privileges include • Reading secrets, tokens or credentials from Kubernetes or application configuration • For managed Kubernetes on AWS, GCP or Azure, a service token account mapped to the IAM of the platform is tied to the cluster as well. This is usually accessible as a mount point within the pod • Detecting which pods are running as privileged pods and then using exec to gain access to them and from there, the underlying node if possible • Accessing the metadata endpoints to read IAM credentials or environment variables containing higher privilege credential material 

Kubernetes ATT&CK like Threat Matrix 

Defence Evasion: Pod name similarity 

Defence Evasion • As privileges increase within the cluster, attackers find novel ways to remove their traces and remain undetected. • Clearing logs by accessing the /var/log/pods directory on the underlying node • Clearing Kubernetes events using kubectl delete events –all • Launch pods in reserved namespaces to match other workloads that are running • Hide origin IP addresses using proxies, TOR etc. 

Kubernetes ATT&CK like Threat Matrix 

Credential Access: Container service account 

Credential Access • Most attackers will try and identify alternate regions, resources and networks that they can gain access to from their vantage point • To reach protected regions, they may require credentials that would unlock doors for them. Here’s some techniques that attackers use to get their hands on credentials • The Kubernetes secrets store • Using the service principal mounted within the underlying node to access Cloud resources (this allows the attacker to escape the Kubernetes environment into the managed cloud provider platform) • Pod mounted container service account tokens • App creds in configMaps and pod descriptions or hard coded within app files 

Kubernetes ATT&CK like Threat Matrix 

Discovery: Instance Metadata API 

Discovery • IMO this tactic is described too late in the Threat Matrix although most attackers will perform additional discovery during earlier stages • To find additional resources that attackers can reach and exploit, various combination of techniques, some of them already covered, are executed • Either via a kubeconfig or service tokens within the cluster, access to the K8S API server is obtained. If an attacker has reached this far, then this access is already assumed. • Kubelet REST API on TCP ports 10250 and 10255 on older K8s versions allow read and execute capabilities within the cluster • Attackers may also perform additional network mapping using tools like Nmap • Once inside the cluster, the Kubernetes dashboard may also be accessible • For managed Kubernetes, access to the underlying cloud platform may be possible via secrets leaked through instance metadata APIs 

Kubernetes ATT&CK like Threat Matrix 

Lateral Movement: Dashboard Access 

Lateral Movement • All the techniques covered in this tactic, barring the Helm Tiller Endpoint are all covered and reused by attackers at this point in the chain • Lateral movement tactic is simply used to capture techniques that can be used to move between resources within the cluster • Older versions of Helm (up to v2) had a server side component called Tiller which exposed an API endpoint that could be accessed without authentication on port 44134. This could be used to make a request to install resources using the helm client • helm --host tiller-deploy.kube-system:44134 install 

Kubernetes ATT&CK like Threat Matrix 

Impact: Resource Hijacking https://redlock.io/blog/cryptojacking-tesla 

Impact • Impact tactic is used to identify potential endgame goals of the attacker. Some documented techniques include • Data Destruction by rolling back deployments and removing volumes and claims • Hijacking resources to perform alternate tasks. Most commonly heard technique, examples include utilizing the cluster to perform cryptocurrency mining • Altering any state within the cluster can result in a Denial of Service. For example, rolling back a deployment, removal of secrets or altering configMaps, updating a NetworkPolicy etc. can all restrict application functionality and flow of network traffic 

Kubernetes ATT&CK like Threat Matrix 

Tips and Tricks to secure your cluster 

Tips for cluster security • Keep Kubernetes updated, including core components, API server, etcd etc. • Restrict SSH access to the nodes to only trusted IP addresses • Use Kubernetes NetworkPolicies judiciously, allow only what is required with a default deny otherwise • Audit roles and bindings, serviceaccounts, IAM access, look for shadow admins, use RBAC judiciously • Scan images in your pipeline using tools like Trivy to identify vulnerabilities that may become potential hazards • Remove deployments that are no longer required • Run pods with a security context and a defined user, say no to root 

Tips for cluster security • Use namespaces to create logical segregation of resources • Do not store secrets in environment variables, in pod description or configMaps • Ensure minimum visibility from the outside • Take application security into consideration, practice defense in depth • Be wary of suspicious looking outbound traffic • Use a hardening guideline like the CIS Kubernetes benchmark and tools like kube-bench to identify potential misconfigurations • Monitor and create an actionable roadmap into what your cluster is doing, who has access, what kind of access, what privileges are different components running, where are the logs going, what privileges does the IAM user have etc. 

To close the talk, we have Dilbert 

Q&A Site: https://kloudle.com Blog: https://kloudle.com/blog Twitter: - @kloudleinc - @riyazwalikar Reach us at: [email protected]