Who else is in your pod? - An Attacker's Approach to Kubernetes Security

Attacker's use one of three things to exploit and gain access to protected systems. The lack of (or misconfiguration) of security controls, vulnerabilities in the infrastructure and code itself or social engineering techniques to obtain information that can be used to gain access.

The presentation covers various techniques and tactics that attacker's use to collect information about a target's infrastructure, gain access to Kubernetes environments and perform various tactics to persist and escalate privileges. These tactics being mapped to the Kubernetes MITRE ATT&CK like framework that Microsoft created.

The presentation highlights the journey of how an attacker moves from gathering information and obtaining initial access to planting backdoors, evading defenses, escalating privileges and eventually causing an impact on the infrastructure.

Transcript

1. Who else is in your pod? An attacker’s approach to Kubernetes Security Riyaz Walikar / 18th March 2021
2. Riyaz Walikar Co-founder & Chief Breaker of Things @ Kloudle Inc. Twitter: @riyazwalikar, Blog: https://ibreak.software • Over a decade of experience in breaking web & mobile apps, networks, wireless, cloud and most recently container and Kubernetes • Have led Product Security and Consulting teams at Citrix and PwC SDC in a past life and was the OffSec Lead at Appsecco before moving on to co-founding Kloudle • Have multiple certifications as a result of my learning curve over the years, including CKA, CKAD, OSCP, CREST etc. (also, because they look cool on LinkedIn ☺) • Speaker at multiple cons, co-author, trainer, security evangelist with a short attention span and attracted to all things shiny
3. Who else is in your pod? • What does your organisation’s footprint look like on the Internet? • What are attackers doing to get into your infrastructure or attack your Kubernetes? What Tactics and Techniques are they using? • What are some things you can do to minimize spread if an attack occurs or to prevent one in the first place
5. What does your attack footprint look like? • Employee information via forums/LinkedIn • Internal technology details via job postings, StackOverflow/Dev.to responses, blogposts • DNS records, history, IP ranges • TCP/IP services that get detected by Internet wide port scans • App stack and OS/server version info via HTTP Headers/page responses • Keys/Secrets embedded in client side JS • Internal hostnames, software and usernames embedded in PDF file properties • Default credentials for 3rd party apps • Older and vulnerable software/packages • Hidden default files that reveal package information for your app • Configuration files containing app internals and credentials • Repositories on Github, Bitbucket etc. • Commits containing secrets that have been removed but which linger in git history • Credentials and staging/dev API endpoints in documentation • Data in docker images • Misconfigured cloud storage/block storage
6. Do folks really expose stuff online?
7. ALL
8. THE
9. TIME
10. What do attackers do? • All attackers, irrespective of how unstructured they are, follow a set of Tactics to reach their end goal • Each of these Tactics (consider them as milestones) in an attack have various Techniques by which the Tactic is fulfilled • Depending on the technology being targeted the Tactic may differ but often align with • Initial Discovery • Code execution • Privilege Escalation • Persistence • Evasion / Evidence Removal • Abuse / Impact
11. MITRE ATT&CK Threat Matrix • MITRE ATT&CK® is a globally-accessible knowledge base of adversary tactics and techniques based on real-world observations - https://attack.mitre.org/ • Matrices have been created for various technology verticals – Windows, Linux, AWS, GCP, Azure, Android, iOS etc. • The Threat Matrix describes Tactics, Techniques and Procedures (TTPs) for these technologies and can be used by both offensive security folks to plan test cases and defenders to identify attack paths that the bad guys will take • Microsoft created a similar Threat Matrix for Kubernetes
12. Kubernetes ATT&CK like Threat Matrix
13. Initial Access: Exposed Dashboard https://redlock.io/blog/cryptojacking-tesla
14. Initial Access: Application Vulnerability https://hackerone.com/reports/341876
15. Initial Access • Multiple large companies have in the past exposed their Kubernetes Dashboards without authentication to the Internet • Kubelet REST API ports 10250 (read/write) and 10255 from older Kubernetes are still exposed on the Internet • Application vulnerabilities that can be used to make network requests or read files like SSRF, XXEs, LFIs, Path Traversal, arbitrary file downloads, SQL Injections or plain RCEs can be used to gain access to environment variables or files containing credentials
16. Kubernetes ATT&CK like Threat Matrix
17. Execution: New container
18. Execution • Once the ability to interact with the cluster is obtained - either via an exposed dashboard, a kubeconfig file taken off a developer laptop or secrets leaked via an application vulnerability, attackers will attempt to gain proper shell access or run custom code within the cluster • Sometimes this Tactic can lead to a privilege escalation as well, leading to a compromise of the underlying node via a container escape weakness
19. Kubernetes ATT&CK like Threat Matrix
20. Persistence: Kubernetes Cronjob
21. Persistence • Attackers can persist within the Kubernetes infrastructure and maintain access in various different ways including but not limited to • Adding SSH keys to the underlying node • Creating jobs/cronjobs that cause a reverse shell to be connected • Create shadow admin clusterroles or reuse a cluster-admin role • Use a writable hostPath mount to add custom code to apps or config files • Read and reuse secrets and configMaps from the cluster • Add a backdoor binary to a running container after gaining exec • Attackers like to persist in most networks for the thrill of seeing how long it takes owners to boot them out, to use the network as a means of a further attack/abuse or as a symbolic trophy in their collection
22. Kubernetes ATT&CK like Threat Matrix
23. Privilege Escalation: Cluster-admin binding
24. Privilege Escalation • Escalation of privileges is a key step to gain additional access or to ensure the attacker remains hidden within the infrastructure • Some ways by which an attacker can escalate privileges include • Reading secrets, tokens or credentials from Kubernetes or application configuration • For managed Kubernetes on AWS, GCP or Azure, a service token account mapped to the IAM of the platform is tied to the cluster as well. This is usually accessible as a mount point within the pod • Detecting which pods are running as privileged pods and then using exec to gain access to them and from there, the underlying node if possible • Accessing the metadata endpoints to read IAM credentials or environment variables containing higher privilege credential material
25. Kubernetes ATT&CK like Threat Matrix
26. Defence Evasion: Pod name similarity
27. Defence Evasion • As privileges increase within the cluster, attackers find novel ways to remove their traces and remain undetected. • Clearing logs by accessing the /var/log/pods directory on the underlying node • Clearing Kubernetes events using kubectl delete events –all • Launch pods in reserved namespaces to match other workloads that are running • Hide origin IP addresses using proxies, TOR etc.
28. Kubernetes ATT&CK like Threat Matrix
29. Credential Access: Container service account
30. Credential Access • Most attackers will try and identify alternate regions, resources and networks that they can gain access to from their vantage point • To reach protected regions, they may require credentials that would unlock doors for them. Here’s some techniques that attackers use to get their hands on credentials • The Kubernetes secrets store • Using the service principal mounted within the underlying node to access Cloud resources (this allows the attacker to escape the Kubernetes environment into the managed cloud provider platform) • Pod mounted container service account tokens • App creds in configMaps and pod descriptions or hard coded within app files
31. Kubernetes ATT&CK like Threat Matrix
32. Discovery: Instance Metadata API
33. Discovery • IMO this tactic is described too late in the Threat Matrix although most attackers will perform additional discovery during earlier stages • To find additional resources that attackers can reach and exploit, various combination of techniques, some of them already covered, are executed • Either via a kubeconfig or service tokens within the cluster, access to the K8S API server is obtained. If an attacker has reached this far, then this access is already assumed. • Kubelet REST API on TCP ports 10250 and 10255 on older K8s versions allow read and execute capabilities within the cluster • Attackers may also perform additional network mapping using tools like Nmap • Once inside the cluster, the Kubernetes dashboard may also be accessible • For managed Kubernetes, access to the underlying cloud platform may be possible via secrets leaked through instance metadata APIs
34. Kubernetes ATT&CK like Threat Matrix
35. Lateral Movement: Dashboard Access
36. Lateral Movement • All the techniques covered in this tactic, barring the Helm Tiller Endpoint are all covered and reused by attackers at this point in the chain • Lateral movement tactic is simply used to capture techniques that can be used to move between resources within the cluster • Older versions of Helm (up to v2) had a server side component called Tiller which exposed an API endpoint that could be accessed without authentication on port 44134. This could be used to make a request to install resources using the helm client • helm --host tiller-deploy.kube-system:44134 install <packagename>
37. Kubernetes ATT&CK like Threat Matrix
38. Impact: Resource Hijacking https://redlock.io/blog/cryptojacking-tesla
39. Impact • Impact tactic is used to identify potential endgame goals of the attacker. Some documented techniques include • Data Destruction by rolling back deployments and removing volumes and claims • Hijacking resources to perform alternate tasks. Most commonly heard technique, examples include utilizing the cluster to perform cryptocurrency mining • Altering any state within the cluster can result in a Denial of Service. For example, rolling back a deployment, removal of secrets or altering configMaps, updating a NetworkPolicy etc. can all restrict application functionality and flow of network traffic
40. Kubernetes ATT&CK like Threat Matrix
41. Tips and Tricks to secure your cluster
42. Tips for cluster security • Keep Kubernetes updated, including core components, API server, etcd etc. • Restrict SSH access to the nodes to only trusted IP addresses • Use Kubernetes NetworkPolicies judiciously, allow only what is required with a default deny otherwise • Audit roles and bindings, serviceaccounts, IAM access, look for shadow admins, use RBAC judiciously • Scan images in your pipeline using tools like Trivy to identify vulnerabilities that may become potential hazards • Remove deployments that are no longer required • Run pods with a security context and a defined user, say no to root
43. Tips for cluster security • Use namespaces to create logical segregation of resources • Do not store secrets in environment variables, in pod description or configMaps • Ensure minimum visibility from the outside • Take application security into consideration, practice defense in depth • Be wary of suspicious looking outbound traffic • Use a hardening guideline like the CIS Kubernetes benchmark and tools like kube-bench to identify potential misconfigurations • Monitor and create an actionable roadmap into what your cluster is doing, who has access, what kind of access, what privileges are different components running, where are the logs going, what privileges does the IAM user have etc.
44. To close the talk, we have Dilbert
45. Q&A Site: https://kloudle.com Blog: https://kloudle.com/blog Twitter: - @kloudleinc - @riyazwalikar Reach us at: riyaz@kloudle.com