Container Security for RED and BLUE Teams! - All Day DevOps 2020 Spring Break Edition

In this talk will focus on two different perspectives of container security. We will see, how attackers can break into docker container and kubernetes clusters to gain access, escalate privileges to infrastructure by using misconfigurations and application security vulnerabilities. Also, we can see how defenders can leverage the power of automation at different layers like Infrastructure Security, Supply chain Security and Run time Security to protect against the container security attacks.

By the end of this talk participants able to use this knowledge to identify and exploit vulnerabilities in applications running on containers inside kubernetes clusters. Also, it will be useful to apply pragmatic security using tools, techniques and procedures (TTPs) to secure the container infrastructure.

**About Madhu Akula**

Madhu Akula is a security ninja, published author and cloud native security researcher with an extensive experience. Also he is an active member of the international security, devops and cloud native communities (null, DevSecOps, AllDayDevOps, etc). Hold industry certifications like OSCP (Offensive Security Certified Professional), CKA (Certified Kubernetes Administrator), etc. His research has identified vulnerabilities in over 200+ companies and organisations including; Google, Microsoft, LinkedIn, eBay, AT&T, WordPress, NTOP and Adobe, etc and credited with multiple CVE’s, Acknowledgements and rewards. He is co-author of Security Automation with Ansible2 (ISBN-13: 978-1788394512), which is listed as a technical resource by Red Hat Ansible. Also won 1st prize for building Infrastructure Security Monitoring solution at InMobi flagship hackathon among 100+ engineering teams.

Transcript

1. B R
2. About - Madhu Akula • Cloud Native Security Specialist @ Xebia • Security (Cloud, Containers, Kubernetes & Automation) • Speaker & Trainer @ BlackHat, DEF CON, USENIX LISA, OWASP Appsec EU, All Day DevOps, DevSecCon, c0c0n, Nullcon, null, etc. • Co-Author of Security Automation with Ansible2 book • Discovered vulnerabilities in over 200+ organizations including; Google, Microsoft, LinkedIn, Coudflare, AT&T, Wordpress, NTOP and Adobe, etc. • Holds industry certifications like OSCP and CKA • Never Ending Learner! • https://madhuakula.com @madhuakula B R
3. What you will learn in next 30 minutes! • This talk is NOT about what is Docker, Kubernetes, etc. • This talk is about • Why container security? • Red team overview of container security • Blue team overview of container security • Tools, Techniques and Procedures (TTP’s) • Real-world scenarios • Approach to attackers and defenders • Case studies and reference resources • Next steps for learning more and more @madhuakula B R
4. Would you like to learn Docker & Kubernetes? • https://docs.docker.com • https://kubernetes.io/docs/home • https://container.training • https://training.play-with-docker.com • https://labs.play-with-k8s.com • https://training.play-with-kubernetes.com • https://www.katacoda.com/learn • Many more... @madhuakula B R
5. Why container security? @madhuakula https://kromtech.com/blog/security-center/cryptojacking-invades-cloud-how-modern-containerization-trend-is-exploited-by-attackers
6. Why container security? @madhuakula https://blog.madhuakula.com/some-tips-to-review-docker-hub-hack-of-190k-accounts-addcd602aade
7. Why container security? @madhuakula https://hackerone.com/reports/341876
8. Why container security? Many other vulnerabilities and real-world impacts... @madhuakula
9. Container Security Overview • Container Attack Surface • Namespaces • Control Groups • Daemon • Configuration • Capabilities • Content Trust • Container Registry • Volumes • Networks • Many other... @madhuakula B R
10. Kubernetes Security Overview @madhuakula https://www.microsoft.com/security/blog/2020/04/02/attack-matrix-kubernetes/ B R
11. amicontained - Container Introspection Tool https://github.com/genuinetools/amicontained It helps to find out what container runtime is being used as well as features available like capabilities, profiles applied, etc. amicontained - Container Introspection Tool B R
12. • Committing the sensitive information to version control systems • Not including the sensitive files in the build process using .dockerignore file • This is one of the common mistake in modern era trufflehog - Hardcoded sensitive information B R
13. Insecurely configured docker service R
14. Insecure docker socket service Insecure docker socket service R
15. Insecure docker socket service Securing Daemon Configuration B • By default, Docker runs through a non-networked UNIX socket. It can also optionally communicate using an HTTP socket. • If you need Docker to be reachable through the network in a safe manner, you can enable TLS by specifying the tlsverify flag and pointing Docker’s tlscacert flag to a trusted CA certificate. • If you want to secure your Docker client connections by default, you can move the files to the .docker directory in your home directory --- and set the DOCKER_HOST and DOCKER_TLS_VERIFY variables as well (instead of passing -H=tcp://$HOST:2376 and --tlsverify on every call). https://docs.docker.com/engine/security/https
16. Analysing or Understanding unknown image Analyzing or Understanding unknown image B R
17. Insecure docker socket service Content Trust B • Docker Content Trust (DCT) provides the ability to use digital signatures for data sent to and received from remote Docker registries. These signatures allow client-side or runtime verification of the integrity and publisher of specific image tags • We can sign and push a container image with the docker trust command • Content trust is disabled by default in the Docker Client. To enable it, set the DOCKER_CONTENT_TRUST environment variable to 1. This prevents users from working with tagged images unless they contain a signature. https://docs.docker.com/engine/security/trust/content_trust/
18. dive - Exploring each layer in a docker image https://github.com/wagoodman/dive dive - Exploring each layer in a docker image B R
19. Inspecting container volumes Inspecting container volumes B R
20. Volume analysis for sensitive information Volume analysis for sensitive information R
21. Inspecting container networking Inspecting container networking B R
22. Always look for env variables • This is one of the common places most developers and operations teams store secrets, API keys, etc. • Also it contains other information like different service or cluster related information Always look for env variables B R
23. docker diff - comparing with base image docker diff - comparing with base image B R
24. container escape - extra capability and host pid container escape - extra capability and host pid R
25. container escape - extra capability and host pid container escape - extra capability and host pid R
26. Insecure docker socket service Security Profiles B • The Linux Security Module (LSM) framework provides a mechanism for various security checks to be hooked by new kernel extensions • AppArmor profile generator for docker containers using bane https://github.com/genuinetools/bane
27. Kubernetes secrets are not encrypted! Kubernetes secrets are not encrypted! B R
28. Insecure docker socket service Injecting Vault Secrets Into K8S Pods via a Sidecar B https://www.hashicorp.com/blog/injecting-vault-secrets-into-kubernetes-pods-via-a-sidecar/ # patch-basic-annotations.yaml spec: template: metadata: annotations: vault.hashicorp.com/agent-inject: "true" vault.hashicorp.com/agent-inject-secret-helloworld: "secrets/helloworld" vault.hashicorp.com/role: "myapp"
29. SSRF in the kubernetes world like a Cluster Pwn • In the Google Cloud (GCP), we have to use Metadata-Flavor: Google to obtain the metadata • Now GKE offers to protect kube-env using metadata concealment proxy and workload identity SSRF in the kubernetes world like a Cluster Pwn R
30. SSRF in the kubernetes world like a Cluster Pwn SSRF in the kubernetes world like a Cluster Pwn R
31. Insecure docker socket service Metadata Concealment B • Most of the cloud providers has fix for this in some way • GKE: Workload Identity, Metadata Concealment for Nodes https://cloud.google.com/kubernetes-engine/docs/how-to/workload-identity • AWS: IMDSv2 for SSRF https://aws.amazon.com/blogs/security/defense-in-depth-open-firewalls-reverse-proxies-ssrf-vulnerabilities-ec2-instance- metadata-service/ https://github.com/genuinetools/bane
32. Command Injection to node access (host) Command Injection to node access (host) R
33. Command Injection to node access (host) Command Injection to node access (host) R
34. Command Injection to node access (host) Command Injection to node access (host) R
35. Kubernetes centralised logs in stack driver Runtime Security Detection – Sysdig Falco https://www.youtube.com/watch?v=zd0ksjZI5Vk https://falco.org/ Falco, the open source cloud-native runtime security project, is the defacto Kubernetes threat detection engine. Falco detects unexpected application behaviour and alerts on threats at runtime. B
36. No default security boundary in k8s namespaces No default security boundary in k8s namespaces B R
37. Kubernetes centralised logs in stack driver Network Security Policies https://github.com/ahmetb/kubernetes-network-policy-recipes Provides isolation between Kubernetes resources (pods, namespaces, svc, etc.) using labels and selectors across the cluster. B
38. Default service account in a Pod Default service account in a Pod R
39. Default service account in a Pod https://hackernoon.com/capturing-all-the-flags-in-bsidessf-ctf-by-pwning-our-infrastructure-3570b99b4dd0 Default service account in a Pod Default service account in a Pod https://hackernoon.com/capturing-all-the-flags-in-bsidessf-ctf-by-pwning-our-infrastructure-3570b99b4dd0 Default service account in a Pod R
40. Default misconfigured Helm Tiller = Cluster Pwn https://engineering.bitnami.com/articles/helm-security.html Default misconfigured Helm Tiller = Cluster Pwn R
41. Default misconfigured Helm Tiller = Cluster Pwn Default misconfigured Helm Tiller = Cluster Pwn R
42. Kubernetes centralised logs in stack driver RBAC with least privilege access possible https://kubernetes.io/docs/reference/access-authn-authz/rbac/ Role-based access control (RBAC) is a method of regulating access to computer or network resources based on the roles of individual users within your organization. B Useful utilities to check out is https://github.com/liggitt/audit2rbac https://github.com/FairwindsOps/rbac -manager https://github.com/jtblin/kube2iam
43. Trivy - Vulnerability Scanner for Containers https://github.com/aquasecurity/trivy Trivy - Vulnerability Scanner for Containers B R
44. dockle - Container Image Linter for Security https://github.com/goodwithtech/dockle dockle - Container Image Linter for Security B R
45. docker-bench-security https://github.com/docker/docker-bench-security • A script that checks for dozens of common best-practices around deploying Docker containers in production ◦ Host configuration ◦ Docker daemon configuration and files ◦ Docker container images ◦ Docker runtime ◦ Docker security operations ◦ Docker swarm configuration docker-bench-security B R
46. kube-bench - CIS Kubernetes Benchmark https://github.com/aquasecurity/kube-bench • Master Node Security Configuration ◦ API Server ◦ Scheduler ◦ Controller Manager ◦ Configuration Files ◦ etcd ◦ General Security Primitives ◦ PodSecurityPolicices • Worker Node Security Configuration ◦ Kubelet ◦ Configuration Files kube-bench - CIS Kubernetes Benchmark B R
47. kube-hunter • Kube-hunter hunts for security weaknesses in Kubernetes clusters. The tool was developed to increase awareness and visibility for security issues in Kubernetes environments. You should NOT run kube-hunter on a Kubernetes cluster you don't own! kube-hunter B R
48. kubeaudit - Audit your kubernetes clusters https://github.com/Shopify/kubeaudit kubeaudit - Audit your kubernetes clusters B R
49. kubesec.io - Risk analysis for k8s resources https://kubesec.io/ kubesec.io - Risk analysis for k8s resources B R
50. CVE-2018-1002105 https://www.youtube.com/watch?v=4CTK2aUXTHo Known Vulnerabilities (CVE-2018-1002105) B R
51. https://www.youtube.com/watch?v=4CTK2aUXTHo CVE-2018-1002105 Known Vulnerabilities (CVE-2018-1002105) B R
52. https://github.com/Frichetten/CVE-2019-5736-PoC • This is a Go implementation of CVE-2019-5736, a container escape for Docker. The exploit works by overwriting and executing the host systems runc binary from within the container CVE-2019-5736 Known Vulnerabilities (CVE-2019-5736) B R
53. https://github.com/eoftedal/writings/blob/master/published/CVE-2019-9901-path-traversal.md CVE-2019-9901 - Istio/Envoy Path traversal CVE-2019-9901 : Istio/Envoy Path traversal B R
54. docker logs and events docker logs and events B R
55. Kubernetes centralised logs in stack driver Centralized logging and Monitoring B
56. Kubernetes centralised logs in stack driver Open Policy Agent – Policy Engine https://www.openpolicyagent.org/ Policy-based control for cloud native environments Flexible, fine-grained control for administrators across the stack B
57. Kubernetes centralised logs in stack driver Container Runtime Sandbox gVisor, Firecracker, etc. B • gVisor is a user-space kernel, written in Go, that implements a substantial portion of the Linux system call interface. It provides an additional layer of isolation between running applications and the host operating system. • Firecracker is an open source virtualization technology that is purpose- built for creating and managing secure, multi-tenant container and function-based services. • Many other…
58. Kubernetes centralised logs in stack driver TLS – Let’s Encrypt with cert-manager https://cert-manager.io/ Automate certificate management in cloud native environments. cert-manager builds on top of Kubernetes, introducing certificate authorities and certificates as first-class resource types in the Kubernetes API. This makes it possible to provide 'certificates as a service' to developers working within your Kubernetes cluster. B
59. Kubernetes centralised logs in stack driver Pod Security Policies https://kubernetes.io/docs/concepts/policy/pod-security-policy A Pod Security Policy is a cluster- level resource that controls security sensitive aspects of the pod specification. The PodSecurityPolicy objects define a set of conditions that a pod must run with in order to be accepted into the system, as well as defaults for the related fields. Good utility to check out is https://github.com/sysdiglabs/k ube-psp-advisor B
60. References & Resources • Docker Security Docs • Kubernetes Security Docs • Attack matrix for Kubernetes • Breaking & Pwning Docker Containers & Kubernetes Clusters • Advanced Persistence Threats: The Future of Kubernetes Attacks • Kubernetes Security Resources • 11 Ways (Not) to Get Hacked • Attacking & Auditing Docker Containers using Open Source @ DEFCON 26 • Attacking and Auditing Docker Containers and Kubernetes Clusters @ DEFCON 27 @madhuakula B R
61. References & Resources • contained.af • CIS Benchmarks Docker • Understanding and Hardening Linux Containers • Abusing Privileged and Unprivileged Linux Containers • Container Security Notes • Linux Container Security • Docker Runtime Privileges and Capabilities • Apparmor Security Profiles on Docker • Seccomp Security Profiles on Docker • Docker Labs Capabilities • Practical SELinux and Containers • Containers and Operating systems morning paper gist • Kubernetes Network Policies • Kubernetes Webinar series @madhuakula B R
62. Are you looking for more and more tools? https://tools.cloudnativesecurity.info @madhuakula B R
63. Thank You! Madhu Akula @madhuakula https://madhuakula.com @madhuakula B R