Cloud Native Security For The Rest Of Us

Your mission is to secure the vast tracts of land of the Cloud Native security landscape. Where do you even start?!? It would be preposterous to cover that whole topic in a single session, but we can at least map it out. The plan is to break it down into three key areas and review each in turn.

* Platform - securing and upgrading our control planes and nodes; isolating compute, storage, and network resources; managing privileges and secrets.
* User management and permissions - various ways to authenticate and authorize user access; leveraging tools like RBAC and Namespaces, and some common "gotchas".
* Software supply chain - what that means, some actual threat models, and how to mitigate them.

You will leave this session with a stronger understanding of the breadth and depth of Cloud Native security and resources to further develop your knowledge.

Transcript

1. Cloud Native Security For The Rest Of Us Tiffany Jernigan Developer Advocate VMware tiffanyfayj Open Source Summit EU - Sept 2022
2. T I F F A N Y F A Y J FOLLOW SECURITY BEST PRACTICES • Less is more: having less is more secure e.g. ◦ Less code: prefer managed services and take software off the shelf rather than implementing your own security solutions, wherever possible ◦ Fewer permissions: e.g. avoid long-lived secrets ◦ Fewer dependencies: using smaller images (e.g distroless) • Keep up with latest recommendations (e.g. take a look at k8s.io/docs/concepts/security/security-checklist/ periodically)
3. T I F F A N Y F A Y J 4 C's OF CLOUD NATIVE SECURITY k8s.io/docs/concepts/security/overview/
4. T I F F A N Y F A Y J PLATFORM
5. T I F F A N Y F A Y J MANAGED >> DIY • Don’t run it yourself, unless you really need to • Use a "Kubernetes as a Service" (e.g. EKS, GKE, …) or hire specialists to do it (if you need that to be on prem)
6. T I F F A N Y F A Y J SECURING CONTROL PLANES & NODES Some things that could go wrong (and have gone wrong even in clusters managed by very smart people): • Restrict control plane access to just the Kubernetes API server (avoid exposing other ports/services on the same host) • Don't get lazy with insecure-skip-tls-verify! • TLS problems (see "PKI The Wrong Way" KubeCon talk) ◦ github.com/tabbysable/pki-the-wrong-way
7. T I F F A N Y F A Y J SECURING CONTROL PLANES & NODES CONT’D Some things that could go wrong (and have gone wrong even in clusters managed by very smart people): • Lack of Pod Security ◦ See bit.ly/hacktheplanetyaml, which gives an attacker root access to your nodes if you lack PSP/PSS • Access to cloud instance metadata (SSRF)
8. T I F F A N Y F A Y J UPGRADING KUBERNETES • Update, update, update • Take advantage of managed Kubernetes offerings • Pay attention to deprecation cycles
9. T I F F A N Y F A Y J THE WORK OF UPGRADING K8S CLUSTERS Just for your enjoyment, here's what needs to be done and in which specific order: • Upgrade control plane • Upgrade nodes • Upgrade clients such as kubectl • Adjust manifests and other resources based on the API changes that accompany the new Kubernetes version k8s.io/docs/tasks/administer-cluster/cluster-upgrade/
10. T I F F A N Y F A Y J ISOLATING COMPUTE • Quota and limit ranges • Use taints and tolerations to schedule workloads away from each other • Sandboxed container runtimes e.g. gVisor, Kata Containers, Firecracker
11. T I F F A N Y F A Y J ISOLATING STORAGE • Container Storage Interface (CSI)
12. T I F F A N Y F A Y J ISOLATING NETWORK RESOURCES • Ingress/Egress ◦ AKA firewalling with Network Policies ◦ k8s.io/docs/concepts/services-networking/network-policies/ • Service Mesh ◦ To include mutual TLS authentication • Going the extra mile: advanced network policies with tools like Cilium
13. T I F F A N Y F A Y J MANAGING SECRETS • Encryption at REST ◦ k8s.io/docs/tasks/administer-cluster/encrypt-data/ • Don’t put secret data directly in ConfigMaps; put them in Secrets
14. T I F F A N Y F A Y J MANAGING SECRETS • Even better: don't put stuff DIRECTLY in secrets, e.g.: ◦ Key Management Service (KMS) ◦ Stuff like Hashicorp Vault ◦ SealedSecrets ◦ Kamus ◦ SOPS
15. T I F F A N Y F A Y J USER MANAGEMENT & PERMISSIONS
16. T I F F A N Y F A Y J AUTHN & AUTHZ Good old separation between: • AUTHN (authentication): who are you? • AUTHZ (authorization): what are you allowed to do?
17. T I F F A N Y F A Y J AUTHENTICATION (AUTHN) • Recall best practices: prefer to use short-lived credentials, e.g. OAuth access_tokens instead of username and password • Humans: TLS, OIDC, Service Account/Client Certs, etc. • Robots: use Service Accounts • SPIFFE.io k8s.io/docs/reference/access-authn-authz/authentication/
18. T I F F A N Y F A Y J AUTHORIZATION (AUTHZ) • Do you have permission to do what you’re trying to do? • Kubernetes API Server Authorization modes: ◦ Node ◦ ABAC ◦ RBAC ◦ Webhook k8s.io/docs/reference/access-authn-authz/authorization
19. T I F F A N Y F A Y J ROLE-BASED ACCESS CONTROL (RBAC) High level idea on Kubernetes: 1. Define a ROLE, which is a collection of permissions ("things that can be done") a. e.g. list pods, create deployments, set the scaling for this one particular deployment, etc. 2. Bind the ROLE to a USER or GROUP or SERVICEACCOUNT (with a RoleBinding or ClusterRoleBinding)
20. T I F F A N Y F A Y J RBAC AUDITING After setting permissions, audit them: • kubectl can-i --list • kubectl who-can / kubectl-who-can by Aqua Security • kubectl access-matrix / Rakkess (Review Access) by Cornelius Weig • kubectl rbac-lookup / RBAC Lookup by FairwindsOps • kubectl rbac-tool / RBAC Tool by insightCloudSec
21. T I F F A N Y F A Y J NAMESPACES • Permissions can be defined clusterwide, or only in a specific namespace • Someone can have access restricted to a specific namespace, • They can even be an admin in their own namespace (i.e. delegate permissions)
22. T I F F A N Y F A Y J GOTCHA: ADMIN PERMISSIONS • Don’t blanket give admin permissions • Namespaces make it easier to be lazy!
23. T I F F A N Y F A Y J GOTCHA: LIST SECRETS • Kubernetes has separate list and get permissions • Expected: ◦ list: enumerates ◦ get: gets details • Reality: ◦ list: enumerates AND gets details O.o
24. T I F F A N Y F A Y J SOFTWARE SUPPLY CHAIN
25. T I F F A N Y F A Y J SOFTWARE SUPPLY CHAIN Everything PR to PRoduction and everything in between: • Commit access to source repositories • Its dependencies • How software is built • Where built artifacts are stored • How they are deployed • The trail of breadcrumbs leading from a running workload all the way back to the source
26. T I F F A N Y F A Y J HARDEN EVERY LINK IN THE CHAIN • Is the code from a trusted person? • Is it from your source repo? • Push authorization? • Sign everything with Sigstore! ◦ github.com/sigstore/gitsign ◦ github.com/sigstore/cosign
27. T I F F A N Y F A Y J HARDEN EVERY LINK IN THE CHAIN CONT’D • Trusted source when building? • Trusted build system? • Trusted platform build system is running on? • Who/what has push access?
28. T I F F A N Y F A Y J ATTESTATION • Vulnerability Scanning for CVEs ◦ e.g. Clair, Trivy • Policy enforcement ◦ e.g. Open Policy Agent (OPA), Gatekeepergithub.com/sigstore/policy-controller
29. T I F F A N Y F A Y J SOME ACTUAL THREAT MODELS & HOW TO MITIGATE THEM • Someone makes changes to code etc. that you depend on which causes your stuff to break ◦ Have immutable dependences • Attack on build infra (e.g. Solarwinds) ◦ Have ephemeral builds
30. T I F F A N Y F A Y J A FEW MORE THINGS
31. T I F F A N Y F A Y J ADDITIONAL COMPONENTS Even with managed Kubernetes, we're still potentially missing a lot of critical components, like: • Backups (e.g. Velero.io) • Observability ◦ Metrics (e.g. Prometheus.io, Grafana.com) ◦ Auditing/Logging
32. T I F F A N Y F A Y J DON’T FORGET ABOUT YOUR CODE • Source code analysis (e.g. OWASP.org)
33. T I F F A N Y F A Y J SOME RESOURCES • landscape.cncf.io • kubernetes.io/docs/concepts/security • kubernetes.io/docs/tasks/administer-cluster • sigstore.dev
34. tiffanyfayj Special thanks to: Andrés Vega Jake Sanders Jérôme Petazzoni Lewis Denham-Parry THANK YOU!