Cloud Native Security For The Rest Of Us Tiffany Jernigan Developer Advocate VMware tiffanyfayj Open Source Summit EU - Sept 2022

T I F F A N Y F A Y J FOLLOW SECURITY BEST PRACTICES ● Less is more: having less is more secure e.g. ○ Less code: prefer managed services and take software off the shelf rather than implementing your own security solutions, wherever possible ○ Fewer permissions: e.g. avoid long-lived secrets ○ Fewer dependencies: using smaller images (e.g distroless) ● Keep up with latest recommendations (e.g. take a look at k8s.io/docs/concepts/security/security-checklist/ periodically)

T I F F A N Y F A Y J 4 C's OF CLOUD NATIVE SECURITY k8s.io/docs/concepts/security/overview/

T I F F A N Y F A Y J PLATFORM

T I F F A N Y F A Y J MANAGED >> DIY ● Don’t run it yourself, unless you really need to ● Use a "Kubernetes as a Service" (e.g. EKS, GKE, …) or hire specialists to do it (if you need that to be on prem)

T I F F A N Y F A Y J SECURING CONTROL PLANES & NODES Some things that could go wrong (and have gone wrong even in clusters managed by very smart people): ● Restrict control plane access to just the Kubernetes API server (avoid exposing other ports/services on the same host) ● Don't get lazy with insecure-skip-tls-verify! ● TLS problems (see "PKI The Wrong Way" KubeCon talk) ○ github.com/tabbysable/pki-the-wrong-way

T I F F A N Y F A Y J SECURING CONTROL PLANES & NODES CONT’D Some things that could go wrong (and have gone wrong even in clusters managed by very smart people): ● Lack of Pod Security ○ See bit.ly/hacktheplanetyaml, which gives an attacker root access to your nodes if you lack PSP/PSS ● Access to cloud instance metadata (SSRF)

T I F F A N Y F A Y J UPGRADING KUBERNETES ● Update, update, update ● Take advantage of managed Kubernetes offerings ● Pay attention to deprecation cycles

T I F F A N Y F A Y J THE WORK OF UPGRADING K8S CLUSTERS Just for your enjoyment, here's what needs to be done and in which specific order: ● Upgrade control plane ● Upgrade nodes ● Upgrade clients such as kubectl ● Adjust manifests and other resources based on the API changes that accompany the new Kubernetes version k8s.io/docs/tasks/administer-cluster/cluster-upgrade/

T I F F A N Y F A Y J ISOLATING COMPUTE ● Quota and limit ranges ● Use taints and tolerations to schedule workloads away from each other ● Sandboxed container runtimes e.g. gVisor, Kata Containers, Firecracker

T I F F A N Y F A Y J ISOLATING STORAGE ● Container Storage Interface (CSI)

T I F F A N Y F A Y J ISOLATING NETWORK RESOURCES ● Ingress/Egress ○ AKA firewalling with Network Policies ○ k8s.io/docs/concepts/services-networking/network-policies/ ● Service Mesh ○ To include mutual TLS authentication ● Going the extra mile: advanced network policies with tools like Cilium

T I F F A N Y F A Y J MANAGING SECRETS ● Encryption at REST ○ k8s.io/docs/tasks/administer-cluster/encrypt-data/ ● Don’t put secret data directly in ConfigMaps; put them in Secrets

T I F F A N Y F A Y J MANAGING SECRETS ● Even better: don't put stuff DIRECTLY in secrets, e.g.: ○ Key Management Service (KMS) ○ Stuff like Hashicorp Vault ○ SealedSecrets ○ Kamus ○ SOPS

T I F F A N Y F A Y J USER MANAGEMENT & PERMISSIONS

T I F F A N Y F A Y J AUTHN & AUTHZ Good old separation between: ● AUTHN (authentication): who are you? ● AUTHZ (authorization): what are you allowed to do?

T I F F A N Y F A Y J AUTHENTICATION (AUTHN) ● Recall best practices: prefer to use short-lived credentials, e.g. OAuth access_tokens instead of username and password ● Humans: TLS, OIDC, Service Account/Client Certs, etc. ● Robots: use Service Accounts ● SPIFFE.io k8s.io/docs/reference/access-authn-authz/authentication/

T I F F A N Y F A Y J AUTHORIZATION (AUTHZ) ● Do you have permission to do what you’re trying to do? ● Kubernetes API Server Authorization modes: ○ Node ○ ABAC ○ RBAC ○ Webhook k8s.io/docs/reference/access-authn-authz/authorization

T I F F A N Y F A Y J ROLE-BASED ACCESS CONTROL (RBAC) High level idea on Kubernetes: 1. Define a ROLE, which is a collection of permissions ("things that can be done") a. e.g. list pods, create deployments, set the scaling for this one particular deployment, etc. 2. Bind the ROLE to a USER or GROUP or SERVICEACCOUNT (with a RoleBinding or ClusterRoleBinding)

T I F F A N Y F A Y J RBAC AUDITING After setting permissions, audit them: ● kubectl can-i --list ● kubectl who-can / kubectl-who-can by Aqua Security ● kubectl access-matrix / Rakkess (Review Access) by Cornelius Weig ● kubectl rbac-lookup / RBAC Lookup by FairwindsOps ● kubectl rbac-tool / RBAC Tool by insightCloudSec

T I F F A N Y F A Y J NAMESPACES ● Permissions can be defined clusterwide, or only in a specific namespace ● Someone can have access restricted to a specific namespace, ● They can even be an admin in their own namespace (i.e. delegate permissions)

T I F F A N Y F A Y J GOTCHA: ADMIN PERMISSIONS ● Don’t blanket give admin permissions ● Namespaces make it easier to be lazy!

T I F F A N Y F A Y J GOTCHA: LIST SECRETS ● Kubernetes has separate list and get permissions ● Expected: ○ list: enumerates ○ get: gets details ● Reality: ○ list: enumerates AND gets details O.o

T I F F A N Y F A Y J SOFTWARE SUPPLY CHAIN

T I F F A N Y F A Y J SOFTWARE SUPPLY CHAIN Everything PR to PRoduction and everything in between: ● Commit access to source repositories ● Its dependencies ● How software is built ● Where built artifacts are stored ● How they are deployed ● The trail of breadcrumbs leading from a running workload all the way back to the source

T I F F A N Y F A Y J HARDEN EVERY LINK IN THE CHAIN ● Is the code from a trusted person? ● Is it from your source repo? ● Push authorization? ● Sign everything with Sigstore! ○ github.com/sigstore/gitsign ○ github.com/sigstore/cosign

T I F F A N Y F A Y J HARDEN EVERY LINK IN THE CHAIN CONT’D ● Trusted source when building? ● Trusted build system? ● Trusted platform build system is running on? ● Who/what has push access?

T I F F A N Y F A Y J ATTESTATION ● Vulnerability Scanning for CVEs ○ e.g. Clair, Trivy ● Policy enforcement ○ e.g. Open Policy Agent (OPA), Gatekeepergithub.com/sigstore/policy-controller

T I F F A N Y F A Y J SOME ACTUAL THREAT MODELS & HOW TO MITIGATE THEM ● Someone makes changes to code etc. that you depend on which causes your stuff to break ○ Have immutable dependences ● Attack on build infra (e.g. Solarwinds) ○ Have ephemeral builds

T I F F A N Y F A Y J A FEW MORE THINGS

T I F F A N Y F A Y J ADDITIONAL COMPONENTS Even with managed Kubernetes, we're still potentially missing a lot of critical components, like: ● Backups (e.g. Velero.io) ● Observability ○ Metrics (e.g. Prometheus.io, Grafana.com) ○ Auditing/Logging

T I F F A N Y F A Y J DON’T FORGET ABOUT YOUR CODE • Source code analysis (e.g. OWASP.org)

T I F F A N Y F A Y J SOME RESOURCES ● landscape.cncf.io ● kubernetes.io/docs/concepts/security ● kubernetes.io/docs/tasks/administer-cluster ● sigstore.dev

tiffanyfayj Special thanks to: Andrés Vega Jake Sanders Jérôme Petazzoni Lewis Denham-Parry THANK YOU!