Kubernetes Security Challenges

Kops Kubernetes Operations

Kops - templating - default cluster template

Kops - templating - default values

Kops - templating - cluster specific values

Kops - templating - make Targets!

Kops - vpc / subnet management Why not use kops defaults? Routing Tables - VPC Peering - Direct Connections - Egress control

Terraform tags!

Kops - templating - cluster specific values

Kops - clusterSpec

Kops - Edge Nodes Why? - Compliance requirements (WAF) How? - Dedicated nodePool / instanceGroup - AWS LB limited to edge nodes

Kops - Edge Nodes Dedicated nodePool / instanceGroup:

Kops - Edge Nodes AWS LB limited to edge nodes: Problem: - Kubernetes service type LoadBalancer (targets all worker nodes) Solution: - https://github.com/zalando-incubator/kube-ingress-aws-controller CUSTOM_FILTERS

Kops - Edge Nodes AWS LB limited to edge nodes:

Kops - Edge Nodes AWS LB limited to edge nodes:

Prevent misuse of privileges Use Authentication and Authorization - Authentication: SSO (onboarding / offboarding / consistency ) - OpenID Connect (Google / Dex / … ) - Exec (Heptio Authenticator) - Authorization & Logs - RBAC is mandatory - Audit Event Logging (k8s 1.9+) - Audit2rbac: Generates RBAC role and binding objects based on audit log of API requests made by a user - Admission Controls (webhooks after Auth{n,z}) - Image Whitelisting - Workload mutation

SSO: AWS Authenticator Kops bootstrapping Overview: - IAM Role - TLS (self-signed CA) - kube-apiserver webhook configuration - Authenticator daemonset on masters How? Terraform + Kops hooks and addons

Kops bootstrapping: AWS Authenticator - IAM role (Terraform)

Kops bootstrapping: AWS Authenticator - TLS (Terraform)

Kops bootstrapping: AWS Authenticator - kube-apiserver webhook configuration file (Terraform)

Kops bootstrapping: AWS Authenticator - Node bootstrapping hooks (kops clusterSpec.hooks) (host = CoreOS) Copy webhook config and TLS

Kops bootstrapping: AWS Authenticator - Kube Addon bootstrapping (kops clusterSpec.addons)

Kops bootstrapping: AWS Authenticator - kops channels manifest:

Kops bootstrapping: AWS Authenticator - Templated addons (Terraform)

Kubernetes Components Isolate from external access … - API server - Kubelet - Etcd - Firewall between master and worker nodes - Overlay network (Flannel / Calico / Romana / … ) & Etcd And use TLS (etcd / kubelet bootstrapping / …) https://kubernetes.io/blog/2018/07/18/11-ways-not-to-get-hacked/

When to Opt for Custom Deployment? - Responsibility of maintaining the clusters lies solely with the customer - Master symmetric keys need to be manually rotated https://www.twistlock.com/2017/08/02/kubernetes-secrets-encryption/ - Etcd isolation & TLS configuration https://coreos.com/etcd/docs/latest/op-guide/security.html - Node bootstrapping & TLS configuration https://medium.com/@toddrosner/kubernetes-tls-bootstrapping-cf203776abc7 - Trade-offs - Bleeding edge - Choice (Machine Configuration, Operating Systems, Storage Backends, Network Plugins and HA configuration)

Application Lifecycle (containers) ● Security shift Left ● Container Image security ● Container Registry Management ● Immutability Image credit: Aquasec “One of the characteristics of containers is that they’re very predictable, or they should be, This allows you to do security in a more predictable, automated way.” - John Morello CTO Twistlock

Feature re-cap (container orchestrators) - Image scanning - Registry scanning - Admission hooks to only run allowed images - Process whitelisting in containers - Binary whitelisting - Node protection - RBAC with least privilege approach

Security Platforms ● Aqua Sec ● Twistlock ● Sysdig Secure ● ... Image credit: Aquasec

Thank you Questions? More extensive overview: https://kubernetes.io/blog/2018/07/18/11-ways-not-to-get-hacked/