Container Infrastructure Security Session 3 Talk: Securing a Kubernetes Distribution Eric Chiang, CoreOS/Red Hat 11:30 AM Talk: Pluggable Identity For Container Infrastructure Security Somik Behera, Mesosphere 1:00 PM 

Session 3 Panel: Identity, Secrets & Trust 1:30 PM Panel: Monitoring & Logging 2:00 PM Container Infrastructure Security 

Securing a Kubernetes Distribution Eric Chiang, CoreOS/Red Hat 11:30 AM - 12:00 PM 

4 

Agenda Overview of Kubernetes security Operational realities of a distribution Pain points and opportunity for improvement 5 

Kubernetes security A quick taste 6 

7 

Kubernetes security Hacking and Hardening Kubernetes Clusters by Example - Brad Geesaman Micro-services are easier to exploit than the kernel 8 

Example: the pod network In a (default) cluster every container can access every other container ● Metrics APIs (cAdvisor, Prometheus) ● Escalating endpoints (Kubernetes Dashboards) ● Backend databases 9 

Example: cloud metadata services In a (default) cluster every container can access cloud metadata services (AWS, GCE, Azure) 10 

Example: cloud metadata services $ URL="http://169.254.169.254/latest" $ wget -- -qO- $URL/user-data # Get provisioning data $ wget -- -qO- $URL/meta-data/iam/security-credentials/$ROLE 11 

12 

The list goes on OS/app vulnerabilities Container registries Worker credential management 13 

Modes of operation 14 

Modes of operation for CoreOS Hard multi-tenancy vs. soft multi-tenancy Clusters we run vs. clusters our customers run 15 

Quay docker builds Hard multi-tenancy Any user may submit a Dockerfile, we’ll build it Scary, but straight forward 16 

Quay docker builds 17 

Tectonic Soft multi-tenancy Identity provider integrations through Dex (LDAP, Google Accounts, GitHub, etc.) Customers are the admins Users expect API access and the ability to run workloads 18 

We rely on admins to enforce security 19 

20 https://twitter.com/JGamblin/status/966084475917500416 

Example: limit metadata service Admins wants to prevent users from accessing cloud metadata services Network policy? 21 

Example: limit metadata service Kubernetes exposes policy through the API Default RBAC allows regular users to modify network policy Clusters require third party capable of enforcing global network policy Easy to get around with host networking, need to also enforce pod security policies 22 https://www.projectcalico.org/ 

Kubernetes administration Admins must understand the nuances of every Kubernetes API resource ● How can you restrict access? (RBAC, PodSecurityPolicy, NetworkPolicy, ResourceQuota) ● What resources can be used to trivially escalate? ● What access can safely be given to non-privileged users? 23 

Pain points What makes this hard for admins? 24 

Build first, secure later 25 

Defaults, defaults, defaults Authorization - allow all Network policy - allow all Kubelet API - allow unauthenticated access etc. 26 

“The Security Capability Gap”* RBAC - basic API access control (1.6) Node authorization - compromised worker can't escalate (1.7) Pod security policies - preventing privileged workloads (1.8) Network policy - egress (1.8) Audit logs (1.8) Secrets (?) * Hacking and Hardening Kubernetes Clusters by Example - Brad Geesaman 27 

User disruption Users get used to allow all Every security change is a breaking change (or architectural shift) Best practices are constantly evolving 28 

User disruption Apps build before policy capabilities mature have no direction for best practice Proposal: Authentication and Authorization support for Tiller (kubernetes/helm#1918) 29 

Complex systems require complex policy An application can require three or four different kind of policy ● Network access ● Kubernetes API access ● Quota 30 

The future 31 

Security capabilities are catching up RBAC is standard, NetworkPolicy is becoming an expectation PodSecurityPolicies will be the next thing to roll out External secret integration through container identity 32 

Best practices for multi-tenancy Better docs ● RBAC, PodSecurityPolicies, ResourceQuota, NetworkPolicy Better defaults (e.g. default RBAC roles) Better defined security model 33 

Chop wood, carry water “I completely, totally, 100% believe that moving from ‘single use mode’ to ‘multi-user’ clusters is the right thing to do, and we should do it ASAP (last year, if possible). It *is* going to hurt a LOT of people, and we need to respect that and allow users to opt-out for quite a while. We also need REALLY good docs, and error messages that can be googled along with SEO'ed solutions to those error messages. Writing code is easy, rolling it out is hard.” - Tim Hockin (kubernetes/kubernetes#39722) 34 

Thank you. 

LUNCH + Networking 36 12:15-1:00 PM