Securing a Kubernetes Distribution - Container Security Summit

9a5b28c1bf706b9138017bf0fd23ac45?s=47 Eric Chiang
January 26, 2018

Securing a Kubernetes Distribution - Container Security Summit


Eric Chiang

January 26, 2018


  1. Container Infrastructure Security Session 3 Talk: Securing a Kubernetes Distribution

    Eric Chiang, CoreOS/Red Hat 11:30 AM Talk: Pluggable Identity For Container Infrastructure Security Somik Behera, Mesosphere 1:00 PM
  2. Session 3 Panel: Identity, Secrets & Trust 1:30 PM Panel:

    Monitoring & Logging 2:00 PM Container Infrastructure Security
  3. Securing a Kubernetes Distribution Eric Chiang, CoreOS/Red Hat 11:30 AM

    - 12:00 PM
  4. 4

  5. Agenda Overview of Kubernetes security Operational realities of a distribution

    Pain points and opportunity for improvement 5
  6. Kubernetes security A quick taste 6

  7. 7

  8. Kubernetes security Hacking and Hardening Kubernetes Clusters by Example -

    Brad Geesaman Micro-services are easier to exploit than the kernel 8
  9. Example: the pod network In a (default) cluster every container

    can access every other container • Metrics APIs (cAdvisor, Prometheus) • Escalating endpoints (Kubernetes Dashboards) • Backend databases 9
  10. Example: cloud metadata services In a (default) cluster every container

    can access cloud metadata services (AWS, GCE, Azure) 10
  11. Example: cloud metadata services $ URL="" $ wget -- -qO-

    $URL/user-data # Get provisioning data $ wget -- -qO- $URL/meta-data/iam/security-credentials/$ROLE 11
  12. 12

  13. The list goes on OS/app vulnerabilities Container registries Worker credential

    management 13
  14. Modes of operation 14

  15. Modes of operation for CoreOS Hard multi-tenancy vs. soft multi-tenancy

    Clusters we run vs. clusters our customers run 15
  16. Quay docker builds Hard multi-tenancy Any user may submit a

    Dockerfile, we’ll build it Scary, but straight forward 16
  17. Quay docker builds 17

  18. Tectonic Soft multi-tenancy Identity provider integrations through Dex (LDAP, Google

    Accounts, GitHub, etc.) Customers are the admins Users expect API access and the ability to run workloads 18
  19. We rely on admins to enforce security 19

  20. 20

  21. Example: limit metadata service Admins wants to prevent users from

    accessing cloud metadata services Network policy? 21
  22. Example: limit metadata service Kubernetes exposes policy through the API

    Default RBAC allows regular users to modify network policy Clusters require third party capable of enforcing global network policy Easy to get around with host networking, need to also enforce pod security policies 22
  23. Kubernetes administration Admins must understand the nuances of every Kubernetes

    API resource • How can you restrict access? (RBAC, PodSecurityPolicy, NetworkPolicy, ResourceQuota) • What resources can be used to trivially escalate? • What access can safely be given to non-privileged users? 23
  24. Pain points What makes this hard for admins? 24

  25. Build first, secure later 25

  26. Defaults, defaults, defaults Authorization - allow all Network policy -

    allow all Kubelet API - allow unauthenticated access etc. 26
  27. “The Security Capability Gap”* RBAC - basic API access control

    (1.6) Node authorization - compromised worker can't escalate (1.7) Pod security policies - preventing privileged workloads (1.8) Network policy - egress (1.8) Audit logs (1.8) Secrets (?) * Hacking and Hardening Kubernetes Clusters by Example - Brad Geesaman 27
  28. User disruption Users get used to allow all Every security

    change is a breaking change (or architectural shift) Best practices are constantly evolving 28
  29. User disruption Apps build before policy capabilities mature have no

    direction for best practice Proposal: Authentication and Authorization support for Tiller (kubernetes/helm#1918) 29
  30. Complex systems require complex policy An application can require three

    or four different kind of policy • Network access • Kubernetes API access • Quota 30
  31. The future 31

  32. Security capabilities are catching up RBAC is standard, NetworkPolicy is

    becoming an expectation PodSecurityPolicies will be the next thing to roll out External secret integration through container identity 32
  33. Best practices for multi-tenancy Better docs • RBAC, PodSecurityPolicies, ResourceQuota,

    NetworkPolicy Better defaults (e.g. default RBAC roles) Better defined security model 33
  34. Chop wood, carry water “I completely, totally, 100% believe that

    moving from ‘single use mode’ to ‘multi-user’ clusters is the right thing to do, and we should do it ASAP (last year, if possible). It *is* going to hurt a LOT of people, and we need to respect that and allow users to opt-out for quite a while. We also need REALLY good docs, and error messages that can be googled along with SEO'ed solutions to those error messages. Writing code is easy, rolling it out is hard.” - Tim Hockin (kubernetes/kubernetes#39722) 34
  35. Thank you.

  36. LUNCH + Networking 36 12:15-1:00 PM