Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Securing Helm

Alex Khaerov
May 28, 2019
72

Securing Helm

Helm, the tool for managing Kubernetes packages called charts
Agenda
===
Helm… What is Helm?
Helm architecture
An attack vector
Securing Helm: RBAC, Release, Chart repo, gRPC…
Helm future and alternatives
Q&A

Alex Khaerov

May 28, 2019
Tweet

Transcript

  1. @hayorov Alex Khaerov company who I am Development Lead doing

    software development in the recent decade junior speaker - Python, Kubernetes committee member (Moscow Python, Helm Summit) huge fan of laptop stickers
  2. @hayorov Chainstack multi-cloud and multi-blockchain platform as a service based

    in Singapore * Alex Khaerov company who I am Development Lead doing software development in the recent decade junior speaker - Python, Kubernetes committee member (Moscow Python, Helm Summit) huge fan of laptop stickers
  3. @hayorov Helm… What is Helm? Helm architecture An attack vector

    Securing Helm: RBAC, Release, Chart repo, gRPC… Helm future and alternatives Q&A Agenda
  4. @hayorov 12k 1k* * GitHub starts, Jan 2019 the tool

    for managing Kubernetes packages called charts
  5. @hayorov nurtured by 12k 1k* * GitHub starts, Jan 2019

    the tool for managing Kubernetes packages called charts
  6. @hayorov September 11 - 12, 2019 
 Pakhuis de Zwijger


    Amsterdam, The Netherlands https://events.linuxfoundation.org/events/helm-summit-2019/ CFP is open – Apply now! | #helmsummit
  7. @hayorov Manage complexity Packaging Application lifecycle Application metadata Repositories Package

    dependencies Helm addresses several needs Parametrisation Templating
  8. @hayorov Manage complexity Packaging Application lifecycle Application metadata Repositories Package

    dependencies Helm addresses several needs Parametrisation Templating Deploy/config revisions Rollbacks Hooks Application probes
  9. @hayorov “Batteries included” Manage complexity Packaging Application lifecycle Application metadata

    Repositories Package dependencies Helm addresses several needs Parametrisation Templating Deploy/config revisions Rollbacks Hooks Application probes
  10. @hayorov “Batteries included” Manage complexity Packaging Application lifecycle Application metadata

    Repositories Package dependencies Helm addresses several needs Parametrisation Templating Deploy/config revisions Rollbacks Hooks Application probes CLI plugins
  11. @hayorov several angles from which someone might try to abuse

    Helm/Tiller: An attack vector • A privileged API user, such as a cluster-admin.
  12. @hayorov several angles from which someone might try to abuse

    Helm/Tiller: An attack vector • A privileged API user, such as a cluster-admin.
  13. @hayorov • A low-privilege API user, such as a user

    who has been restricted to a single namespace using RBAC several angles from which someone might try to abuse Helm/Tiller: An attack vector • A privileged API user, such as a cluster-admin.
  14. @hayorov • A low-privilege API user, such as a user

    who has been restricted to a single namespace using RBAC several angles from which someone might try to abuse Helm/Tiller: An attack vector • A privileged API user, such as a cluster-admin.
  15. @hayorov • A low-privilege API user, such as a user

    who has been restricted to a single namespace using RBAC • An in-cluster process, such as a compromised webserver. several angles from which someone might try to abuse Helm/Tiller: An attack vector • A privileged API user, such as a cluster-admin.
  16. @hayorov • A low-privilege API user, such as a user

    who has been restricted to a single namespace using RBAC • An in-cluster process, such as a compromised webserver. several angles from which someone might try to abuse Helm/Tiller: An attack vector • A privileged API user, such as a cluster-admin.
  17. @hayorov • A hostile chart author can create a chart

    containing unexpected resources. These can either escalate one of the other groups above, or run other malicious jobs. • A low-privilege API user, such as a user who has been restricted to a single namespace using RBAC • An in-cluster process, such as a compromised webserver. several angles from which someone might try to abuse Helm/Tiller: An attack vector • A privileged API user, such as a cluster-admin.
  18. @hayorov Helm CLI Chart Repo K8s cluster kube-system ns microservice

    ns Kube-api
 server Tiller-deploy svc Release configmap Microservice pod release ns(s) Release deployment Tiller-deploy pod Service Account Release svc Release configmap Kubeconfig Helm architecture
  19. @hayorov Helm CLI Chart Repo K8s cluster kube-system ns microservice

    ns Kube-api
 server Tiller-deploy svc Release configmap Microservice pod release ns(s) Release deployment Tiller-deploy pod Service Account Release svc Release configmap Kubeconfig Helm architecture
  20. @hayorov Helm CLI Chart Repo K8s cluster kube-system ns microservice

    ns Kube-api
 server Tiller-deploy svc Release configmap Microservice pod release ns(s) Release deployment Tiller-deploy pod Service Account Release svc Release configmap Kubeconfig Helm architecture
  21. @hayorov Helm CLI Chart Repo K8s cluster kube-system ns microservice

    ns Kube-api
 server Tiller-deploy svc Release configmap Microservice pod release ns(s) Release deployment Tiller-deploy pod Service Account Release svc Release configmap Kubeconfig Helm architecture
  22. @hayorov Helm CLI Chart Repo K8s cluster kube-system ns microservice

    ns Kube-api
 server Tiller-deploy svc Release configmap Microservice pod release ns(s) Release deployment Tiller-deploy pod Service Account Release svc Release configmap Kubeconfig Helm architecture
  23. @hayorov Helm CLI Chart Repo K8s cluster kube-system ns microservice

    ns Kube-api
 server Tiller-deploy svc Release configmap Microservice pod release ns(s) Release deployment Tiller-deploy pod Service Account Release svc Release configmap Kubeconfig Helm architecture
  24. @hayorov Helm CLI Chart Repo K8s cluster kube-system ns microservice

    ns Kube-api
 server Tiller-deploy svc Release configmap Microservice pod release ns(s) Release deployment Tiller-deploy pod Service Account Release svc Release configmap Kubeconfig An attack vector
  25. @hayorov Helm CLI Chart Repo K8s cluster kube-system ns microservice

    ns Kube-api
 server Tiller-deploy svc Release configmap Microservice pod release ns(s) Release deployment Tiller-deploy pod Service Account Release svc Release configmap HTTP Kubeconfig An attack vector
  26. @hayorov Helm CLI Chart Repo K8s cluster kube-system ns microservice

    ns Kube-api
 server Tiller-deploy svc Release configmap Microservice pod release ns(s) Release deployment Tiller-deploy pod Service Account Release svc Release configmap HTTP Kubeconfig gRPC An attack vector
  27. @hayorov Helm CLI Chart Repo K8s cluster kube-system ns microservice

    ns Kube-api
 server Tiller-deploy svc Release configmap Microservice pod release ns(s) Release deployment Tiller-deploy pod Service Account Release svc Release configmap HTTP Kubeconfig gRPC gRPC An attack vector
  28. @hayorov • Turn RBAC on • Tiller uses the default

    service account 
 in a namespace Kubernetes RBAC Tiller-deploy svc Tiller-deploy pod Service Account Release configmap (RBAC) Default Service Account Rest API
  29. @hayorov • Turn RBAC on • Tiller uses the default

    service account 
 in a namespace Kubernetes RBAC Tiller-deploy svc Tiller-deploy pod Service Account Release configmap (RBAC) Default Service Account Rest API advocacy site for RBAC https://rbac.dev/
  30. @hayorov • Turn RBAC on • Tiller uses the default

    service account 
 in a namespace Kubernetes RBAC Tiller-deploy svc Tiller-deploy pod Service Account Release configmap (RBAC) Default Service Account Rest API
  31. @hayorov • Turn RBAC on • Tiller uses the default

    service account 
 in a namespace Kubernetes RBAC Tiller-deploy svc Tiller-deploy pod Service Account Release configmap (RBAC) Rest API
  32. @hayorov Role RoleBinding • Turn RBAC on • Tiller uses

    the default service account 
 in a namespace Kubernetes RBAC Tiller-deploy svc Tiller-deploy pod Service Account Release configmap (RBAC) Rest API
  33. @hayorov Role RoleBinding • Turn RBAC on • Tiller uses

    the default service account 
 in a namespace Kubernetes RBAC • helm init does not create the associated
 ServiceAccount/Roles/RoleBindings ☝ Tiller-deploy svc Tiller-deploy pod Service Account Release configmap (RBAC) Rest API
  34. @hayorov Kubernetes RBAC Helm CLI Kubeconfig gRPC K8s cluster kube-system

    ns Kube-api
 server Tiller-deploy svc Release configmap Tiller-deploy pod Service Account Helm CLI Kubeconfig Cluster Admin X Team Kube-api
 server Tiller-deploy svc Release configmap Tiller-deploy pod Service Account Release deployment Release svc Release configmap RestAPI RBAC gRPC RestAPI RBAC • No multi-tenancy support • Solution - Multi-Tiller installations • Per developer, per team, per environment X Team ns ?
  35. @hayorov Kubernetes RBAC Helm CLI Kubeconfig gRPC K8s cluster kube-system

    ns Kube-api
 server Tiller-deploy svc Release configmap Tiller-deploy pod Service Account Helm CLI Kubeconfig Cluster Admin X Team Kube-api
 server Tiller-deploy svc Release configmap Tiller-deploy pod Service Account Release deployment Release svc Release configmap RestAPI RBAC gRPC RestAPI RBAC • No multi-tenancy support • Solution - Multi-Tiller installations • Per developer, per team, per environment X Team ns
  36. @hayorov • By default Tiller stores releases
 information in ConfigMaps

    Release information Tiller-deploy svc Tiller-deploy pod Service Account Role RoleBinding Release
 secrets RBAC Release
 secrets Release
 configmap
  37. @hayorov • By default Tiller stores releases
 information in ConfigMaps

    Release information Tiller-deploy svc Tiller-deploy pod Service Account Role RoleBinding Release
 secrets RBAC Release
 secrets Release
 configmap
  38. @hayorov • By default Tiller stores releases
 information in ConfigMaps

    Release information Tiller-deploy svc Tiller-deploy pod Service Account Role RoleBinding Release
 secrets RBAC Release
 secrets Release
 configmap Is this the 1MB limit?
  39. @hayorov • Use HTTPS always • Publish signed charts •

    Helm client supports TLS • Chartmuseum supports basic auth • helm-gcs plugin with GCP auth Chart Repos Helm CLI Signed Chart Repo fetch Chart Repo
  40. @hayorov • Use HTTPS always • Publish signed charts •

    Helm client supports TLS • Chartmuseum supports basic auth • helm-gcs plugin with GCP auth Chart Repos Helm CLI Signed Chart Repo fetch HTTPS mTLS or basic auth
  41. @hayorov gRPC API Helm CLI Kubeconfig gRPC • Tiller supports

    TLS on gRPC Tiller-deploy svc Tiller-deploy pod Service Account Role RoleBinding Release configmap RBAC Tiller-deploy svc
  42. @hayorov gRPC API Helm CLI Kubeconfig gRPC • Tiller supports

    TLS on gRPC Tiller-deploy svc Tiller-deploy pod Service Account Role RoleBinding Release configmap RBAC TLS
  43. @hayorov Secured Helm Helm CLI Chart Repo K8s cluster kube-system

    ns microservice ns Kube-api
 server Tiller-deploy svc Release secret Microservice pod release ns(s) Release deployment Release svc Release configmap fetch Kubeconfig HTTPS gRPC gRPC Service Account HTTPS TLS or basic auth mTLS RBAC mTLS Tiller-deploy pod
  44. @hayorov Secured Helm Helm CLI Chart Repo K8s cluster kube-system

    ns microservice ns Kube-api
 server Tiller-deploy svc Release secret Microservice pod release ns(s) Release deployment Release svc Release configmap fetch Kubeconfig HTTPS gRPC gRPC Service Account HTTPS TLS or basic auth mTLS RBAC mTLS Tiller-deploy pod instruction at bit.ly/helm-secure
  45. @hayorov Secured Helm Helm CLI Chart Repo K8s cluster kube-system

    ns microservice ns Kube-api
 server Tiller-deploy svc Release secret Microservice pod release ns(s) Release deployment Release svc Release configmap fetch Kubeconfig HTTPS gRPC gRPC Service Account HTTPS TLS or basic auth mTLS RBAC mTLS Tiller-deploy pod
  46. @hayorov My Cluster Cloud Provider Open Service Broker API integration

    What are Service Brokers? Managed MySQL manually
  47. @hayorov My Cluster Cloud Provider Open Service Broker API integration

    What are Service Brokers? Managed MySQL Service Broker
  48. @hayorov My Cluster Cloud Provider Open Service Broker API integration

    What are Service Brokers? Managed MySQL Service Broker OSBA
  49. @hayorov My Cluster Cloud Provider Open Service Broker API integration

    What are Service Brokers? Managed MySQL Tier: Basic Service Broker OSBA
  50. @hayorov My Cluster Cloud Provider Open Service Broker API integration

    What are Service Brokers? Managed MySQL Tier: Basic Service Broker OSBA DB credentials
  51. @hayorov My Cluster Cloud Provider Open Service Broker API integration

    What are Service Brokers? Managed MySQL Tier: Basic Service Broker OSBA DB credentials
  52. @hayorov My Cluster Cloud Provider Open Service Broker API integration

    What are Service Brokers? Managed MySQL Tier: Basic Service Broker OSBA DB credentials Helm glues Service Broker and charts that consume OSB resources
  53. @hayorov Repositories on GCS ChartMuseum de-facto is a standard helm-gcs

    is a plugin that allows to manage private repos on GCS Authentification using: • application default credentials • service account 
 (via the global flag)

  54. @hayorov Any alternatives? • Ksonnet, Metaparticle • Your configuration mngt

    system 
 (ansible, terraform, chef …) • Operator Framework (2k ⭐) Helm add-ons: • Draft, Scaffold Alternatives
  55. @hayorov Any alternatives? • Ksonnet, Metaparticle • Your configuration mngt

    system 
 (ansible, terraform, chef …) • Operator Framework (2k ⭐) Helm add-ons: • Draft, Scaffold Alternatives more control more k8s native
  56. @hayorov Any alternatives? • Ksonnet, Metaparticle • Your configuration mngt

    system 
 (ansible, terraform, chef …) • Operator Framework (2k ⭐) Helm add-ons: • Draft, Scaffold Alternatives more control more k8s native v2
  57. @hayorov Any alternatives? • Ksonnet, Metaparticle • Your configuration mngt

    system 
 (ansible, terraform, chef …) • Operator Framework (2k ⭐) Helm add-ons: • Draft, Scaffold Alternatives more control more k8s native v3 v2
  58. @hayorov Any alternatives? • Ksonnet, Metaparticle • Your configuration mngt

    system 
 (ansible, terraform, chef …) • Operator Framework (2k ⭐) Helm add-ons: • Draft, Scaffold Alternatives more control more k8s native v3 v2 K Ksonnet, Metaparticle
  59. @hayorov Any alternatives? • Ksonnet, Metaparticle • Your configuration mngt

    system 
 (ansible, terraform, chef …) • Operator Framework (2k ⭐) Helm add-ons: • Draft, Scaffold Alternatives more control more k8s native v3 Y Your Cfg
 management v2 K Ksonnet, Metaparticle
  60. @hayorov Any alternatives? • Ksonnet, Metaparticle • Your configuration mngt

    system 
 (ansible, terraform, chef …) • Operator Framework (2k ⭐) Helm add-ons: • Draft, Scaffold Alternatives more control more k8s native v3 Operator Framework Y Your Cfg
 management v2 K Ksonnet, Metaparticle
  61. @hayorov Any alternatives? • Ksonnet, Metaparticle • Your configuration mngt

    system 
 (ansible, terraform, chef …) • Operator Framework (2k ⭐) Helm add-ons: • Draft, Scaffold Alternatives more control more k8s native v3 Operator Framework Y Your Cfg
 management v2 A Draft, Skaffold K Ksonnet, Metaparticle
  62. @hayorov Helm 3 is the next big thing • Simplified

    client only architecture (no more Tiller) • State storage based of Release object (based on CRD) • Initial support for OCI repositories • (optional) Embedded Lua engine for scripting • Schematised values files (using JSONSchema) • Single event-driven model
 Current status: 3.0.0-alpha.1 released The future of Helm 3