Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Securing Helm

Sponsored · Your Podcast. Everywhere. Effortlessly. Share. Educate. Inspire. Entertain. You do you. We'll handle the rest.
Avatar for Alex Khaerov Alex Khaerov
May 28, 2019
99

Securing Helm

Helm, the tool for managing Kubernetes packages called charts
Agenda
===
Helm… What is Helm?
Helm architecture
An attack vector
Securing Helm: RBAC, Release, Chart repo, gRPC…
Helm future and alternatives
Q&A

Avatar for Alex Khaerov

Alex Khaerov

May 28, 2019
Tweet

Transcript

  1. @hayorov Alex Khaerov company who I am Development Lead doing

    software development in the recent decade junior speaker - Python, Kubernetes committee member (Moscow Python, Helm Summit) huge fan of laptop stickers
  2. @hayorov Chainstack multi-cloud and multi-blockchain platform as a service based

    in Singapore * Alex Khaerov company who I am Development Lead doing software development in the recent decade junior speaker - Python, Kubernetes committee member (Moscow Python, Helm Summit) huge fan of laptop stickers
  3. @hayorov Helm… What is Helm? Helm architecture An attack vector

    Securing Helm: RBAC, Release, Chart repo, gRPC… Helm future and alternatives Q&A Agenda
  4. @hayorov 12k 1k* * GitHub starts, Jan 2019 the tool

    for managing Kubernetes packages called charts
  5. @hayorov nurtured by 12k 1k* * GitHub starts, Jan 2019

    the tool for managing Kubernetes packages called charts
  6. @hayorov September 11 - 12, 2019 
 Pakhuis de Zwijger


    Amsterdam, The Netherlands https://events.linuxfoundation.org/events/helm-summit-2019/ CFP is open – Apply now! | #helmsummit
  7. @hayorov Manage complexity Packaging Application lifecycle Application metadata Repositories Package

    dependencies Helm addresses several needs Parametrisation Templating
  8. @hayorov Manage complexity Packaging Application lifecycle Application metadata Repositories Package

    dependencies Helm addresses several needs Parametrisation Templating Deploy/config revisions Rollbacks Hooks Application probes
  9. @hayorov “Batteries included” Manage complexity Packaging Application lifecycle Application metadata

    Repositories Package dependencies Helm addresses several needs Parametrisation Templating Deploy/config revisions Rollbacks Hooks Application probes
  10. @hayorov “Batteries included” Manage complexity Packaging Application lifecycle Application metadata

    Repositories Package dependencies Helm addresses several needs Parametrisation Templating Deploy/config revisions Rollbacks Hooks Application probes CLI plugins
  11. @hayorov several angles from which someone might try to abuse

    Helm/Tiller: An attack vector • A privileged API user, such as a cluster-admin.
  12. @hayorov several angles from which someone might try to abuse

    Helm/Tiller: An attack vector • A privileged API user, such as a cluster-admin.
  13. @hayorov • A low-privilege API user, such as a user

    who has been restricted to a single namespace using RBAC several angles from which someone might try to abuse Helm/Tiller: An attack vector • A privileged API user, such as a cluster-admin.
  14. @hayorov • A low-privilege API user, such as a user

    who has been restricted to a single namespace using RBAC several angles from which someone might try to abuse Helm/Tiller: An attack vector • A privileged API user, such as a cluster-admin.
  15. @hayorov • A low-privilege API user, such as a user

    who has been restricted to a single namespace using RBAC • An in-cluster process, such as a compromised webserver. several angles from which someone might try to abuse Helm/Tiller: An attack vector • A privileged API user, such as a cluster-admin.
  16. @hayorov • A low-privilege API user, such as a user

    who has been restricted to a single namespace using RBAC • An in-cluster process, such as a compromised webserver. several angles from which someone might try to abuse Helm/Tiller: An attack vector • A privileged API user, such as a cluster-admin.
  17. @hayorov • A hostile chart author can create a chart

    containing unexpected resources. These can either escalate one of the other groups above, or run other malicious jobs. • A low-privilege API user, such as a user who has been restricted to a single namespace using RBAC • An in-cluster process, such as a compromised webserver. several angles from which someone might try to abuse Helm/Tiller: An attack vector • A privileged API user, such as a cluster-admin.
  18. @hayorov Helm CLI Chart Repo K8s cluster kube-system ns microservice

    ns Kube-api
 server Tiller-deploy svc Release configmap Microservice pod release ns(s) Release deployment Tiller-deploy pod Service Account Release svc Release configmap Kubeconfig Helm architecture
  19. @hayorov Helm CLI Chart Repo K8s cluster kube-system ns microservice

    ns Kube-api
 server Tiller-deploy svc Release configmap Microservice pod release ns(s) Release deployment Tiller-deploy pod Service Account Release svc Release configmap Kubeconfig Helm architecture
  20. @hayorov Helm CLI Chart Repo K8s cluster kube-system ns microservice

    ns Kube-api
 server Tiller-deploy svc Release configmap Microservice pod release ns(s) Release deployment Tiller-deploy pod Service Account Release svc Release configmap Kubeconfig Helm architecture
  21. @hayorov Helm CLI Chart Repo K8s cluster kube-system ns microservice

    ns Kube-api
 server Tiller-deploy svc Release configmap Microservice pod release ns(s) Release deployment Tiller-deploy pod Service Account Release svc Release configmap Kubeconfig Helm architecture
  22. @hayorov Helm CLI Chart Repo K8s cluster kube-system ns microservice

    ns Kube-api
 server Tiller-deploy svc Release configmap Microservice pod release ns(s) Release deployment Tiller-deploy pod Service Account Release svc Release configmap Kubeconfig Helm architecture
  23. @hayorov Helm CLI Chart Repo K8s cluster kube-system ns microservice

    ns Kube-api
 server Tiller-deploy svc Release configmap Microservice pod release ns(s) Release deployment Tiller-deploy pod Service Account Release svc Release configmap Kubeconfig Helm architecture
  24. @hayorov Helm CLI Chart Repo K8s cluster kube-system ns microservice

    ns Kube-api
 server Tiller-deploy svc Release configmap Microservice pod release ns(s) Release deployment Tiller-deploy pod Service Account Release svc Release configmap Kubeconfig An attack vector
  25. @hayorov Helm CLI Chart Repo K8s cluster kube-system ns microservice

    ns Kube-api
 server Tiller-deploy svc Release configmap Microservice pod release ns(s) Release deployment Tiller-deploy pod Service Account Release svc Release configmap HTTP Kubeconfig An attack vector
  26. @hayorov Helm CLI Chart Repo K8s cluster kube-system ns microservice

    ns Kube-api
 server Tiller-deploy svc Release configmap Microservice pod release ns(s) Release deployment Tiller-deploy pod Service Account Release svc Release configmap HTTP Kubeconfig gRPC An attack vector
  27. @hayorov Helm CLI Chart Repo K8s cluster kube-system ns microservice

    ns Kube-api
 server Tiller-deploy svc Release configmap Microservice pod release ns(s) Release deployment Tiller-deploy pod Service Account Release svc Release configmap HTTP Kubeconfig gRPC gRPC An attack vector
  28. @hayorov • Turn RBAC on • Tiller uses the default

    service account 
 in a namespace Kubernetes RBAC Tiller-deploy svc Tiller-deploy pod Service Account Release configmap (RBAC) Default Service Account Rest API
  29. @hayorov • Turn RBAC on • Tiller uses the default

    service account 
 in a namespace Kubernetes RBAC Tiller-deploy svc Tiller-deploy pod Service Account Release configmap (RBAC) Default Service Account Rest API advocacy site for RBAC https://rbac.dev/
  30. @hayorov • Turn RBAC on • Tiller uses the default

    service account 
 in a namespace Kubernetes RBAC Tiller-deploy svc Tiller-deploy pod Service Account Release configmap (RBAC) Default Service Account Rest API
  31. @hayorov • Turn RBAC on • Tiller uses the default

    service account 
 in a namespace Kubernetes RBAC Tiller-deploy svc Tiller-deploy pod Service Account Release configmap (RBAC) Rest API
  32. @hayorov Role RoleBinding • Turn RBAC on • Tiller uses

    the default service account 
 in a namespace Kubernetes RBAC Tiller-deploy svc Tiller-deploy pod Service Account Release configmap (RBAC) Rest API
  33. @hayorov Role RoleBinding • Turn RBAC on • Tiller uses

    the default service account 
 in a namespace Kubernetes RBAC • helm init does not create the associated
 ServiceAccount/Roles/RoleBindings ☝ Tiller-deploy svc Tiller-deploy pod Service Account Release configmap (RBAC) Rest API
  34. @hayorov Kubernetes RBAC Helm CLI Kubeconfig gRPC K8s cluster kube-system

    ns Kube-api
 server Tiller-deploy svc Release configmap Tiller-deploy pod Service Account Helm CLI Kubeconfig Cluster Admin X Team Kube-api
 server Tiller-deploy svc Release configmap Tiller-deploy pod Service Account Release deployment Release svc Release configmap RestAPI RBAC gRPC RestAPI RBAC • No multi-tenancy support • Solution - Multi-Tiller installations • Per developer, per team, per environment X Team ns ?
  35. @hayorov Kubernetes RBAC Helm CLI Kubeconfig gRPC K8s cluster kube-system

    ns Kube-api
 server Tiller-deploy svc Release configmap Tiller-deploy pod Service Account Helm CLI Kubeconfig Cluster Admin X Team Kube-api
 server Tiller-deploy svc Release configmap Tiller-deploy pod Service Account Release deployment Release svc Release configmap RestAPI RBAC gRPC RestAPI RBAC • No multi-tenancy support • Solution - Multi-Tiller installations • Per developer, per team, per environment X Team ns
  36. @hayorov • By default Tiller stores releases
 information in ConfigMaps

    Release information Tiller-deploy svc Tiller-deploy pod Service Account Role RoleBinding Release
 secrets RBAC Release
 secrets Release
 configmap
  37. @hayorov • By default Tiller stores releases
 information in ConfigMaps

    Release information Tiller-deploy svc Tiller-deploy pod Service Account Role RoleBinding Release
 secrets RBAC Release
 secrets Release
 configmap
  38. @hayorov • By default Tiller stores releases
 information in ConfigMaps

    Release information Tiller-deploy svc Tiller-deploy pod Service Account Role RoleBinding Release
 secrets RBAC Release
 secrets Release
 configmap Is this the 1MB limit?
  39. @hayorov • Use HTTPS always • Publish signed charts •

    Helm client supports TLS • Chartmuseum supports basic auth • helm-gcs plugin with GCP auth Chart Repos Helm CLI Signed Chart Repo fetch Chart Repo
  40. @hayorov • Use HTTPS always • Publish signed charts •

    Helm client supports TLS • Chartmuseum supports basic auth • helm-gcs plugin with GCP auth Chart Repos Helm CLI Signed Chart Repo fetch HTTPS mTLS or basic auth
  41. @hayorov gRPC API Helm CLI Kubeconfig gRPC • Tiller supports

    TLS on gRPC Tiller-deploy svc Tiller-deploy pod Service Account Role RoleBinding Release configmap RBAC Tiller-deploy svc
  42. @hayorov gRPC API Helm CLI Kubeconfig gRPC • Tiller supports

    TLS on gRPC Tiller-deploy svc Tiller-deploy pod Service Account Role RoleBinding Release configmap RBAC TLS
  43. @hayorov Secured Helm Helm CLI Chart Repo K8s cluster kube-system

    ns microservice ns Kube-api
 server Tiller-deploy svc Release secret Microservice pod release ns(s) Release deployment Release svc Release configmap fetch Kubeconfig HTTPS gRPC gRPC Service Account HTTPS TLS or basic auth mTLS RBAC mTLS Tiller-deploy pod
  44. @hayorov Secured Helm Helm CLI Chart Repo K8s cluster kube-system

    ns microservice ns Kube-api
 server Tiller-deploy svc Release secret Microservice pod release ns(s) Release deployment Release svc Release configmap fetch Kubeconfig HTTPS gRPC gRPC Service Account HTTPS TLS or basic auth mTLS RBAC mTLS Tiller-deploy pod instruction at bit.ly/helm-secure
  45. @hayorov Secured Helm Helm CLI Chart Repo K8s cluster kube-system

    ns microservice ns Kube-api
 server Tiller-deploy svc Release secret Microservice pod release ns(s) Release deployment Release svc Release configmap fetch Kubeconfig HTTPS gRPC gRPC Service Account HTTPS TLS or basic auth mTLS RBAC mTLS Tiller-deploy pod
  46. @hayorov My Cluster Cloud Provider Open Service Broker API integration

    What are Service Brokers? Managed MySQL manually
  47. @hayorov My Cluster Cloud Provider Open Service Broker API integration

    What are Service Brokers? Managed MySQL Service Broker
  48. @hayorov My Cluster Cloud Provider Open Service Broker API integration

    What are Service Brokers? Managed MySQL Service Broker OSBA
  49. @hayorov My Cluster Cloud Provider Open Service Broker API integration

    What are Service Brokers? Managed MySQL Tier: Basic Service Broker OSBA
  50. @hayorov My Cluster Cloud Provider Open Service Broker API integration

    What are Service Brokers? Managed MySQL Tier: Basic Service Broker OSBA DB credentials
  51. @hayorov My Cluster Cloud Provider Open Service Broker API integration

    What are Service Brokers? Managed MySQL Tier: Basic Service Broker OSBA DB credentials
  52. @hayorov My Cluster Cloud Provider Open Service Broker API integration

    What are Service Brokers? Managed MySQL Tier: Basic Service Broker OSBA DB credentials Helm glues Service Broker and charts that consume OSB resources
  53. @hayorov Repositories on GCS ChartMuseum de-facto is a standard helm-gcs

    is a plugin that allows to manage private repos on GCS Authentification using: • application default credentials • service account 
 (via the global flag)

  54. @hayorov Any alternatives? • Ksonnet, Metaparticle • Your configuration mngt

    system 
 (ansible, terraform, chef …) • Operator Framework (2k ⭐) Helm add-ons: • Draft, Scaffold Alternatives
  55. @hayorov Any alternatives? • Ksonnet, Metaparticle • Your configuration mngt

    system 
 (ansible, terraform, chef …) • Operator Framework (2k ⭐) Helm add-ons: • Draft, Scaffold Alternatives more control more k8s native
  56. @hayorov Any alternatives? • Ksonnet, Metaparticle • Your configuration mngt

    system 
 (ansible, terraform, chef …) • Operator Framework (2k ⭐) Helm add-ons: • Draft, Scaffold Alternatives more control more k8s native v2
  57. @hayorov Any alternatives? • Ksonnet, Metaparticle • Your configuration mngt

    system 
 (ansible, terraform, chef …) • Operator Framework (2k ⭐) Helm add-ons: • Draft, Scaffold Alternatives more control more k8s native v3 v2
  58. @hayorov Any alternatives? • Ksonnet, Metaparticle • Your configuration mngt

    system 
 (ansible, terraform, chef …) • Operator Framework (2k ⭐) Helm add-ons: • Draft, Scaffold Alternatives more control more k8s native v3 v2 K Ksonnet, Metaparticle
  59. @hayorov Any alternatives? • Ksonnet, Metaparticle • Your configuration mngt

    system 
 (ansible, terraform, chef …) • Operator Framework (2k ⭐) Helm add-ons: • Draft, Scaffold Alternatives more control more k8s native v3 Y Your Cfg
 management v2 K Ksonnet, Metaparticle
  60. @hayorov Any alternatives? • Ksonnet, Metaparticle • Your configuration mngt

    system 
 (ansible, terraform, chef …) • Operator Framework (2k ⭐) Helm add-ons: • Draft, Scaffold Alternatives more control more k8s native v3 Operator Framework Y Your Cfg
 management v2 K Ksonnet, Metaparticle
  61. @hayorov Any alternatives? • Ksonnet, Metaparticle • Your configuration mngt

    system 
 (ansible, terraform, chef …) • Operator Framework (2k ⭐) Helm add-ons: • Draft, Scaffold Alternatives more control more k8s native v3 Operator Framework Y Your Cfg
 management v2 A Draft, Skaffold K Ksonnet, Metaparticle
  62. @hayorov Helm 3 is the next big thing • Simplified

    client only architecture (no more Tiller) • State storage based of Release object (based on CRD) • Initial support for OCI repositories • (optional) Embedded Lua engine for scripting • Schematised values files (using JSONSchema) • Single event-driven model
 Current status: 3.0.0-alpha.1 released The future of Helm 3