Securing Helm

320f3b558c773592bba16c976d1b28d1?s=47 Alex Khaerov
May 28, 2019
17

Securing Helm

Helm, the tool for managing Kubernetes packages called charts
Agenda
===
Helm… What is Helm?
Helm architecture
An attack vector
Securing Helm: RBAC, Release, Chart repo, gRPC…
Helm future and alternatives
Q&A

320f3b558c773592bba16c976d1b28d1?s=128

Alex Khaerov

May 28, 2019
Tweet

Transcript

  1. @hayorov # % ' ⛓

  2. Securing Helm Alex Khaerov hayorov

  3. @hayorov @hayorov Привет ✋

  4. @hayorov company who I am

  5. @hayorov Alex Khaerov company who I am Development Lead

  6. @hayorov Alex Khaerov company who I am Development Lead doing

    software development in the recent decade junior speaker - Python, Kubernetes committee member (Moscow Python, Helm Summit) huge fan of laptop stickers
  7. @hayorov Chainstack multi-cloud and multi-blockchain platform as a service based

    in Singapore * Alex Khaerov company who I am Development Lead doing software development in the recent decade junior speaker - Python, Kubernetes committee member (Moscow Python, Helm Summit) huge fan of laptop stickers
  8. @hayorov Helm… What is Helm? Helm architecture An attack vector

    Securing Helm: RBAC, Release, Chart repo, gRPC… Helm future and alternatives Q&A Agenda
  9. @hayorov

  10. @hayorov the tool for managing Kubernetes packages called charts

  11. @hayorov the tool for managing Kubernetes packages called charts

  12. @hayorov 12k 1k* * GitHub starts, Jan 2019 the tool

    for managing Kubernetes packages called charts
  13. @hayorov nurtured by 12k 1k* * GitHub starts, Jan 2019

    the tool for managing Kubernetes packages called charts
  14. @hayorov September 11 - 12, 2019 
 Pakhuis de Zwijger


    Amsterdam, The Netherlands https://events.linuxfoundation.org/events/helm-summit-2019/ CFP is open – Apply now! | #helmsummit
  15. @hayorov Helm addresses several needs

  16. @hayorov Packaging Helm addresses several needs

  17. @hayorov Manage complexity Packaging Helm addresses several needs

  18. @hayorov Manage complexity Packaging Application lifecycle Helm addresses several needs

  19. @hayorov Manage complexity Packaging Application lifecycle Application metadata Repositories Package

    dependencies Helm addresses several needs
  20. @hayorov Manage complexity Packaging Application lifecycle Application metadata Repositories Package

    dependencies Helm addresses several needs Parametrisation Templating
  21. @hayorov Manage complexity Packaging Application lifecycle Application metadata Repositories Package

    dependencies Helm addresses several needs Parametrisation Templating Deploy/config revisions Rollbacks Hooks Application probes
  22. @hayorov “Batteries included” Manage complexity Packaging Application lifecycle Application metadata

    Repositories Package dependencies Helm addresses several needs Parametrisation Templating Deploy/config revisions Rollbacks Hooks Application probes
  23. @hayorov “Batteries included” Manage complexity Packaging Application lifecycle Application metadata

    Repositories Package dependencies Helm addresses several needs Parametrisation Templating Deploy/config revisions Rollbacks Hooks Application probes CLI plugins
  24. @hayorov There are three important concepts

  25. @hayorov Chart There are three important concepts

  26. @hayorov Chart Config There are three important concepts

  27. @hayorov Chart Config Release There are three important concepts

  28. @hayorov Chart Config Release Config There are three important concepts

  29. @hayorov Helm architecture

  30. @hayorov Helm architecture kube-apiserver

  31. @hayorov Helm architecture Kubeconfig kube-apiserver

  32. @hayorov Helm architecture Helm CLI Kubeconfig kube-apiserver

  33. @hayorov Helm architecture Helm CLI Tiller Kubeconfig kube-apiserver

  34. @hayorov Helm architecture Helm CLI Chart Repo Tiller Kubeconfig kube-apiserver

  35. @hayorov Helm architecture Helm CLI Chart Repo Tiller Kubeconfig kube-apiserver

  36. @hayorov Helm architecture Helm CLI Chart Repo Tiller Kubeconfig kube-apiserver

  37. @hayorov Helm architecture Helm CLI Chart Repo Tiller Kubeconfig kube-apiserver

  38. @hayorov Helm architecture Helm CLI Chart Repo Tiller Kubeconfig kube-apiserver

  39. @hayorov several angles from which someone might try to abuse

    Helm/Tiller: An attack vector
  40. @hayorov several angles from which someone might try to abuse

    Helm/Tiller: An attack vector • A privileged API user, such as a cluster-admin.
  41. @hayorov Helm architecture Helm CLI Chart Repo Kubeconfig Tiller

  42. @hayorov Helm architecture Helm CLI Chart Repo Kubeconfig Tiller Admin

  43. @hayorov several angles from which someone might try to abuse

    Helm/Tiller: An attack vector • A privileged API user, such as a cluster-admin.
  44. @hayorov • A low-privilege API user, such as a user

    who has been restricted to a single namespace using RBAC several angles from which someone might try to abuse Helm/Tiller: An attack vector • A privileged API user, such as a cluster-admin.
  45. @hayorov Helm architecture Helm CLI Chart Repo Kubeconfig Tiller Admin

  46. @hayorov Helm architecture Helm CLI Chart Repo Kubeconfig Tiller Admin

    Non-admin
  47. @hayorov • A low-privilege API user, such as a user

    who has been restricted to a single namespace using RBAC several angles from which someone might try to abuse Helm/Tiller: An attack vector • A privileged API user, such as a cluster-admin.
  48. @hayorov • A low-privilege API user, such as a user

    who has been restricted to a single namespace using RBAC • An in-cluster process, such as a compromised webserver. several angles from which someone might try to abuse Helm/Tiller: An attack vector • A privileged API user, such as a cluster-admin.
  49. @hayorov Helm CLI Chart Repo Kubeconfig An attack vector Tiller

    Admin Non-admin
  50. @hayorov Helm CLI Chart Repo Kubeconfig An attack vector Tiller

    Admin Non-admin
  51. @hayorov • A low-privilege API user, such as a user

    who has been restricted to a single namespace using RBAC • An in-cluster process, such as a compromised webserver. several angles from which someone might try to abuse Helm/Tiller: An attack vector • A privileged API user, such as a cluster-admin.
  52. @hayorov • A hostile chart author can create a chart

    containing unexpected resources. These can either escalate one of the other groups above, or run other malicious jobs. • A low-privilege API user, such as a user who has been restricted to a single namespace using RBAC • An in-cluster process, such as a compromised webserver. several angles from which someone might try to abuse Helm/Tiller: An attack vector • A privileged API user, such as a cluster-admin.
  53. @hayorov Helm CLI Chart Repo K8s cluster Kubeconfig An attack

    vector Tiller Admin Non-admin
  54. @hayorov Helm CLI Chart Repo K8s cluster Kubeconfig An attack

    vector Tiller Admin Non-admin
  55. @hayorov Helm CLI Chart Repo K8s cluster kube-system ns microservice

    ns Kube-api
 server Tiller-deploy svc Release configmap Microservice pod release ns(s) Release deployment Tiller-deploy pod Service Account Release svc Release configmap Kubeconfig Helm architecture
  56. @hayorov Helm CLI Chart Repo K8s cluster kube-system ns microservice

    ns Kube-api
 server Tiller-deploy svc Release configmap Microservice pod release ns(s) Release deployment Tiller-deploy pod Service Account Release svc Release configmap Kubeconfig Helm architecture
  57. @hayorov Helm CLI Chart Repo K8s cluster kube-system ns microservice

    ns Kube-api
 server Tiller-deploy svc Release configmap Microservice pod release ns(s) Release deployment Tiller-deploy pod Service Account Release svc Release configmap Kubeconfig Helm architecture
  58. @hayorov Helm CLI Chart Repo K8s cluster kube-system ns microservice

    ns Kube-api
 server Tiller-deploy svc Release configmap Microservice pod release ns(s) Release deployment Tiller-deploy pod Service Account Release svc Release configmap Kubeconfig Helm architecture
  59. @hayorov Helm CLI Chart Repo K8s cluster kube-system ns microservice

    ns Kube-api
 server Tiller-deploy svc Release configmap Microservice pod release ns(s) Release deployment Tiller-deploy pod Service Account Release svc Release configmap Kubeconfig Helm architecture
  60. @hayorov Helm CLI Chart Repo K8s cluster kube-system ns microservice

    ns Kube-api
 server Tiller-deploy svc Release configmap Microservice pod release ns(s) Release deployment Tiller-deploy pod Service Account Release svc Release configmap Kubeconfig Helm architecture
  61. @hayorov Helm CLI Chart Repo K8s cluster kube-system ns microservice

    ns Kube-api
 server Tiller-deploy svc Release configmap Microservice pod release ns(s) Release deployment Tiller-deploy pod Service Account Release svc Release configmap Kubeconfig An attack vector
  62. @hayorov Helm CLI Chart Repo K8s cluster kube-system ns microservice

    ns Kube-api
 server Tiller-deploy svc Release configmap Microservice pod release ns(s) Release deployment Tiller-deploy pod Service Account Release svc Release configmap HTTP Kubeconfig An attack vector
  63. @hayorov Helm CLI Chart Repo K8s cluster kube-system ns microservice

    ns Kube-api
 server Tiller-deploy svc Release configmap Microservice pod release ns(s) Release deployment Tiller-deploy pod Service Account Release svc Release configmap HTTP Kubeconfig gRPC An attack vector
  64. @hayorov Helm CLI Chart Repo K8s cluster kube-system ns microservice

    ns Kube-api
 server Tiller-deploy svc Release configmap Microservice pod release ns(s) Release deployment Tiller-deploy pod Service Account Release svc Release configmap HTTP Kubeconfig gRPC gRPC An attack vector
  65. @hayorov • Turn RBAC on • Tiller uses the default

    service account 
 in a namespace Kubernetes RBAC Tiller-deploy svc Tiller-deploy pod Service Account Release configmap (RBAC) Default Service Account Rest API
  66. @hayorov • Turn RBAC on • Tiller uses the default

    service account 
 in a namespace Kubernetes RBAC Tiller-deploy svc Tiller-deploy pod Service Account Release configmap (RBAC) Default Service Account Rest API advocacy site for RBAC https://rbac.dev/
  67. @hayorov • Turn RBAC on • Tiller uses the default

    service account 
 in a namespace Kubernetes RBAC Tiller-deploy svc Tiller-deploy pod Service Account Release configmap (RBAC) Default Service Account Rest API
  68. @hayorov • Turn RBAC on • Tiller uses the default

    service account 
 in a namespace Kubernetes RBAC Tiller-deploy svc Tiller-deploy pod Service Account Release configmap (RBAC) Rest API
  69. @hayorov Role RoleBinding • Turn RBAC on • Tiller uses

    the default service account 
 in a namespace Kubernetes RBAC Tiller-deploy svc Tiller-deploy pod Service Account Release configmap (RBAC) Rest API
  70. @hayorov Role RoleBinding • Turn RBAC on • Tiller uses

    the default service account 
 in a namespace Kubernetes RBAC • helm init does not create the associated
 ServiceAccount/Roles/RoleBindings ☝ Tiller-deploy svc Tiller-deploy pod Service Account Release configmap (RBAC) Rest API
  71. @hayorov Kubernetes RBAC Helm CLI Kubeconfig gRPC K8s cluster kube-system

    ns Kube-api
 server Tiller-deploy svc Release configmap Tiller-deploy pod Service Account Helm CLI Kubeconfig Cluster Admin X Team Kube-api
 server Tiller-deploy svc Release configmap Tiller-deploy pod Service Account Release deployment Release svc Release configmap RestAPI RBAC gRPC RestAPI RBAC • No multi-tenancy support • Solution - Multi-Tiller installations • Per developer, per team, per environment X Team ns ?
  72. @hayorov Kubernetes RBAC Helm CLI Kubeconfig gRPC K8s cluster kube-system

    ns Kube-api
 server Tiller-deploy svc Release configmap Tiller-deploy pod Service Account Helm CLI Kubeconfig Cluster Admin X Team Kube-api
 server Tiller-deploy svc Release configmap Tiller-deploy pod Service Account Release deployment Release svc Release configmap RestAPI RBAC gRPC RestAPI RBAC • No multi-tenancy support • Solution - Multi-Tiller installations • Per developer, per team, per environment X Team ns
  73. @hayorov • By default Tiller stores releases
 information in ConfigMaps

    Release information Tiller-deploy svc Tiller-deploy pod Service Account Role RoleBinding Release
 secrets RBAC Release
 secrets Release
 configmap
  74. @hayorov • By default Tiller stores releases
 information in ConfigMaps

    Release information Tiller-deploy svc Tiller-deploy pod Service Account Role RoleBinding Release
 secrets RBAC Release
 secrets Release
 configmap
  75. @hayorov • By default Tiller stores releases
 information in ConfigMaps

    Release information Tiller-deploy svc Tiller-deploy pod Service Account Role RoleBinding Release
 secrets RBAC Release
 secrets Release
 configmap Is this the 1MB limit?
  76. @hayorov • Use HTTPS always • Publish signed charts •

    Helm client supports TLS • Chartmuseum supports basic auth • helm-gcs plugin with GCP auth Chart Repos Helm CLI Signed Chart Repo fetch Chart Repo
  77. @hayorov • Use HTTPS always • Publish signed charts •

    Helm client supports TLS • Chartmuseum supports basic auth • helm-gcs plugin with GCP auth Chart Repos Helm CLI Signed Chart Repo fetch HTTPS mTLS or basic auth
  78. @hayorov gRPC API Helm CLI Kubeconfig gRPC • Tiller supports

    TLS on gRPC Tiller-deploy svc Tiller-deploy pod Service Account Role RoleBinding Release configmap RBAC Tiller-deploy svc
  79. @hayorov gRPC API Helm CLI Kubeconfig gRPC • Tiller supports

    TLS on gRPC Tiller-deploy svc Tiller-deploy pod Service Account Role RoleBinding Release configmap RBAC TLS
  80. @hayorov Secured Helm Helm CLI Chart Repo K8s cluster kube-system

    ns microservice ns Kube-api
 server Tiller-deploy svc Release secret Microservice pod release ns(s) Release deployment Release svc Release configmap fetch Kubeconfig HTTPS gRPC gRPC Service Account HTTPS TLS or basic auth mTLS RBAC mTLS Tiller-deploy pod
  81. @hayorov Secured Helm Helm CLI Chart Repo K8s cluster kube-system

    ns microservice ns Kube-api
 server Tiller-deploy svc Release secret Microservice pod release ns(s) Release deployment Release svc Release configmap fetch Kubeconfig HTTPS gRPC gRPC Service Account HTTPS TLS or basic auth mTLS RBAC mTLS Tiller-deploy pod instruction at bit.ly/helm-secure
  82. @hayorov Secured Helm Helm CLI Chart Repo K8s cluster kube-system

    ns microservice ns Kube-api
 server Tiller-deploy svc Release secret Microservice pod release ns(s) Release deployment Release svc Release configmap fetch Kubeconfig HTTPS gRPC gRPC Service Account HTTPS TLS or basic auth mTLS RBAC mTLS Tiller-deploy pod
  83. @hayorov Bonus

  84. @hayorov Where to browse charts? github.com/helm/charts

  85. @hayorov Where to browse charts? github.com/helm/charts

  86. @hayorov Where to browse charts? github.com/helm/charts ~300 charts

  87. @hayorov Where to browse charts? github.com/helm/charts ~300 charts stable
 incubator

  88. @hayorov Where to browse charts? github.com/helm/charts ~300 charts stable
 incubator

  89. @hayorov Helm Hub

  90. @hayorov Helm Hub hub.helm.sh

  91. @hayorov Helm Hub 629+ charts hub.helm.sh

  92. @hayorov Helm Hub 629+ charts 30+ external 
 repos hub.helm.sh

  93. @hayorov Helm Hub 629+ charts 30+ external 
 repos repo-values.yml

    hub.helm.sh
  94. @hayorov Helm Hub 629+ charts 30+ external 
 repos repo-values.yml

    hub.helm.sh
  95. @hayorov Open Service Broker API integration What are Service Brokers?

  96. @hayorov My Cluster Open Service Broker API integration What are

    Service Brokers?
  97. @hayorov My Cluster Open Service Broker API integration What are

    Service Brokers?
  98. @hayorov My Cluster Open Service Broker API integration What are

    Service Brokers?
  99. @hayorov My Cluster Cloud Provider Open Service Broker API integration

    What are Service Brokers?
  100. @hayorov My Cluster Cloud Provider Open Service Broker API integration

    What are Service Brokers? Managed MySQL
  101. @hayorov My Cluster Cloud Provider Open Service Broker API integration

    What are Service Brokers? Managed MySQL manually
  102. @hayorov My Cluster Cloud Provider Open Service Broker API integration

    What are Service Brokers? Managed MySQL
  103. @hayorov My Cluster Cloud Provider Open Service Broker API integration

    What are Service Brokers? Managed MySQL Service Broker
  104. @hayorov My Cluster Cloud Provider Open Service Broker API integration

    What are Service Brokers? Managed MySQL Service Broker OSBA
  105. @hayorov My Cluster Cloud Provider Open Service Broker API integration

    What are Service Brokers? Managed MySQL Tier: Basic Service Broker OSBA
  106. @hayorov My Cluster Cloud Provider Open Service Broker API integration

    What are Service Brokers? Managed MySQL Tier: Basic Service Broker OSBA DB credentials
  107. @hayorov My Cluster Cloud Provider Open Service Broker API integration

    What are Service Brokers? Managed MySQL Tier: Basic Service Broker OSBA DB credentials
  108. @hayorov My Cluster Cloud Provider Open Service Broker API integration

    What are Service Brokers? Managed MySQL Tier: Basic Service Broker OSBA DB credentials Helm glues Service Broker and charts that consume OSB resources
  109. @hayorov Repositories on GCS ChartMuseum de-facto is a standard helm-gcs

    is a plugin that allows to manage private repos on GCS Authentification using: • application default credentials • service account 
 (via the global flag)

  110. @hayorov Any alternatives? • Ksonnet, Metaparticle • Your configuration mngt

    system 
 (ansible, terraform, chef …) • Operator Framework (2k ⭐) Helm add-ons: • Draft, Scaffold Alternatives
  111. @hayorov Any alternatives? • Ksonnet, Metaparticle • Your configuration mngt

    system 
 (ansible, terraform, chef …) • Operator Framework (2k ⭐) Helm add-ons: • Draft, Scaffold Alternatives more control more k8s native
  112. @hayorov Any alternatives? • Ksonnet, Metaparticle • Your configuration mngt

    system 
 (ansible, terraform, chef …) • Operator Framework (2k ⭐) Helm add-ons: • Draft, Scaffold Alternatives more control more k8s native v2
  113. @hayorov Any alternatives? • Ksonnet, Metaparticle • Your configuration mngt

    system 
 (ansible, terraform, chef …) • Operator Framework (2k ⭐) Helm add-ons: • Draft, Scaffold Alternatives more control more k8s native v3 v2
  114. @hayorov Any alternatives? • Ksonnet, Metaparticle • Your configuration mngt

    system 
 (ansible, terraform, chef …) • Operator Framework (2k ⭐) Helm add-ons: • Draft, Scaffold Alternatives more control more k8s native v3 v2 K Ksonnet, Metaparticle
  115. @hayorov Any alternatives? • Ksonnet, Metaparticle • Your configuration mngt

    system 
 (ansible, terraform, chef …) • Operator Framework (2k ⭐) Helm add-ons: • Draft, Scaffold Alternatives more control more k8s native v3 Y Your Cfg
 management v2 K Ksonnet, Metaparticle
  116. @hayorov Any alternatives? • Ksonnet, Metaparticle • Your configuration mngt

    system 
 (ansible, terraform, chef …) • Operator Framework (2k ⭐) Helm add-ons: • Draft, Scaffold Alternatives more control more k8s native v3 Operator Framework Y Your Cfg
 management v2 K Ksonnet, Metaparticle
  117. @hayorov Any alternatives? • Ksonnet, Metaparticle • Your configuration mngt

    system 
 (ansible, terraform, chef …) • Operator Framework (2k ⭐) Helm add-ons: • Draft, Scaffold Alternatives more control more k8s native v3 Operator Framework Y Your Cfg
 management v2 A Draft, Skaffold K Ksonnet, Metaparticle
  118. @hayorov Helm 3 is the next big thing • Simplified

    client only architecture (no more Tiller) • State storage based of Release object (based on CRD) • Initial support for OCI repositories • (optional) Embedded Lua engine for scripting • Schematised values files (using JSONSchema) • Single event-driven model
 Current status: 3.0.0-alpha.1 released The future of Helm 3
  119. Thank you questions… Alex Khaerov hayorov http://bit.ly/helm-sec-slides