Securing Helm

@hayorov # % ' ⛓

Securing Helm Alex Khaerov hayorov

@hayorov @hayorov Привет ✋

@hayorov company who I am

@hayorov Alex Khaerov company who I am Development Lead

@hayorov Alex Khaerov company who I am Development Lead doing
software development in the recent decade junior speaker - Python, Kubernetes committee member (Moscow Python, Helm Summit) huge fan of laptop stickers

@hayorov Chainstack multi-cloud and multi-blockchain platform as a service based
in Singapore * Alex Khaerov company who I am Development Lead doing software development in the recent decade junior speaker - Python, Kubernetes committee member (Moscow Python, Helm Summit) huge fan of laptop stickers

@hayorov Helm… What is Helm? Helm architecture An attack vector
Securing Helm: RBAC, Release, Chart repo, gRPC… Helm future and alternatives Q&A Agenda

@hayorov

@hayorov the tool for managing Kubernetes packages called charts

@hayorov 12k 1k* * GitHub starts, Jan 2019 the tool
for managing Kubernetes packages called charts

@hayorov nurtured by 12k 1k* * GitHub starts, Jan 2019
the tool for managing Kubernetes packages called charts

@hayorov September 11 - 12, 2019   Pakhuis de Zwijger 
Amsterdam, The Netherlands https://events.linuxfoundation.org/events/helm-summit-2019/ CFP is open – Apply now! | #helmsummit

@hayorov Helm addresses several needs

@hayorov Packaging Helm addresses several needs

@hayorov Manage complexity Packaging Helm addresses several needs

@hayorov Manage complexity Packaging Application lifecycle Helm addresses several needs

@hayorov Manage complexity Packaging Application lifecycle Application metadata Repositories Package
dependencies Helm addresses several needs

dependencies Helm addresses several needs Parametrisation Templating

dependencies Helm addresses several needs Parametrisation Templating Deploy/config revisions Rollbacks Hooks Application probes

@hayorov “Batteries included” Manage complexity Packaging Application lifecycle Application metadata
Repositories Package dependencies Helm addresses several needs Parametrisation Templating Deploy/config revisions Rollbacks Hooks Application probes

@hayorov “Batteries included” Manage complexity Packaging Application lifecycle Application metadata
Repositories Package dependencies Helm addresses several needs Parametrisation Templating Deploy/config revisions Rollbacks Hooks Application probes CLI plugins

@hayorov There are three important concepts

@hayorov Chart There are three important concepts

@hayorov Chart Config There are three important concepts

@hayorov Chart Config Release There are three important concepts

@hayorov Chart Config Release Config There are three important concepts

@hayorov Helm architecture

@hayorov Helm architecture kube-apiserver

@hayorov Helm architecture Kubeconﬁg kube-apiserver

@hayorov Helm architecture Helm CLI Kubeconﬁg kube-apiserver

@hayorov Helm architecture Helm CLI Tiller Kubeconﬁg kube-apiserver

@hayorov Helm architecture Helm CLI Chart Repo Tiller Kubeconﬁg kube-apiserver

@hayorov several angles from which someone might try to abuse
Helm/Tiller: An attack vector

Helm/Tiller: An attack vector • A privileged API user, such as a cluster-admin.

@hayorov Helm architecture Helm CLI Chart Repo Kubeconﬁg Tiller

@hayorov Helm architecture Helm CLI Chart Repo Kubeconﬁg Tiller Admin

Helm/Tiller: An attack vector • A privileged API user, such as a cluster-admin.

@hayorov • A low-privilege API user, such as a user
who has been restricted to a single namespace using RBAC several angles from which someone might try to abuse Helm/Tiller: An attack vector • A privileged API user, such as a cluster-admin.

Non-admin

who has been restricted to a single namespace using RBAC several angles from which someone might try to abuse Helm/Tiller: An attack vector • A privileged API user, such as a cluster-admin.

who has been restricted to a single namespace using RBAC • An in-cluster process, such as a compromised webserver. several angles from which someone might try to abuse Helm/Tiller: An attack vector • A privileged API user, such as a cluster-admin.

@hayorov Helm CLI Chart Repo Kubeconﬁg An attack vector Tiller
Admin Non-admin

who has been restricted to a single namespace using RBAC • An in-cluster process, such as a compromised webserver. several angles from which someone might try to abuse Helm/Tiller: An attack vector • A privileged API user, such as a cluster-admin.

@hayorov • A hostile chart author can create a chart
containing unexpected resources. These can either escalate one of the other groups above, or run other malicious jobs. • A low-privilege API user, such as a user who has been restricted to a single namespace using RBAC • An in-cluster process, such as a compromised webserver. several angles from which someone might try to abuse Helm/Tiller: An attack vector • A privileged API user, such as a cluster-admin.

@hayorov Helm CLI Chart Repo K8s cluster Kubeconﬁg An attack
vector Tiller Admin Non-admin

@hayorov Helm CLI Chart Repo K8s cluster kube-system ns microservice
ns Kube-api  server Tiller-deploy svc Release configmap Microservice pod release ns(s) Release deployment Tiller-deploy pod Service Account Release svc Release configmap Kubeconfig Helm architecture

ns Kube-api  server Tiller-deploy svc Release configmap Microservice pod release ns(s) Release deployment Tiller-deploy pod Service Account Release svc Release configmap Kubeconfig An attack vector

ns Kube-api  server Tiller-deploy svc Release configmap Microservice pod release ns(s) Release deployment Tiller-deploy pod Service Account Release svc Release configmap HTTP Kubeconfig An attack vector

ns Kube-api  server Tiller-deploy svc Release configmap Microservice pod release ns(s) Release deployment Tiller-deploy pod Service Account Release svc Release configmap HTTP Kubeconfig gRPC An attack vector

ns Kube-api  server Tiller-deploy svc Release configmap Microservice pod release ns(s) Release deployment Tiller-deploy pod Service Account Release svc Release configmap HTTP Kubeconfig gRPC gRPC An attack vector

@hayorov • Turn RBAC on • Tiller uses the default
service account   in a namespace Kubernetes RBAC Tiller-deploy svc Tiller-deploy pod Service Account Release conﬁgmap (RBAC) Default Service Account Rest API

service account   in a namespace Kubernetes RBAC Tiller-deploy svc Tiller-deploy pod Service Account Release conﬁgmap (RBAC) Default Service Account Rest API advocacy site for RBAC https://rbac.dev/

service account   in a namespace Kubernetes RBAC Tiller-deploy svc Tiller-deploy pod Service Account Release conﬁgmap (RBAC) Default Service Account Rest API

service account   in a namespace Kubernetes RBAC Tiller-deploy svc Tiller-deploy pod Service Account Release conﬁgmap (RBAC) Rest API

@hayorov Role RoleBinding • Turn RBAC on • Tiller uses
the default service account   in a namespace Kubernetes RBAC Tiller-deploy svc Tiller-deploy pod Service Account Release conﬁgmap (RBAC) Rest API

@hayorov Role RoleBinding • Turn RBAC on • Tiller uses
the default service account   in a namespace Kubernetes RBAC • helm init does not create the associated  ServiceAccount/Roles/RoleBindings ☝ Tiller-deploy svc Tiller-deploy pod Service Account Release conﬁgmap (RBAC) Rest API

@hayorov Kubernetes RBAC Helm CLI Kubeconfig gRPC K8s cluster kube-system
ns Kube-api  server Tiller-deploy svc Release configmap Tiller-deploy pod Service Account Helm CLI Kubeconfig Cluster Admin X Team Kube-api  server Tiller-deploy svc Release configmap Tiller-deploy pod Service Account Release deployment Release svc Release configmap RestAPI RBAC gRPC RestAPI RBAC • No multi-tenancy support • Solution - Multi-Tiller installations • Per developer, per team, per environment X Team ns ?

@hayorov Kubernetes RBAC Helm CLI Kubeconfig gRPC K8s cluster kube-system
ns Kube-api  server Tiller-deploy svc Release configmap Tiller-deploy pod Service Account Helm CLI Kubeconfig Cluster Admin X Team Kube-api  server Tiller-deploy svc Release configmap Tiller-deploy pod Service Account Release deployment Release svc Release configmap RestAPI RBAC gRPC RestAPI RBAC • No multi-tenancy support • Solution - Multi-Tiller installations • Per developer, per team, per environment X Team ns

@hayorov • By default Tiller stores releases  information in ConfigMaps
Release information Tiller-deploy svc Tiller-deploy pod Service Account Role RoleBinding Release  secrets RBAC Release  secrets Release  conﬁgmap

@hayorov • By default Tiller stores releases  information in ConfigMaps
Release information Tiller-deploy svc Tiller-deploy pod Service Account Role RoleBinding Release  secrets RBAC Release  secrets Release  conﬁgmap Is this the 1MB limit?

@hayorov • Use HTTPS always • Publish signed charts •
Helm client supports TLS • Chartmuseum supports basic auth • helm-gcs plugin with GCP auth Chart Repos Helm CLI Signed Chart Repo fetch Chart Repo

@hayorov • Use HTTPS always • Publish signed charts •
Helm client supports TLS • Chartmuseum supports basic auth • helm-gcs plugin with GCP auth Chart Repos Helm CLI Signed Chart Repo fetch HTTPS mTLS or basic auth

@hayorov gRPC API Helm CLI Kubeconﬁg gRPC • Tiller supports
TLS on gRPC Tiller-deploy svc Tiller-deploy pod Service Account Role RoleBinding Release conﬁgmap RBAC Tiller-deploy svc

@hayorov gRPC API Helm CLI Kubeconﬁg gRPC • Tiller supports
TLS on gRPC Tiller-deploy svc Tiller-deploy pod Service Account Role RoleBinding Release conﬁgmap RBAC TLS

@hayorov Secured Helm Helm CLI Chart Repo K8s cluster kube-system
ns microservice ns Kube-api  server Tiller-deploy svc Release secret Microservice pod release ns(s) Release deployment Release svc Release conﬁgmap fetch Kubeconﬁg HTTPS gRPC gRPC Service Account HTTPS TLS or basic auth mTLS RBAC mTLS Tiller-deploy pod

ns microservice ns Kube-api  server Tiller-deploy svc Release secret Microservice pod release ns(s) Release deployment Release svc Release conﬁgmap fetch Kubeconﬁg HTTPS gRPC gRPC Service Account HTTPS TLS or basic auth mTLS RBAC mTLS Tiller-deploy pod instruction at bit.ly/helm-secure

ns microservice ns Kube-api  server Tiller-deploy svc Release secret Microservice pod release ns(s) Release deployment Release svc Release conﬁgmap fetch Kubeconﬁg HTTPS gRPC gRPC Service Account HTTPS TLS or basic auth mTLS RBAC mTLS Tiller-deploy pod

@hayorov Bonus

@hayorov Where to browse charts? github.com/helm/charts

@hayorov Where to browse charts? github.com/helm/charts ~300 charts

@hayorov Where to browse charts? github.com/helm/charts ~300 charts stable  incubator

@hayorov Helm Hub

@hayorov Helm Hub hub.helm.sh

@hayorov Helm Hub 629+ charts hub.helm.sh

@hayorov Helm Hub 629+ charts 30+ external   repos hub.helm.sh

@hayorov Helm Hub 629+ charts 30+ external   repos repo-values.yml
hub.helm.sh

@hayorov Open Service Broker API integration What are Service Brokers?

@hayorov My Cluster Open Service Broker API integration What are
Service Brokers?

@hayorov My Cluster Cloud Provider Open Service Broker API integration
What are Service Brokers?

What are Service Brokers? Managed MySQL

What are Service Brokers? Managed MySQL manually

What are Service Brokers? Managed MySQL

What are Service Brokers? Managed MySQL Service Broker

What are Service Brokers? Managed MySQL Service Broker OSBA

What are Service Brokers? Managed MySQL Tier: Basic Service Broker OSBA

What are Service Brokers? Managed MySQL Tier: Basic Service Broker OSBA DB credentials

What are Service Brokers? Managed MySQL Tier: Basic Service Broker OSBA DB credentials Helm glues Service Broker and charts that consume OSB resources

@hayorov Repositories on GCS ChartMuseum de-facto is a standard helm-gcs
is a plugin that allows to manage private repos on GCS Authentification using: • application default credentials • service account   (via the global flag) 

@hayorov Any alternatives? • Ksonnet, Metaparticle • Your configuration mngt
system   (ansible, terraform, chef …) • Operator Framework (2k ⭐) Helm add-ons: • Draft, Scaﬀold Alternatives

system   (ansible, terraform, chef …) • Operator Framework (2k ⭐) Helm add-ons: • Draft, Scaﬀold Alternatives more control more k8s native

system   (ansible, terraform, chef …) • Operator Framework (2k ⭐) Helm add-ons: • Draft, Scaﬀold Alternatives more control more k8s native v2

system   (ansible, terraform, chef …) • Operator Framework (2k ⭐) Helm add-ons: • Draft, Scaﬀold Alternatives more control more k8s native v3 v2

system   (ansible, terraform, chef …) • Operator Framework (2k ⭐) Helm add-ons: • Draft, Scaﬀold Alternatives more control more k8s native v3 v2 K Ksonnet, Metaparticle

system   (ansible, terraform, chef …) • Operator Framework (2k ⭐) Helm add-ons: • Draft, Scaﬀold Alternatives more control more k8s native v3 Y Your Cfg  management v2 K Ksonnet, Metaparticle

system   (ansible, terraform, chef …) • Operator Framework (2k ⭐) Helm add-ons: • Draft, Scaﬀold Alternatives more control more k8s native v3 Operator Framework Y Your Cfg  management v2 K Ksonnet, Metaparticle

system   (ansible, terraform, chef …) • Operator Framework (2k ⭐) Helm add-ons: • Draft, Scaﬀold Alternatives more control more k8s native v3 Operator Framework Y Your Cfg  management v2 A Draft, Skaffold K Ksonnet, Metaparticle

@hayorov Helm 3 is the next big thing • Simplified
client only architecture (no more Tiller) • State storage based of Release object (based on CRD) • Initial support for OCI repositories • (optional) Embedded Lua engine for scripting • Schematised values files (using JSONSchema) • Single event-driven model  Current status: 3.0.0-alpha.1 released The future of Helm 3

Thank you questions… Alex Khaerov hayorov http://bit.ly/helm-sec-slides

Securing Helm

Securing Helm

More Decks by Alex Khaerov

Featured

Transcript