Tweet
Tweet

Slide 1

Slide 1 text

@hayorov # % ' ⛓

Slide 2

Slide 2 text

Securing Helm Alex Khaerov hayorov

Slide 3

Slide 3 text

@hayorov @hayorov Привет ✋

Slide 4

Slide 4 text

@hayorov company who I am

Slide 5

Slide 5 text

@hayorov Alex Khaerov company who I am Development Lead

Slide 6

Slide 6 text

@hayorov Alex Khaerov company who I am Development Lead doing software development in the recent decade junior speaker - Python, Kubernetes committee member (Moscow Python, Helm Summit) huge fan of laptop stickers

Slide 7

Slide 7 text

@hayorov Chainstack multi-cloud and multi-blockchain platform as a service based in Singapore * Alex Khaerov company who I am Development Lead doing software development in the recent decade junior speaker - Python, Kubernetes committee member (Moscow Python, Helm Summit) huge fan of laptop stickers

Slide 8

Slide 8 text

@hayorov Helm… What is Helm? Helm architecture An attack vector Securing Helm: RBAC, Release, Chart repo, gRPC… Helm future and alternatives Q&A Agenda

Slide 9

Slide 9 text

@hayorov

Slide 10

Slide 10 text

@hayorov the tool for managing Kubernetes packages called charts

Slide 11

Slide 11 text

@hayorov the tool for managing Kubernetes packages called charts

Slide 12

Slide 12 text

@hayorov 12k 1k* * GitHub starts, Jan 2019 the tool for managing Kubernetes packages called charts

Slide 13

Slide 13 text

@hayorov nurtured by 12k 1k* * GitHub starts, Jan 2019 the tool for managing Kubernetes packages called charts

Slide 14

Slide 14 text

@hayorov September 11 - 12, 2019 
 Pakhuis de Zwijger
 Amsterdam, The Netherlands https://events.linuxfoundation.org/events/helm-summit-2019/ CFP is open – Apply now! | #helmsummit

Slide 15

Slide 15 text

@hayorov Helm addresses several needs

Slide 16

Slide 16 text

@hayorov Packaging Helm addresses several needs

Slide 17

Slide 17 text

@hayorov Manage complexity Packaging Helm addresses several needs

Slide 18

Slide 18 text

@hayorov Manage complexity Packaging Application lifecycle Helm addresses several needs

Slide 19

Slide 19 text

@hayorov Manage complexity Packaging Application lifecycle Application metadata Repositories Package dependencies Helm addresses several needs

Slide 20

Slide 20 text

@hayorov Manage complexity Packaging Application lifecycle Application metadata Repositories Package dependencies Helm addresses several needs Parametrisation Templating

Slide 21

Slide 21 text

@hayorov Manage complexity Packaging Application lifecycle Application metadata Repositories Package dependencies Helm addresses several needs Parametrisation Templating Deploy/config revisions Rollbacks Hooks Application probes

Slide 22

Slide 22 text

@hayorov “Batteries included” Manage complexity Packaging Application lifecycle Application metadata Repositories Package dependencies Helm addresses several needs Parametrisation Templating Deploy/config revisions Rollbacks Hooks Application probes

Slide 23

Slide 23 text

@hayorov “Batteries included” Manage complexity Packaging Application lifecycle Application metadata Repositories Package dependencies Helm addresses several needs Parametrisation Templating Deploy/config revisions Rollbacks Hooks Application probes CLI plugins

Slide 24

Slide 24 text

@hayorov There are three important concepts

Slide 25

Slide 25 text

@hayorov Chart There are three important concepts

Slide 26

Slide 26 text

@hayorov Chart Config There are three important concepts

Slide 27

Slide 27 text

@hayorov Chart Config Release There are three important concepts

Slide 28

Slide 28 text

@hayorov Chart Config Release Config There are three important concepts

Slide 29

Slide 29 text

@hayorov Helm architecture

Slide 30

Slide 30 text

@hayorov Helm architecture kube-apiserver

Slide 31

Slide 31 text

@hayorov Helm architecture Kubeconfig kube-apiserver

Slide 32

Slide 32 text

@hayorov Helm architecture Helm CLI Kubeconfig kube-apiserver

Slide 33

Slide 33 text

@hayorov Helm architecture Helm CLI Tiller Kubeconfig kube-apiserver

Slide 34

Slide 34 text

@hayorov Helm architecture Helm CLI Chart Repo Tiller Kubeconfig kube-apiserver

Slide 35

Slide 35 text

@hayorov Helm architecture Helm CLI Chart Repo Tiller Kubeconfig kube-apiserver

Slide 36

Slide 36 text

@hayorov Helm architecture Helm CLI Chart Repo Tiller Kubeconfig kube-apiserver

Slide 37

Slide 37 text

@hayorov Helm architecture Helm CLI Chart Repo Tiller Kubeconfig kube-apiserver

Slide 38

Slide 38 text

@hayorov Helm architecture Helm CLI Chart Repo Tiller Kubeconfig kube-apiserver

Slide 39

Slide 39 text

@hayorov several angles from which someone might try to abuse Helm/Tiller: An attack vector

Slide 40

Slide 40 text

@hayorov several angles from which someone might try to abuse Helm/Tiller: An attack vector • A privileged API user, such as a cluster-admin.

Slide 41

Slide 41 text

@hayorov Helm architecture Helm CLI Chart Repo Kubeconfig Tiller

Slide 42

Slide 42 text

@hayorov Helm architecture Helm CLI Chart Repo Kubeconfig Tiller Admin

Slide 43

Slide 43 text

@hayorov several angles from which someone might try to abuse Helm/Tiller: An attack vector • A privileged API user, such as a cluster-admin.

Slide 44

Slide 44 text

@hayorov • A low-privilege API user, such as a user who has been restricted to a single namespace using RBAC several angles from which someone might try to abuse Helm/Tiller: An attack vector • A privileged API user, such as a cluster-admin.

Slide 45

Slide 45 text

@hayorov Helm architecture Helm CLI Chart Repo Kubeconfig Tiller Admin

Slide 46

Slide 46 text

@hayorov Helm architecture Helm CLI Chart Repo Kubeconfig Tiller Admin Non-admin

Slide 47

Slide 47 text

@hayorov • A low-privilege API user, such as a user who has been restricted to a single namespace using RBAC several angles from which someone might try to abuse Helm/Tiller: An attack vector • A privileged API user, such as a cluster-admin.

Slide 48

Slide 48 text

@hayorov • A low-privilege API user, such as a user who has been restricted to a single namespace using RBAC • An in-cluster process, such as a compromised webserver. several angles from which someone might try to abuse Helm/Tiller: An attack vector • A privileged API user, such as a cluster-admin.

Slide 49

Slide 49 text

@hayorov Helm CLI Chart Repo Kubeconfig An attack vector Tiller Admin Non-admin

Slide 50

Slide 50 text

@hayorov Helm CLI Chart Repo Kubeconfig An attack vector Tiller Admin Non-admin

Slide 51

Slide 51 text

@hayorov • A low-privilege API user, such as a user who has been restricted to a single namespace using RBAC • An in-cluster process, such as a compromised webserver. several angles from which someone might try to abuse Helm/Tiller: An attack vector • A privileged API user, such as a cluster-admin.

Slide 52

Slide 52 text

@hayorov • A hostile chart author can create a chart containing unexpected resources. These can either escalate one of the other groups above, or run other malicious jobs. • A low-privilege API user, such as a user who has been restricted to a single namespace using RBAC • An in-cluster process, such as a compromised webserver. several angles from which someone might try to abuse Helm/Tiller: An attack vector • A privileged API user, such as a cluster-admin.

Slide 53

Slide 53 text

@hayorov Helm CLI Chart Repo K8s cluster Kubeconfig An attack vector Tiller Admin Non-admin

Slide 54

Slide 54 text

@hayorov Helm CLI Chart Repo K8s cluster Kubeconfig An attack vector Tiller Admin Non-admin

Slide 55

Slide 55 text

@hayorov Helm CLI Chart Repo K8s cluster kube-system ns microservice ns Kube-api
 server Tiller-deploy svc Release configmap Microservice pod release ns(s) Release deployment Tiller-deploy pod Service Account Release svc Release configmap Kubeconfig Helm architecture

Slide 56

Slide 56 text

@hayorov Helm CLI Chart Repo K8s cluster kube-system ns microservice ns Kube-api
 server Tiller-deploy svc Release configmap Microservice pod release ns(s) Release deployment Tiller-deploy pod Service Account Release svc Release configmap Kubeconfig Helm architecture

Slide 57

Slide 57 text

@hayorov Helm CLI Chart Repo K8s cluster kube-system ns microservice ns Kube-api
 server Tiller-deploy svc Release configmap Microservice pod release ns(s) Release deployment Tiller-deploy pod Service Account Release svc Release configmap Kubeconfig Helm architecture

Slide 58

Slide 58 text

@hayorov Helm CLI Chart Repo K8s cluster kube-system ns microservice ns Kube-api
 server Tiller-deploy svc Release configmap Microservice pod release ns(s) Release deployment Tiller-deploy pod Service Account Release svc Release configmap Kubeconfig Helm architecture

Slide 59

Slide 59 text

@hayorov Helm CLI Chart Repo K8s cluster kube-system ns microservice ns Kube-api
 server Tiller-deploy svc Release configmap Microservice pod release ns(s) Release deployment Tiller-deploy pod Service Account Release svc Release configmap Kubeconfig Helm architecture

Slide 60

Slide 60 text

@hayorov Helm CLI Chart Repo K8s cluster kube-system ns microservice ns Kube-api
 server Tiller-deploy svc Release configmap Microservice pod release ns(s) Release deployment Tiller-deploy pod Service Account Release svc Release configmap Kubeconfig Helm architecture

Slide 61

Slide 61 text

@hayorov Helm CLI Chart Repo K8s cluster kube-system ns microservice ns Kube-api
 server Tiller-deploy svc Release configmap Microservice pod release ns(s) Release deployment Tiller-deploy pod Service Account Release svc Release configmap Kubeconfig An attack vector

Slide 62

Slide 62 text

@hayorov Helm CLI Chart Repo K8s cluster kube-system ns microservice ns Kube-api
 server Tiller-deploy svc Release configmap Microservice pod release ns(s) Release deployment Tiller-deploy pod Service Account Release svc Release configmap HTTP Kubeconfig An attack vector

Slide 63

Slide 63 text

@hayorov Helm CLI Chart Repo K8s cluster kube-system ns microservice ns Kube-api
 server Tiller-deploy svc Release configmap Microservice pod release ns(s) Release deployment Tiller-deploy pod Service Account Release svc Release configmap HTTP Kubeconfig gRPC An attack vector

Slide 64

Slide 64 text

@hayorov Helm CLI Chart Repo K8s cluster kube-system ns microservice ns Kube-api
 server Tiller-deploy svc Release configmap Microservice pod release ns(s) Release deployment Tiller-deploy pod Service Account Release svc Release configmap HTTP Kubeconfig gRPC gRPC An attack vector

Slide 65

Slide 65 text

@hayorov • Turn RBAC on • Tiller uses the default service account 
 in a namespace Kubernetes RBAC Tiller-deploy svc Tiller-deploy pod Service Account Release configmap (RBAC) Default Service Account Rest API

Slide 66

Slide 66 text

@hayorov • Turn RBAC on • Tiller uses the default service account 
 in a namespace Kubernetes RBAC Tiller-deploy svc Tiller-deploy pod Service Account Release configmap (RBAC) Default Service Account Rest API advocacy site for RBAC https://rbac.dev/

Slide 67

Slide 67 text

@hayorov • Turn RBAC on • Tiller uses the default service account 
 in a namespace Kubernetes RBAC Tiller-deploy svc Tiller-deploy pod Service Account Release configmap (RBAC) Default Service Account Rest API

Slide 68

Slide 68 text

@hayorov • Turn RBAC on • Tiller uses the default service account 
 in a namespace Kubernetes RBAC Tiller-deploy svc Tiller-deploy pod Service Account Release configmap (RBAC) Rest API

Slide 69

Slide 69 text

@hayorov Role RoleBinding • Turn RBAC on • Tiller uses the default service account 
 in a namespace Kubernetes RBAC Tiller-deploy svc Tiller-deploy pod Service Account Release configmap (RBAC) Rest API

Slide 70

Slide 70 text

@hayorov Role RoleBinding • Turn RBAC on • Tiller uses the default service account 
 in a namespace Kubernetes RBAC • helm init does not create the associated
 ServiceAccount/Roles/RoleBindings ☝ Tiller-deploy svc Tiller-deploy pod Service Account Release configmap (RBAC) Rest API

Slide 71

Slide 71 text

@hayorov Kubernetes RBAC Helm CLI Kubeconfig gRPC K8s cluster kube-system ns Kube-api
 server Tiller-deploy svc Release configmap Tiller-deploy pod Service Account Helm CLI Kubeconfig Cluster Admin X Team Kube-api
 server Tiller-deploy svc Release configmap Tiller-deploy pod Service Account Release deployment Release svc Release configmap RestAPI RBAC gRPC RestAPI RBAC • No multi-tenancy support • Solution - Multi-Tiller installations • Per developer, per team, per environment X Team ns ?

Slide 72

Slide 72 text

@hayorov Kubernetes RBAC Helm CLI Kubeconfig gRPC K8s cluster kube-system ns Kube-api
 server Tiller-deploy svc Release configmap Tiller-deploy pod Service Account Helm CLI Kubeconfig Cluster Admin X Team Kube-api
 server Tiller-deploy svc Release configmap Tiller-deploy pod Service Account Release deployment Release svc Release configmap RestAPI RBAC gRPC RestAPI RBAC • No multi-tenancy support • Solution - Multi-Tiller installations • Per developer, per team, per environment X Team ns

Slide 73

Slide 73 text

@hayorov • By default Tiller stores releases
 information in ConfigMaps Release information Tiller-deploy svc Tiller-deploy pod Service Account Role RoleBinding Release
 secrets RBAC Release
 secrets Release
 configmap

Slide 74

Slide 74 text

@hayorov • By default Tiller stores releases
 information in ConfigMaps Release information Tiller-deploy svc Tiller-deploy pod Service Account Role RoleBinding Release
 secrets RBAC Release
 secrets Release
 configmap

Slide 75

Slide 75 text

@hayorov • By default Tiller stores releases
 information in ConfigMaps Release information Tiller-deploy svc Tiller-deploy pod Service Account Role RoleBinding Release
 secrets RBAC Release
 secrets Release
 configmap Is this the 1MB limit?

Slide 76

Slide 76 text

@hayorov • Use HTTPS always • Publish signed charts • Helm client supports TLS • Chartmuseum supports basic auth • helm-gcs plugin with GCP auth Chart Repos Helm CLI Signed Chart Repo fetch Chart Repo

Slide 77

Slide 77 text

@hayorov • Use HTTPS always • Publish signed charts • Helm client supports TLS • Chartmuseum supports basic auth • helm-gcs plugin with GCP auth Chart Repos Helm CLI Signed Chart Repo fetch HTTPS mTLS or basic auth

Slide 78

Slide 78 text

@hayorov gRPC API Helm CLI Kubeconfig gRPC • Tiller supports TLS on gRPC Tiller-deploy svc Tiller-deploy pod Service Account Role RoleBinding Release configmap RBAC Tiller-deploy svc

Slide 79

Slide 79 text

@hayorov gRPC API Helm CLI Kubeconfig gRPC • Tiller supports TLS on gRPC Tiller-deploy svc Tiller-deploy pod Service Account Role RoleBinding Release configmap RBAC TLS

Slide 80

Slide 80 text

@hayorov Secured Helm Helm CLI Chart Repo K8s cluster kube-system ns microservice ns Kube-api
 server Tiller-deploy svc Release secret Microservice pod release ns(s) Release deployment Release svc Release configmap fetch Kubeconfig HTTPS gRPC gRPC Service Account HTTPS TLS or basic auth mTLS RBAC mTLS Tiller-deploy pod

Slide 81

Slide 81 text

@hayorov Secured Helm Helm CLI Chart Repo K8s cluster kube-system ns microservice ns Kube-api
 server Tiller-deploy svc Release secret Microservice pod release ns(s) Release deployment Release svc Release configmap fetch Kubeconfig HTTPS gRPC gRPC Service Account HTTPS TLS or basic auth mTLS RBAC mTLS Tiller-deploy pod instruction at bit.ly/helm-secure

Slide 82

Slide 82 text

@hayorov Secured Helm Helm CLI Chart Repo K8s cluster kube-system ns microservice ns Kube-api
 server Tiller-deploy svc Release secret Microservice pod release ns(s) Release deployment Release svc Release configmap fetch Kubeconfig HTTPS gRPC gRPC Service Account HTTPS TLS or basic auth mTLS RBAC mTLS Tiller-deploy pod

Slide 83

Slide 83 text

@hayorov Bonus

Slide 84

Slide 84 text

@hayorov Where to browse charts? github.com/helm/charts

Slide 85

Slide 85 text

@hayorov Where to browse charts? github.com/helm/charts

Slide 86

Slide 86 text

@hayorov Where to browse charts? github.com/helm/charts ~300 charts

Slide 87

Slide 87 text

@hayorov Where to browse charts? github.com/helm/charts ~300 charts stable
 incubator

Slide 88

Slide 88 text

@hayorov Where to browse charts? github.com/helm/charts ~300 charts stable
 incubator

Slide 89

Slide 89 text

@hayorov Helm Hub

Slide 90

Slide 90 text

@hayorov Helm Hub hub.helm.sh

Slide 91

Slide 91 text

@hayorov Helm Hub 629+ charts hub.helm.sh

Slide 92

Slide 92 text

@hayorov Helm Hub 629+ charts 30+ external 
 repos hub.helm.sh

Slide 93

Slide 93 text

@hayorov Helm Hub 629+ charts 30+ external 
 repos repo-values.yml hub.helm.sh

Slide 94

Slide 94 text

@hayorov Helm Hub 629+ charts 30+ external 
 repos repo-values.yml hub.helm.sh

Slide 95

Slide 95 text

@hayorov Open Service Broker API integration What are Service Brokers?

Slide 96

Slide 96 text

@hayorov My Cluster Open Service Broker API integration What are Service Brokers?

Slide 97

Slide 97 text

@hayorov My Cluster Open Service Broker API integration What are Service Brokers?

Slide 98

Slide 98 text

@hayorov My Cluster Open Service Broker API integration What are Service Brokers?

Slide 99

Slide 99 text

@hayorov My Cluster Cloud Provider Open Service Broker API integration What are Service Brokers?

Slide 100

Slide 100 text

@hayorov My Cluster Cloud Provider Open Service Broker API integration What are Service Brokers? Managed MySQL

Slide 101

Slide 101 text

@hayorov My Cluster Cloud Provider Open Service Broker API integration What are Service Brokers? Managed MySQL manually

Slide 102

Slide 102 text

@hayorov My Cluster Cloud Provider Open Service Broker API integration What are Service Brokers? Managed MySQL

Slide 103

Slide 103 text

@hayorov My Cluster Cloud Provider Open Service Broker API integration What are Service Brokers? Managed MySQL Service Broker

Slide 104

Slide 104 text

@hayorov My Cluster Cloud Provider Open Service Broker API integration What are Service Brokers? Managed MySQL Service Broker OSBA

Slide 105

Slide 105 text

@hayorov My Cluster Cloud Provider Open Service Broker API integration What are Service Brokers? Managed MySQL Tier: Basic Service Broker OSBA

Slide 106

Slide 106 text

@hayorov My Cluster Cloud Provider Open Service Broker API integration What are Service Brokers? Managed MySQL Tier: Basic Service Broker OSBA DB credentials

Slide 107

Slide 107 text

@hayorov My Cluster Cloud Provider Open Service Broker API integration What are Service Brokers? Managed MySQL Tier: Basic Service Broker OSBA DB credentials

Slide 108

Slide 108 text

@hayorov My Cluster Cloud Provider Open Service Broker API integration What are Service Brokers? Managed MySQL Tier: Basic Service Broker OSBA DB credentials Helm glues Service Broker and charts that consume OSB resources

Slide 109

Slide 109 text

@hayorov Repositories on GCS ChartMuseum de-facto is a standard helm-gcs is a plugin that allows to manage private repos on GCS Authentification using: • application default credentials • service account 
 (via the global flag)


Slide 110

Slide 110 text

@hayorov Any alternatives? • Ksonnet, Metaparticle • Your configuration mngt system 
 (ansible, terraform, chef …) • Operator Framework (2k ⭐) Helm add-ons: • Draft, Scaffold Alternatives

Slide 111

Slide 111 text

@hayorov Any alternatives? • Ksonnet, Metaparticle • Your configuration mngt system 
 (ansible, terraform, chef …) • Operator Framework (2k ⭐) Helm add-ons: • Draft, Scaffold Alternatives more control more k8s native

Slide 112

Slide 112 text

@hayorov Any alternatives? • Ksonnet, Metaparticle • Your configuration mngt system 
 (ansible, terraform, chef …) • Operator Framework (2k ⭐) Helm add-ons: • Draft, Scaffold Alternatives more control more k8s native v2

Slide 113

Slide 113 text

@hayorov Any alternatives? • Ksonnet, Metaparticle • Your configuration mngt system 
 (ansible, terraform, chef …) • Operator Framework (2k ⭐) Helm add-ons: • Draft, Scaffold Alternatives more control more k8s native v3 v2

Slide 114

Slide 114 text

@hayorov Any alternatives? • Ksonnet, Metaparticle • Your configuration mngt system 
 (ansible, terraform, chef …) • Operator Framework (2k ⭐) Helm add-ons: • Draft, Scaffold Alternatives more control more k8s native v3 v2 K Ksonnet, Metaparticle

Slide 115

Slide 115 text

@hayorov Any alternatives? • Ksonnet, Metaparticle • Your configuration mngt system 
 (ansible, terraform, chef …) • Operator Framework (2k ⭐) Helm add-ons: • Draft, Scaffold Alternatives more control more k8s native v3 Y Your Cfg
 management v2 K Ksonnet, Metaparticle

Slide 116

Slide 116 text

@hayorov Any alternatives? • Ksonnet, Metaparticle • Your configuration mngt system 
 (ansible, terraform, chef …) • Operator Framework (2k ⭐) Helm add-ons: • Draft, Scaffold Alternatives more control more k8s native v3 Operator Framework Y Your Cfg
 management v2 K Ksonnet, Metaparticle

Slide 117

Slide 117 text

@hayorov Any alternatives? • Ksonnet, Metaparticle • Your configuration mngt system 
 (ansible, terraform, chef …) • Operator Framework (2k ⭐) Helm add-ons: • Draft, Scaffold Alternatives more control more k8s native v3 Operator Framework Y Your Cfg
 management v2 A Draft, Skaffold K Ksonnet, Metaparticle

Slide 118

Slide 118 text

@hayorov Helm 3 is the next big thing • Simplified client only architecture (no more Tiller) • State storage based of Release object (based on CRD) • Initial support for OCI repositories • (optional) Embedded Lua engine for scripting • Schematised values files (using JSONSchema) • Single event-driven model
 Current status: 3.0.0-alpha.1 released The future of Helm 3

Slide 119

Slide 119 text

Thank you questions… Alex Khaerov hayorov http://bit.ly/helm-sec-slides