Helm: Container Systeme mit Kubernetes erfolgreich gestalten

Copyright 2019 bee42 solutions gmbh <[email protected]> @PRossbach rethink IT -
We improve your systems with passion 1 HELM Build fashionable container systems with Kubernetes

We improve your systems with passion HELM 2 Originally developed by Deis 2015 and   donated to Cloud Native Computing Foundation 2018 Second generation available   and third generation development is started at July 2018 Package manager for Kubernetes clusters

We improve your systems with passion What is Helm? 3 •Helm is a Kubernetes Package Manager •Helm charts are build on top of Kubernetes manifests •Charts are stored in a registry called Helm Museum (hub.helm.sh) •It use a templating preprocessing and managed release of Kubernetes resources •Ability to consider scalability from the get-go •SRE’s and developer can search of charts and scratch it •You can reuse other charts

We improve your systems with passion 4 • Kubernetes is a container orchestrator. • It’s how to run containers at scale. • It’s a very active open-source platform with lots of contributors, start at 6. June 2014 • Originally developed by Google and   donated to Cloud Native Computing Foundation

We improve your systems with passion 5

We improve your systems with passion Kubernetes primitives 6

We improve your systems with passion K8s Deployment Challenge 7 * Migrate to microservice or serverless architecture * Applications consists from multiple components each component has its own resources and can be scaled individually It’s hard to ... ... manage, edit and update multiple K8s configurations ... deploy multiple K8s configurations as a SINGLE application ... share and reuse K8s configurations and applications ... parametrize and support multiple environments ... manage application releases: rollout, rollback, diff, history ... define deployment lifecycle (control operations to be run in different phases) ... validate release state after deployment

We improve your systems with passion What Helm can do for you? 8 •Create new charts, aka package K8s manifests, from scratch •Package charts into chart archive (tgz) ﬁle •Interact with chart repositories where charts are stored •Reuse exiting charts •Install and uninstall charts into an existing Kubernetes cluster •Manage the release cycle of charts that have been installed with Helm

We improve your systems with passion Chart 9 a package of Kubernetes manifests Release a chart instance is loaded into Kubernetes Repository a download area of published charts Template a Kubernetes conﬁguration ﬁle mixed with Go/Sprig templates Concepts

We improve your systems with passion How Helm 2 works? 10 client api-service tiller service(s) https grpc brew install kubernetes-helm curl https://raw.githubusercontent.com/kubernetes/helm/master/scripts/get | bash or

We improve your systems with passion helm init 11 $ cat >tiller-rbac.yaml <<EOF apiVersion: v1 kind: ServiceAccount metadata: name: tiller namespace: kube-system --- apiVersion: rbac.authorization.k8s.io/v1beta1 kind: ClusterRoleBinding metadata: name: tiller roleRef: apiGroup: rbac.authorization.k8s.io kind: ClusterRole name: cluster-admin subjects: - kind: ServiceAccount name: tiller namespace: kube-system EOF $ kubectl create -f tiller-rbac.yaml $ helm init --service-account tiller --upgrade

We improve your systems with passion Helm flow 12 client api-service tiller service(s) publish fetch install upgrade delete redis create application tomcat postgres

We improve your systems with passion Helm CLI Commands 13 completion Generate autocompletions script for the specified shell (bash or zsh) create create a new chart with the given name delete given a release name, delete the release from Kubernetes dependency manage a chart's dependencies fetch download a chart from a repository and (optionally) unpack it in local directory get download a named release history fetch release history home displays the location of HELM_HOME init initialize Helm on both client and server inspect inspect a chart install install a chart archive lint examines a chart for possible issues list list releases package package a chart directory into a chart archive plugin add, list, or remove Helm plugins repo add, list, remove, update, and index chart repositories reset uninstalls Tiller from a cluster rollback roll back a release to a previous revision search search for a keyword in charts serve start a local http web server status displays the status of the named release template locally render templates test test a release upgrade upgrade a release verify verify that a chart at the given path has been signed and is valid version print the client/server version information

We improve your systems with passion Example: Simple config map 14 $ helm create my-chart # review mychart templates $ rm -rf mychart/templates/*.* $ cat <<EOF >mychart/templates/configmap.yaml apiVersion: v1 kind: ConfigMap metadata: name: mychart-configmap data: myvalue: "Hello World" EOF $ helm install --dry-run --debug ./mychart $ helm install --name full-coral --namespace test ./mychart $ helm get manifest full-coral $ helm delete full-coral

We improve your systems with passion Example: Prometheus & Grafana 15 # pre init  # ————————————————————————————————————————————————————————————————————————— $ helm init --client-only # install prometheus # ————————————————————————————————————————————————————————————————————————— $ helm install stable/prometheus \ --name prometheus \  --namespace monitoring \ --set rbac.create=true,server.persistentVolume.enabled=false,\ alertmanager.enabled=false,pushgateway.enabled=false # install grafana  # ————————————————————————————————————————————————————————————————————————— $ helm install ./grafana \ --name grafana \ --namespace monitoring \ --set server.persistentVolume.enabled=false,server.adminPassword=admin

We improve your systems with passion Example: Traefik 16 $ helm install stable/traefik \  --name traefik --namespace kube-system \  --set rbac.enabled=true,serviceType=NodePort $ kubectl run —image=bee42/whoami:2.0.0 \ --expose --port=80 --target-port=80 --name=whoami   $ cat >whoami-ingress.yaml <<EOF  apiVersion: extensions/v1beta1 kind: Ingress metadata: name: whoami  annotations:  kubernetes.io/ingress.class=traefik spec: rules: - host: whoami.bee42.127.0.0.1.nip.io  http: paths: - path: / backend: serviceName: whoami servicePort: http EOF $ kubectl apply -f whoami-ingress.yaml $ curl <node ip>:<traefik node.port>

We improve your systems with passion Helm chart directory 17 <Helm chart> ├── Chart.yaml ├── <LICENSE> ├── <README>.md ├── charts │ ├── <external chart archive>.tgz │ ├── <subchart> │ │ ├── Chart.yaml │ │ ├── templates │ │ │ ├── _helpers.tpl │ │ │ └── <k8s manifest>.yaml │ │ └── values.yaml ├── requirements.lock ├── requirements.yaml  ├── templates │ ├── _helpers.tpl │ ├── NOTES.txt │ └── <k8s manifest>.yaml └── values.yaml

We improve your systems with passion Chart.yaml 18 apiVersion: v1 description: A Helm chart for Docker Voting App porting to k8s cluster name: voting-app version: 0.3.2 keywords: - demo - kubernetes - helm - voting-app home: https://bee42.com/ icon: https://bee42.com/images/logo.svg sources: - https://gitlab.bee42.com/kubernetes/examples/voting-app maintainers: - name: Peter Rossbach email: [email protected]

We improve your systems with passion values.yaml 19 # default ingress port servicePort: 80 # configure subchart redis redis: master: persistence: enabled: false usePassword: false …  imageCredentials: registry: r.gitlab.bee42.com username: gitlab+deploy-token-1 password: xxx-xxx # ingress ingress: enabled: true nginx: enabled: true traefik: enabled: false

We improve your systems with passion requirements.yaml 20 dependencies: - name: postgresql version: 3.7.0 repository: https://kubernetes-charts.storage.googleapis.com/ alias: postgresql-1 - name: redis version: 5.1.3 repository: https://kubernetes-charts.storage.googleapis.com/ condition: postgresql-1.enabled,global.result.enabled tags: - back-end - subchart2

We improve your systems with passion _helper.tpl 21 {{/* vim: set filetype=mustache: */}} {{/* Expand the name of the chart. */}} {{- define "name" -}} {{- default .Chart.Name .Values.nameOverride | trunc 63 | trimSuffix "-" -}} {{- end -}} {{/* Create a default fully qualified app name. We truncate at 63 chars because some Kubernetes name fields are limited to this (by the DNS naming spec). */}} {{- define "fullname" -}} {{- $name := default .Chart.Name .Values.nameOverride -}} {{- printf "%s-%s" .Release.Name $name | trunc 63 | trimSuffix "-" -}} {{- end -}} {{/* Create a registry image secret to pull voting app images */}} {{- define "imagePullSecret" }} {{- printf "{\"auths\": {\"%s\": {\"auth\": \"%s\"}}}" .Values.imageCredentials.registry   printf "%s:%s" .Values.imageCredentials.username .Values.imageCredentials.password | b64enc) | b64enc }} {{- end }}

We improve your systems with passion Flow Control 22 If/Else {{ if PIPELINE }} # Do something {{ else if OTHER PIPELINE }} # Do something else {{ else }} # Default case {{ end }} data: myvalue: "Hello World" drink: {{ .Values.favorite.drink | default "tea" | quote }} food: {{ .Values.favorite.food | upper | quote }} {{- if eq .Values.favorite.drink "lemonade" }} mug: true {{- end }} # notice the "-" in the left, if will help eliminate newline before variable

We improve your systems with passion Flow Control 23 With data: myvalue: "Hello World" {{- with .Values.favorite }} drink: {{ .drink | default "tea" | quote }} food: {{ .food | upper | quote }} {{- end }} # instead of writing ".Values.favorite.drink"

We improve your systems with passion Flow Control 24 Range # predefined variable pizzaToppings: - mushrooms - cheese - peppers - onions toppings: |- {{- range $i, $val := .Values.pizzaTopping }} - {{ . | title | quote }} # upper first character, then quote {{- end }} sizes: |- {{- range tuple "small" "medium" "large" }} - {{ . }} {{- end }} # make a quick list

We improve your systems with passion Flow Control 25 Variables data: myvalue: "Hello World" {{- $relname := .Release.Name -}} {{- with .Values.favorite }} drink: {{ .drink | default "tea" | quote }} food: {{ .food | upper | quote }} release: {{ $relname }} {{- end }} # use variable in range toppings: |- {{- range $index, $topping := .Values.pizzaToppings }} {{ $index }}: {{ $topping }} {{- end }} {{- range $key,$value := .Values.favorite }} {{ $key }}: {{ $value }} {{- end }} # instead of specify the key, we can actually loop through the values.yaml file and print values values.yaml  pizzaToppings: - mushrooms - cheese - peppers - onions

We improve your systems with passion Flow Control 26 Nested Templates # _helpers.tpl {{/* Generate basic labels */}} {{- define "my_labels" }} labels: generator: helm date: {{ now | htmlDate }} version: {{ .Chart.Version }} name: {{ .Chart.Name }} {{- end }} # configmap.yaml apiVersion: v1 kind: ConfigMap metadata: name: {{ .Release.Name }}-configmap {{- template "my_labels" . }} # Notice the final dot, it will pass the global scope inside template file. Without it version & name will not be generated. {{- include "my_labels" . | indent 2 }} # similar to "template" directive, have the ability to control indentation

We improve your systems with passion Flow Control 27 Files inside Templates # file located at parent folder # config1.toml: |- # message = config 1 here # config2.toml: |- # message = config 2 here # config3.toml: |- # message = config 3 here data: {{- $file := .Files }} # set variable {{- range tuple "config1.toml" "config2.toml" "config3.toml" }} # create list {{ . }}: |- # config file name {{ $file.Get . }} # get file's content {{- end }}

We improve your systems with passion Flow Control Glob-patterns & encoding apiVersion: v1 kind: ConfigMap metadata: name: conf data: +{{ (.Files.Glob "foo/*").AsConfig | indent 2 }} --- apiVersion: v1 kind: Secret metadata: name: very-secret type: Opaque data: +{{ (.Files.Glob "bar/*").AsSecrets | indent 2 }} +token: |- + {{ .Files.Get "config1.toml" | b64enc }} 28

We improve your systems with passion ImagePullSecrets 29 {{- define "imagePullSecret" }} {{- printf "{\"auths\": {\"%s\": {\"auth\": \"%s\"}}}" .Values.imageCredentials.registry (printf "%s: %s" .Values.imageCredentials.username .Values.imageCredentials.password | b64enc) | b64enc }} {{- end }} imageCredentials: registry: quay.io username: someone password: sillyness apiVersion: v1 kind: Secret metadata: name: imageSecretKey type: kubernetes.io/dockerconfigjson data: .dockerconfigjson: {{ template "imagePullSecret" . }} values.yaml templates/_helper.tpl templates/imageSecretKey.yaml

We improve your systems with passion Autodeploy (dependencies) 30 apiVersion: apps/v1 kind: Deployment spec: template: metadata: annotations: checksum/config: {{ include (print $.Template.BasePath "/configmap.yaml") . | sha256sum }} deployment.yaml or  $ helm upgrade --recreate-pods

We improve your systems with passion Helm hook flow 31 kubernetes manifests pre-install post-install pre-delete post-delete pre-upgrade post-upgrade pre-rollback post-rollback crd-install apiVersion: ... kind: ... metadata: annotations: "helm.sh/hook": "pre-install" # ...

We improve your systems with passion Helm: Hook reference 32 pre-install: Executes after templates are rendered, but before any resources are created in Kubernetes. post-install: Executes after all resources are loaded into Kubernetes pre-delete: Executes on a deletion request before any resources are deleted from Kubernetes. post-delete: Executes on a deletion request after all of the release’s resources have been deleted. pre-upgrade: Executes on an upgrade request after templates are rendered,   but before any resources are loaded into Kubernetes (e.g. before a Kubernetes apply operation).   post-upgrade: Executes on an upgrade after all resources have been upgraded. pre-rollback: Executes on a rollback request after templates are rendered, but before any resources have been rolled back. post-rollback: Executes on a rollback request after all resources have been modiﬁed.   crd-install: Adds CRD resources before any other checks a

We improve your systems with passion Helm hook flow 33 Chart load Chart verification Pre-install hooks Sorted by weight lowest first Wait hooks ready Load chart manifests Post-install hooks sorted by weight lowest first Wait hooks ready Return release name Helm client exists

We improve your systems with passion Test Hooks (templates/tests/*.yaml) 34 apiVersion: v1 kind: Pod metadata: name: "{{ include "whoami.fullname" . }}-test-connection" labels: app.kubernetes.io/name: {{ include "whoami.name" . }} helm.sh/chart: {{ include "whoami.chart" . }} app.kubernetes.io/instance: {{ .Release.Name }} app.kubernetes.io/managed-by: {{ .Release.Service }} annotations: "helm.sh/hook": test-success spec: containers: - name: wget image: busybox command: ['wget'] args: ['{{ include "whoami.fullname" . }}:{{ .Values.service.port }}'] restartPolicy: Never or „helm.sh/hook": test-failure helm test --cleanup whoami

We improve your systems with passion Helm unittest plugin 35 suite: test ingress templates: - ingress.yaml tests: - it: should render nothing if not enabled asserts: - hasDocuments: count: 0 - it: should set annotations if given set: ingress.enabled: true ingress.annotations: kubernetes.io/ingress.class: nginx kubernetes.io/tls-acme: "true" ingress.kubernetes.io/rewrite-target: / asserts: - equal: path: metadata.annotations value: kubernetes.io/ingress.class: nginx kubernetes.io/tls-acme: "true" ingress.kubernetes.io/rewrite-target: / https://github.com/lrills/helm-unittest

We improve your systems with passion 37 Text

We improve your systems with passion Voting app 38 Web Ingress Web Service Vote Vote Pod Vote Pod Redis Master Service Redis Pod Vote Result Pod Vote Result Pod PostgreSQL Service PostgreSQL Pod ReplicaSet Deployment ReplicaSet Deployment Web Service Vote Result Vote Worker Pod ReplicaSet Deployment ReplicaSet Deployment ReplicaSet Deployment DNS & LB /<release-name>-vote /<release-name>-result Redis Master Pod StatefulSet PersistenceVolumeClaim Redis Slave Service Volume (data)

We improve your systems with passion Example: voting app Dependencies Subcharts Templates voting-app-reference ├── Chart.yaml ├── charts │ ├── postgresql-3.7.0.tgz │ ├── redis-5.1.3.tgz │ ├── result │ │ ├── Chart.yaml │ │ ├── templates │ │ │ ├── _helpers.tpl │ │ │ ├── deployment.yaml │ │ │ └── service.yaml │ │ └── values.yaml │ ├── vote │ │ ├── Chart.yaml │ │ ├── templates │ │ │ ├── _helpers.tpl │ │ │ ├── deployment.yaml │ │ │ └── service.yaml │ │ └── values.yaml │ └── worker │ ├── Chart.yaml │ ├── templates │ │ ├── _helpers.tpl │ │ └── deployment.yaml │ └── values.yaml ├── requirements.lock ├── requirements.yaml  ├── templates │ ├── _helpers.tpl │ ├── kcr-imageSecret.yaml │ ├── result-ingress.yaml │ └── vote-ingress.yaml └── values.yaml

We improve your systems with passion Demo 40 • Review app code • Chart and Subchart review • Insides • Tips and Tricks W o rk hard Dream big

We improve your systems with passion 41 http://voting-app-traeﬁ-yujm3v.k8s.customer.bee42.cloud/dev-vote http://voting-app-traeﬁ-yujm3v.k8s.customer.bee42.cloud/dev-result

We improve your systems with passion Kubernetes stacking 42 Your   multi tenant  multi stage multi branches  multi services SYSTEM Kubernetes Platform Application Services k8s Extensions Linux OS vm or bare metal k8s node CRI CNI CSI CLI wireguard operator controller alerts tracing RBAC Security Auth metrics dns lb/ingress ReplicaSet Deployment RBAC ServiceAccount PodSecurityPolicy NetWorkPolicy Limits Auth Pod container container Service Ingress ReplicaSet Deployment RBAC ServiceAccount PodSecurityPolicy NetWorkPolicy Limits Auth Namespace Dev Namespace PreProd Kubernets Backplane Pod container container Service Ingress logging clusterAPI Image repo Backup Vault SCM Pipelines Load Balancer / Router / DNS Artefact repo Service Backplane Extensions Volumes Network Kubernetes extensions

We improve your systems with passion Release a chart with security in mind 43

We improve your systems with passion PodSecurityPolicy 44 apiVersion: policy/v1beta1 kind: PodSecurityPolicy metadata: name: restricted annotations: seccomp.security.alpha.kubernetes.io/allowedProfileNames: 'docker/default' apparmor.security.beta.kubernetes.io/allowedProfileNames: 'runtime/default' seccomp.security.alpha.kubernetes.io/defaultProfileName: 'docker/default' apparmor.security.beta.kubernetes.io/defaultProfileName: 'runtime/default' spec: privileged: false # Required to prevent escalations to root. allowPrivilegeEscalation: false # This is redundant with non-root + disallow privilege escalation, # but we can provide it for defense in depth. requiredDropCapabilities: - ALL # Allow core volume types. volumes: - 'configMap' - 'emptyDir' - 'projected' - 'secret' - 'downwardAPI' # Assume that persistentVolumes set up by the cluster admin are safe to use. - 'persistentVolumeClaim' hostNetwork: false hostIPC: false hostPID: false runAsUser: # Require the container to run without root privileges. rule: 'MustRunAsNonRoot' seLinux: # This policy assumes the nodes are using AppArmor rather than SELinux. rule: 'RunAsAny' supplementalGroups: rule: 'MustRunAs' ranges: # Forbid adding the root group. - min: 1 max: 65535 fsGroup: rule: 'MustRunAs' ranges: # Forbid adding the root group. - min: 1 max: 65535 readOnlyRootFilesystem: false https://kubernetes.io/docs/concepts/policy/pod-security-policy/

We improve your systems with passion Lessons learned 45 • Use the immutable deployment pattern • Only system pods need access to the Api Server • Add PodSecurityPolicy and limits of the resource usages • Limit the network access with a NetworkPolicy • Start with a Deny All Policy • Use multiple tiller with limit RBAC roles and track your K8s cluster users • Check your container images with CVE Checkers (Clair, NeuVector, AquaSec, … ) • Add more Know How to your teams: Teach your Ops and Dev colleagues • Share your Knowledge and go to events, conferences and meetups • Think about Chaos Engineering: Replace your system components every time and do that really! • Automate all what you can and never stop this doing! • Don`t allow manuell manipulation at your K8s Cluster

We improve your systems with passion 46 Container System Blueprint: Beehive Orchestration Provisioning Security Conﬁg Metrics Alarming Logging Traceing Artefact Registry Vault Auth Source Code CI/CD Pipeline Storage Network API Gateway Routing Loadbalancer, DNS and Firewall On Premise Cloud Operating Systems Backends VM machines Databases Bare metal Container machines Container machines Data Bastion

We improve your systems with passion 48 We hiring :-)   https://bit.ly/2K8DtRu   [email protected]  @bee42solutions

We improve your systems with passion rethink IT Build fashionable container systems with Kubernetes 49

We improve your systems with passion bee42 Trainings 50 https://bee42.com/de/trainings

We improve your systems with passion Kubernetes poster Get a K8s poster today 51 https://bee42.com/de/kubernetes-poster/ PREVIEW PREVIEW WE

We improve your systems with passion Cloud Native System Architect & bee42 founder Peter Roßbach @PRossbach [email protected] https://bee42.com https://devops-gathering.io 52

We improve your systems with passion References 53 [Sprig library](https://godoc.org/github.com/Masterminds/sprig [Deis Workflow](https://github.com/deis/workflow/tree/master/charts/workflow)) [OpenStack chart](https://github.com/sapcc/openstack-helm) [Helm.sh](https://helm.sh) [Bitnami charts](https://github.com/bitnami/charts) [kubernetes charts](https://github.com/kubernetes/charts)

Helm: Container Systeme mit Kubernetes erfolgreich gestalten

Helm: Container Systeme mit Kubernetes erfolgreich gestalten

More Decks by Peter Rossbach

Other Decks in Technology

Featured

Transcript