Helm: Build fashionable container systems with Kubernetes

Copyright 2018 bee42 solutions gmbh @PRossbach
rethink IT - We improve your systems with passion
1
HELM
Build fashionable
container systems
with Kubernetes

View Slide

HELM
2
Originally developed by Deis 2015 and  
donated to Cloud Native Computing Foundation 2018
Second generation available  
and third generation development is started at July 2018
Package manager for Kubernetes clusters

View Slide

What is Helm?
3
•Helm charts are build on top of Kubernetes manifests
•Charts are stored in a registry called Helm Museum
•It use a templating preprocessing and managed release of Kubernetes resources
•Ability to consider scalability from the get-go
•SRE’s and developer can search of charts and scratch it.

View Slide

4
• Kubernetes is a container orchestrator.
• It’s how to run containers at scale.
• It’s a very active open-source platform with lots of
contributors, start at 6. June 2014
• Originally developed by Google and  
donated to Cloud Native Computing Foundation

View Slide

5

View Slide

Kubernetes
manifests
6

View Slide

K8s Deployment Challenge
7
Move to microservice or serverless architecture.
Applications consists from multiple components
each component has its own resources and can be scaled individually.
It’s hard to ...
... manage, edit and update multiple K8s configurations
... deploy multiple K8s configurations as a SINGLE application
... share and reuse K8s configurations and applications
... parametrize and support multiple environments
... manage application releases: rollout, rollback, diff, history
... define deployment lifecycle (control operations to be run in different phases)
... validate release state after deployment

View Slide

What Helm can do for you?
8
•Create new charts, aka package K8s manifests, from scratch
•Package charts into chart archive (tgz) ﬁle
•Interact with chart repositories where charts are stored
•Install and uninstall charts into an existing Kubernetes cluster
•Manage the release cycle of charts that have been installed with Helm

View Slide

Chart
9
a package of Kubernetes manifests
Release a chart instance is loaded into Kubernetes
Repository a download area of published charts
Template a Kubernetes conﬁguration ﬁle mixed with Go/Sprig templates
Concepts

View Slide

How Helm 2 works?
10
client
api-service
tiller service(s)
https
grpc
brew install kubernetes-helm
curl https://raw.githubusercontent.com/kubernetes/helm/master/scripts/get | bash
or

View Slide

helm init
11
$ cat >tiller-rbac.yaml <apiVersion: v1
kind: ServiceAccount
metadata:
name: tiller
namespace: kube-system
---
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: ClusterRoleBinding
metadata:
name: tiller
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: cluster-admin
subjects:
- kind: ServiceAccount
name: tiller
namespace: kube-system
EOF
$ kubectl create -f tiller-rbac.yaml
$ helm init --service-account tiller --upgrade

View Slide

Helm cli commands
12
completion Generate autocompletions script for the specified shell (bash or zsh)
create create a new chart with the given name
delete given a release name, delete the release from Kubernetes
dependency manage a chart's dependencies
fetch download a chart from a repository and (optionally) unpack it in local directory
get download a named release
history fetch release history
home displays the location of HELM_HOME
init initialize Helm on both client and server
inspect inspect a chart
install install a chart archive
lint examines a chart for possible issues
list list releases
package package a chart directory into a chart archive
plugin add, list, or remove Helm plugins
repo add, list, remove, update, and index chart repositories
reset uninstalls Tiller from a cluster
rollback roll back a release to a previous revision
search search for a keyword in charts
serve start a local http web server
status displays the status of the named release
template locally render templates
test test a release
upgrade upgrade a release
verify verify that a chart at the given path has been signed and is valid
version print the client/server version information

View Slide

Helm flow
13
client
api-service
tiller service(s)
publish
fetch
install
upgrade
delete
redis
create
application
tomcat
postgres

View Slide

Example: Simple config map
14
$ helm create my-chart
# review mychart templates
$ rm -rf mychart/templates/*.*
$ cat <mychart/templates/configmap.yaml
apiVersion: v1
kind: ConfigMap
metadata:
name: mychart-configmap
data:
myvalue: "Hello World"
EOF
$ helm install --dry-run --debug ./mychart
$ helm install --name full-coral --namespace test ./mychart
$ helm get manifest full-coral
$ helm delete full-coral

View Slide

Example: Prometheus & Grafana
15
# pre init 
# —————————————————————————————————————————————————————————————————————————
$ helm init --client-only
$ helm repo add stable \
https://kubernetes-charts.storage.googleapis.com/
$ helm repo add incubator \
https://kubernetes-charts-incubator.storage.googleapis.com/
$ helm repo update
# install prometheus
# —————————————————————————————————————————————————————————————————————————
$ helm install stable/prometheus \
--name prometheus \ 
--namespace monitoring \
--set rbac.create=true,server.persistentVolume.enabled=false,\
alertmanager.enabled=false,pushgateway.enabled=false
# install grafana 
# —————————————————————————————————————————————————————————————————————————
$ helm install ./grafana \
--name grafana \
--namespace monitoring \
--set server.persistentVolume.enabled=false,server.adminPassword=admin

View Slide

Example: Traefik
16
$ helm install stable/traefik \ 
--name traefik --namespace kube-system \ 
--set rbac.enabled=true,serviceType=NodePort
$ kubectl run —image=bee42/whoami:2.0.0 \
--expose --port=80 --target-port=80 --name=whoami  
$ cat >whoami-ingress.yaml <apiVersion: extensions/v1beta1
kind: Ingress
metadata:
name: whoami 
annotations: 
kubernetes.io/ingress.class=traefik
spec:
rules:
- http:
paths:
- path: /
backend:
serviceName: whoami
servicePort: http
EOF
$ kubectl apply -f whoami-ingress.yaml
$ curl :

View Slide

Helm chart directory
17

├── Chart.yaml
├── .md
├── .md
├── charts
│ ├── .tgz
│ ├──
│ │ ├── Chart.yaml
│ │ ├── templates
│ │ │ ├── _helpers.tpl
│ │ │ └── .yaml
│ │ └── values.yaml
├── requirements.lock
├── requirements.yaml 
├── templates
│ ├── _helpers.tpl
│ ├── NOTES.txt
│ └── .yaml
└── values.yaml

View Slide

Chart.yaml
18
apiVersion: v1
description: A Helm chart for Docker Voting App porting to k8s cluster
name: voting-app
version: 0.3.2
keywords:
- demo
- kubernetes
- helm
- voting-app
home: https://bee42.com/
icon: https://bee42.com/images/logo.svg
sources:
- https://gitlab.bee42.com/kubernetes/examples/voting-app
maintainers:
- name: Peter Rossbach
email: [email protected]

View Slide

values.yaml
19
# default ingress port
servicePort: 80
# configure subchart redis
redis:
master:
persistence:
enabled: false
usePassword: false
… 
imageCredentials:
registry: r.gitlab.bee42.com
username: gitlab+deploy-token-1
password: xxx-xxx
# ingress
ingress:
enabled: true
nginx:
enabled: true
traefik:
enabled: false

View Slide

requirements.yaml
20
dependencies:
- name: postgresql
version: 0.13.1
repository: https://kubernetes-charts.storage.googleapis.com/
alias: postgresql-1
- name: redis
version: 3.3.6
repository: https://kubernetes-charts.storage.googleapis.com/
condition: postgresql-1.enabled,global.result.enabled
tags:
- back-end
- subchart2

View Slide

_helper.tpl
21
{{/* vim: set filetype=mustache: */}}
{{/*
Expand the name of the chart.
*/}}
{{- define "name" -}}
{{- default .Chart.Name .Values.nameOverride | trunc 63 | trimSuffix "-" -}}
{{- end -}}
{{/*
Create a default fully qualified app name.
We truncate at 63 chars because some Kubernetes name fields are limited to this (by the DNS naming spec).
*/}}
{{- define "fullname" -}}
{{- $name := default .Chart.Name .Values.nameOverride -}}
{{- printf "%s-%s" .Release.Name $name | trunc 63 | trimSuffix "-" -}}
{{- end -}}
{{/*
Create a registry image secret to pull voting app images
*/}}
{{- define "imagePullSecret" }}
{{- printf "{\"auths\": {\"%s\": {\"auth\": \"%s\"}}}" .Values.imageCredentials.registry  
printf "%s:%s" .Values.imageCredentials.username .Values.imageCredentials.password | b64enc) | b64enc }}
{{- end }}

View Slide

Flow Control
22
If/Else
{{ if PIPELINE }}
# Do something
{{ else if OTHER PIPELINE }}
# Do something else
{{ else }}
# Default case
{{ end }}
data:
drink: {{ .Values.favorite.drink | default "tea" | quote }}
food: {{ .Values.favorite.food | upper | quote }}
{{- if eq .Values.favorite.drink "lemonade" }}
mug: true
{{- end }} # notice the "-" in the left, if will help eliminate newline before variable

View Slide

Flow Control
23
With
data:
{{- with .Values.favorite }}
drink: {{ .drink | default "tea" | quote }}
food: {{ .food | upper | quote }}
{{- end }} # instead of writing ".Values.favorite.drink"

View Slide

Flow Control
24
Range
# predefined variable
pizzaToppings:
- mushrooms
- cheese
- peppers
- onions
toppings: |-
{{- range $i, $val := .Values.pizzaTopping }}
- {{ . | title | quote }} # upper first character, then quote
{{- end }}
sizes: |-
{{- range tuple "small" "medium" "large" }}
- {{ . }}
{{- end }} # make a quick list

View Slide

Flow Control
25
Variables
data:
{{- $relname := .Release.Name -}}
{{- with .Values.favorite }}
drink: {{ .drink | default "tea" | quote }}
food: {{ .food | upper | quote }}
release: {{ $relname }}
{{- end }}
# use variable in range
toppings: |-
{{- range $index, $topping := .Values.pizzaToppings }}
{{ $index }}: {{ $topping }}
{{- end }}
{{- range $key,$value := .Values.favorite }}
{{ $key }}: {{ $value }}
{{- end }} # instead of specify the key, we can actually loop through the values.yaml file
and print values
values.yaml 
pizzaToppings:
- mushrooms
- cheese
- peppers
- onions

View Slide

Flow Control
26
Nested Templates
# _helpers.tpl
{{/* Generate basic labels */}}
{{- define "my_labels" }}
labels:
generator: helm
date: {{ now | htmlDate }}
version: {{ .Chart.Version }}
name: {{ .Chart.Name }}
{{- end }}
# configmap.yaml
apiVersion: v1
kind: ConfigMap
metadata:
name: {{ .Release.Name }}-configmap
{{- template "my_labels" . }} # Notice the final dot, it will pass the global scope inside template
file. Without it version & name will not be generated.
{{- include "my_labels" . | indent 2 }} # similar to "template" directive, have the ability to
control indentation

View Slide

Flow Control
27
Files inside Templates
# file located at parent folder
# config1.toml: |-
# message = config 1 here
# config2.toml: |-
# config3.toml: |-
data:
{{- $file := .Files }} # set variable
{{- range tuple "config1.toml" "config2.toml" "config3.toml" }} # create list
{{ . }}: |- # config file name
{{ $file.Get . }} # get file's content
{{- end }}

View Slide

Flow Control
Glob-patterns & encoding
apiVersion: v1
kind: ConfigMap
metadata:
name: conf
data:
+{{ (.Files.Glob "foo/*").AsConfig | indent 2 }}
---
apiVersion: v1
kind: Secret
metadata:
name: very-secret
type: Opaque
data:
+{{ (.Files.Glob "bar/*").AsSecrets | indent 2 }}
+token: |-
+ {{ .Files.Get "config1.toml" | b64enc }}
28

View Slide

ImagePullSecrets
29
{{- define "imagePullSecret" }}
{{- printf "{\"auths\": {\"%s\": {\"auth\": \"%s\"}}}" .Values.imageCredentials.registry (printf "%s:
%s" .Values.imageCredentials.username .Values.imageCredentials.password | b64enc) | b64enc }}
{{- end }}
imageCredentials:
registry: quay.io
username: someone
password: sillyness
apiVersion: v1
kind: Secret
metadata:
name: imageSecretKey
type: kubernetes.io/dockerconfigjson
data:
.dockerconfigjson: {{ template "imagePullSecret" . }}
values.yaml
templates/_helper.tpl
templates/imageSecretKey.yaml

View Slide

Autodeploy
30
apiVersion: apps/v1
kind: Deployment
spec:
template:
metadata:
annotations:
checksum/config: {{ include (print $.Template.BasePath "/configmap.yaml") . | sha256sum }}
deployment.yaml
or 
$ helm upgrade --recreate-pods

View Slide

Helm hook flow
31
kubernetes
manifests
pre-install
post-install
pre-delete
post-delete
pre-upgrade
post-upgrade
pre-rollback
post-rollback
crd-install
apiVersion: ...
kind: ...
metadata:
annotations:
"helm.sh/hook": "pre-install"
# ...

View Slide

Helm: Hook reference
32
pre-install: Executes after templates are rendered, but before any resources are created in Kubernetes.
post-install: Executes after all resources are loaded into Kubernetes
pre-delete: Executes on a deletion request before any resources are deleted from Kubernetes.
post-delete: Executes on a deletion request after all of the release’s resources have been deleted.
pre-upgrade: Executes on an upgrade request after templates are rendered,  
but before any resources are loaded into Kubernetes (e.g. before a Kubernetes apply operation).
 
post-upgrade: Executes on an upgrade after all resources have been upgraded.
pre-rollback: Executes on a rollback request after templates are rendered, but before any resources have been rolled back.
post-rollback: Executes on a rollback request after all resources have been modiﬁed.
 
crd-install: Adds CRD resources before any other checks a

View Slide

Helm hook flow
33
Chart load Chart verification
Pre-install hooks
Sorted by weight lowest first
Wait hooks ready
Load chart manifests
Post-install hooks
sorted by weight lowest first
Wait hooks ready
Return
release name
Helm client exists

View Slide

34

View Slide

35
Text

View Slide

Voting app
36
Web Ingress
Web Service Vote
Vote Pod Vote Pod
Redis Master Service
Redis Pod
Vote Result Pod Vote Result Pod
PostgreSQL Service
PostgreSQL Pod
ReplicaSet
Deployment
ReplicaSet
Deployment
Web Service Vote Result
Vote Worker Pod
ReplicaSet
Deployment
ReplicaSet
Deployment
ReplicaSet
Deployment
DNS & LB
/-vote /-result
Redis Master Pod
StatefulSet
PersistenceVolumeClaim
Redis Slave Service
Volume (data)

View Slide

Example: voting app
Dependencies
Subcharts
Templates
voting-app-reference
├── Chart.yaml
├── charts
│ ├── postgresql-0.13.1.tgz
│ ├── redis-3.3.6.tgz
│ ├── result
│ │ │ ├── deployment.yaml
│ │ │ └── service.yaml
│ ├── vote
│ │ │ ├── deployment.yaml
│ │ │ └── service.yaml
│ └── worker
│ ├── Chart.yaml
│ ├── templates
│ │ ├── _helpers.tpl
│ │ └── deployment.yaml
│ └── values.yaml
├── requirements.lock
├── requirements.yaml 
├── templates
│ ├── _helpers.tpl
│ ├── kcr-imageSecret.yaml
│ ├── result-ingress.yaml
│ └── vote-ingress.yaml
└── values.yaml

View Slide

Demo
38
• Review app code
• Chart and Subchart review
• Insides
• Tips and Tricks
W
o
rk
hard
Dream
big

View Slide

39
Text
http://voting-app-traeﬁ-yujm3v.k8s.customer.bee42.cloud/dev-vote
http://voting-app-traeﬁ-yujm3v.k8s.customer.bee42.cloud/dev-result

View Slide

Kubernets stacking
40
Your  
multi tenant 
multi stage
multi branches 
multi services
SYSTEM
Kubernetes Platform
Application Services
k8s Extensions
Linux OS
vm or bare metal
k8s node
CRI CNI CSI CLI
wireguard
operator
controller
alerts tracing
RBAC
Security
Auth
metrics
dns lb/ingress
ReplicaSet
Deployment
RBAC
ServiceAccount
PodSecurityPolicy
NetWorkPolicy
Limits
Auth
Pod
container
container
Service
Ingress
ReplicaSet
Deployment
RBAC
ServiceAccount
PodSecurityPolicy
NetWorkPolicy
Limits
Auth
Namespace Dev Namespace PreProd
Kubernets Backplane
Pod
container
container
Service
Ingress
logging
clusterAPI
Image repo
Backup
Vault
SCM
Pipelines
Load Balancer / Router / DNS
Artefact repo
Service Backplane
Extensions
Volumes Network
Kubernetes extensions

View Slide

Release a chart with security in mind
41

View Slide

PodSecurityPolicy
42
apiVersion: policy/v1beta1
kind: PodSecurityPolicy
metadata:
name: restricted
annotations:
seccomp.security.alpha.kubernetes.io/allowedProfileNames: 'docker/default'
apparmor.security.beta.kubernetes.io/allowedProfileNames: 'runtime/default'
seccomp.security.alpha.kubernetes.io/defaultProfileName: 'docker/default'
apparmor.security.beta.kubernetes.io/defaultProfileName: 'runtime/default'
spec:
privileged: false
# Required to prevent escalations to root.
allowPrivilegeEscalation: false
# This is redundant with non-root + disallow privilege escalation,
# but we can provide it for defense in depth.
requiredDropCapabilities:
- ALL
# Allow core volume types.
volumes:
- 'configMap'
- 'emptyDir'
- 'projected'
- 'secret'
- 'downwardAPI'
# Assume that persistentVolumes set up by the cluster admin are safe to use.
- 'persistentVolumeClaim'
hostNetwork: false
hostIPC: false
hostPID: false
runAsUser:
# Require the container to run without root privileges.
rule: 'MustRunAsNonRoot'
seLinux:
# This policy assumes the nodes are using AppArmor rather than SELinux.
rule: 'RunAsAny'
supplementalGroups:
rule: 'MustRunAs'
ranges:
# Forbid adding the root group.
- min: 1
max: 65535
fsGroup:
rule: 'MustRunAs'
ranges:
# Forbid adding the root group.
- min: 1
max: 65535
readOnlyRootFilesystem: false
https://kubernetes.io/docs/concepts/policy/pod-security-policy/

View Slide

Lessons learned
43
• Use the immutable deployment pattern
• Only system pods need access to the Api Server
• Add PodSecurityPolicy and limits of the resource usages
• Limit the network access with a NetworkPolicy
• Start with a Deny All Policy
• Use multiple tiller with limit RBAC roles and track your K8s cluster users
• Check your container images with CVE Checkers (Clair, NeuVector, AquaSec, … )
• Add more Know How to your teams: Teach your Ops and Dev colleagues
• Share your Knowledge and go to events, conferences and meetups
• Think about Chaos Engineering: Replace your system components every time and do that really!
• Automate all what you can and never stop this doing!
• Don`t allow manuell manipulation at your K8s Cluster

View Slide

44
Container System Blueprint: Beehive
Orchestration
Provisioning
Security
Conﬁg
Metrics
Alarming
Logging
Traceing
Artefact
Registry
Vault
Auth
Source Code
CI/CD Pipeline
Storage Network
API Gateway Routing
Loadbalancer, DNS and Firewall
On Premise Cloud
Operating Systems
Backends
VM machines
Databases
Bare metal
Container machines
Container machines
Data
Bastion

View Slide

CSSC: Container System Supply Chain
45

View Slide

46

View Slide

47
We hiring :-)
 
https://bit.ly/2K8DtRu  
[email protected] 
@bee42solutions

View Slide

rethink IT
Build fashionable
container systems
with Kubernetes
48

View Slide

bee42 Trainings
49
https://bee42.com/de/trainings

View Slide

Kubernetes poster
pre registration started

https://tinyurl.com/y9js3p7w
50
delivery starts today
PREVIEW
PREVIEW
WE

View Slide

Cloud Native System Architect
&
bee42 founder
Peter Roßbach

@PRossbach

[email protected]

https://bee42.com

https://devops-gathering.io

51
Save the date…
#DOG19 11.-13. March 2019 at Bochum

View Slide

References
52
[Sprig library](https://godoc.org/github.com/Masterminds/sprig
[Deis Workflow](https://github.com/deis/workflow/tree/master/charts/workflow))
[OpenStack chart](https://github.com/sapcc/openstack-helm)
[Helm.sh](https://helm.sh)
[Bitnami charts](https://github.com/bitnami/charts)
[kubernetes charts](https://github.com/kubernetes/charts)

View Slide

Helm: Build fashionable container systems with Kubernetes

Helm: Build fashionable container systems with Kubernetes

Peter Rossbach

More Decks by Peter Rossbach

Other Decks in Technology

Featured

Transcript