Helm: Build fashionable container systems with Kubernetes

Copyright 2018 bee42 solutions gmbh @PRossbach
rethink IT - improve your systems with passion
1
Helm::
Build fashionable
container systems
with Kubernetes

View Slide

Kubernetes cluster
2
• Kubernetes is container orchestration 
It enables container to scale
• It’s a very active open-source project with lots of
contributors, started at 6. June 2014
• Originally developed by Google and  
donated to Cloud Native Computing Foundation

View Slide

K8s cluster architecture
3
k8s master
k8s minion node
api server
controller
manager
scheduler
ETCD
kubelet
kubeproxy
kubelet kubeproxy
pod
Containers
Containers
pod
Containers
Containers
pod
Containers
Containers
cri
cri
k8s minion node
kubelet kubeproxy
pod
Containers
Containers
pod
Containers
Containers
pod
Containers
Containers
cri
DevOps
User

View Slide

K8s Deployment Challenge
4
Move to microservice architecture
application consists from multiple components
each component has its own resources and can be scaled individually
It’s hard to ...
... manage, edit and update multiple K8s configurations
... deploy multiple K8s configurations as a SINGLE application
... share and reuse K8s configurations and applications
... parametrize and support multiple environments
... manage application releases: rollout, rollback, diff, history
... define deployment lifecycle (control operations to be run in different phases)
... validate release state after deployment

View Slide

What helm can do for you?
5
Create new charts, aka package k8s manifests, from scratch
Package charts into chart archive (tgz) ﬁle
Interact with chart repositories where charts are stored
Install and uninstall charts into an existing kubernetes cluster
Manage the release cycle of charts that have been installed with helm

View Slide

Chart
6
a package of kubernetes manifests
Release a chart instance is loaded into kubernetes
Repository a download area of published charts
Template a kubernetes conﬁguration ﬁle mixed with Go/Sprig templates

View Slide

How helm works?
7
client
api-service
tiller service(s)
https
grpc
brew install kubernetes-helm
curl https://raw.githubusercontent.com/kubernetes/helm/master/scripts/get | bash
or

View Slide

helm init
8
$ kubectl create serviceaccount tiller --namespace kube-system
$ cat >tiller-rbac.ymal <apiVersion: v1
kind: ServiceAccount
metadata:
name: tiller
namespace: kube-system
---
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: ClusterRoleBinding
metadata:
name: tiller
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: cluster-admin
subjects:
- kind: ServiceAccount
name: tiller
namespace: kube-system
EOF
$ kubectl create -f tiller-rbac.yaml
$ helm init --service-account tiller —upgrade

View Slide

helm cli commands
9
completion Generate autocompletions script for the specified shell (bash or zsh)
create create a new chart with the given name
delete given a release name, delete the release from Kubernetes
dependency manage a chart's dependencies
fetch download a chart from a repository and (optionally) unpack it in local directory
get download a named release
history fetch release history
home displays the location of HELM_HOME
init initialize Helm on both client and server
inspect inspect a chart
install install a chart archive
lint examines a chart for possible issues
list list releases
package package a chart directory into a chart archive
plugin add, list, or remove Helm plugins
repo add, list, remove, update, and index chart repositories
reset uninstalls Tiller from a cluster
rollback roll back a release to a previous revision
search search for a keyword in charts
serve start a local http web server
status displays the status of the named release
template locally render templates
test test a release
upgrade upgrade a release
verify verify that a chart at the given path has been signed and is valid
version print the client/server version information

View Slide

Helm flow
10
client
api-service
tiller service(s)
publish
fetch
install
upgrade
delete
redis
create
application
tomcat
postgres

View Slide

Example: Simple config map
11
$ helm create my-chart
# review mychart templates
$ rm -rf mychart/templates/*.*
$ cat <mychart/templates/configmap.yaml
apiVersion: v1
kind: ConfigMap
metadata:
name: mychart-configmap
data:
myvalue: "Hello World"
EOF
$ helm install --dry-run --debug ./mychart
$ helm install --name full-coral --namespace test ./mychart
$ helm get manifest full-coral
$ helm delete full-coral

View Slide

Example: Prometheus & Grafana
12
# pre init 
# —————————————————————————————————————————————————————————————————————————
$ helm init --client-only
$ helm repo add stable \
https://kubernetes-charts.storage.googleapis.com/
$ helm repo add incubator \
https://kubernetes-charts-incubator.storage.googleapis.com/
$ helm repo update
# install prometheus
# —————————————————————————————————————————————————————————————————————————
$ helm install stable/prometheus \
--name prometheus \ 
--namespace monitoring \
--set rbac.create=true,server.persistentVolume.enabled=false,\
alertmanager.enabled=false,pushgateway.enabled=false
# install grafana 
# —————————————————————————————————————————————————————————————————————————
$ helm install ./grafana \
--name grafana \
--namespace monitoring \
--set server.persistentVolume.enabled=false,server.adminPassword=admin

View Slide

Flow Control
13
If/Else
{{ if PIPELINE }}
# Do something
{{ else if OTHER PIPELINE }}
# Do something else
{{ else }}
# Default case
{{ end }}
data:
drink: {{ .Values.favorite.drink | default "tea" | quote }}
food: {{ .Values.favorite.food | upper | quote }}
{{- if eq .Values.favorite.drink "lemonade" }}
mug: true
{{- end }} # notice the "-" in the left, if will help eliminate newline before variable

View Slide

Flow Control
14
With
data:
{{- with .Values.favorite }}
drink: {{ .drink | default "tea" | quote }}
food: {{ .food | upper | quote }}
{{- end }} # instead of writing ".Values.favorite.drink"

View Slide

Flow Control
15
Range
# predefined variable
pizzaToppings:
- mushrooms
- cheese
- peppers
- onions
toppings: |-
{{- range $i, $val := .Values.pizzaTopping }}
- {{ . | title | quote }} # upper first character, then quote
{{- end }}
sizes: |-
{{- range tuple "small" "medium" "large" }}
- {{ . }}
{{- end }} # make a quick list

View Slide

Flow Control
16
Variables
data:
{{- $relname := .Release.Name -}}
{{- with .Values.favorite }}
drink: {{ .drink | default "tea" | quote }}
food: {{ .food | upper | quote }}
release: {{ $relname }}
{{- end }}
# use variable in range
toppings: |-
{{- range $index, $topping := .Values.pizzaToppings }}
{{ $index }}: {{ $topping }}
{{- end }}
{{- range $key,$value := .Values.favorite }}
{{ $key }}: {{ $value }}
{{- end }} # instead of specify the key, we can actually loop through the values.yaml file and print values
values.yaml 
pizzaToppings:
- mushrooms
- cheese
- peppers
- onions

View Slide

Flow Control
17
Nested Templates
# _helpers.tpl
{{/* Generate basic labels */}}
{{- define "my_labels" }}
labels:
generator: helm
date: {{ now | htmlDate }}
version: {{ .Chart.Version }}
name: {{ .Chart.Name }}
{{- end }}
# configmap.yaml
apiVersion: v1
kind: ConfigMap
metadata:
name: {{ .Release.Name }}-configmap
{{- template "my_labels" . }} # Notice the final dot, it will pass the global scope inside template file.
Without it version & name will not be generated.
{{- include "my_labels" . | indent 2 }} # similar to "template" directive, have the ability to control
indentation

View Slide

Flow Control
18
Files inside Templates
# file located at parent folder
# config1.toml: |-
# message = config 1 here
# config2.toml: |-
# config3.toml: |-
data:
{{- $file := .Files }} # set variable
{{- range tuple "config1.toml" "config2.toml" "config3.toml" }} # create list
{{ . }}: |- # config file name
{{ $file.Get . }} # get file's content
{{- end }}

View Slide

Flow Control
Glob-patterns & encoding
apiVersion: v1
kind: ConfigMap
metadata:
name: conf
data:
+{{ (.Files.Glob "foo/*").AsConfig | indent 2 }}
---
apiVersion: v1
kind: Secret
metadata:
name: very-secret
type: Opaque
data:
+{{ (.Files.Glob "bar/*").AsSecrets | indent 2 }}
+token: |-
+ {{ .Files.Get "config1.toml" | b64enc }}
19

View Slide

ImagePullSecrets
20
{{- define "imagePullSecret" }}
{{- printf "{\"auths\": {\"%s\": {\"auth\": \"%s\"}}}" .Values.imageCredentials.registry (printf "%s:
%s" .Values.imageCredentials.username .Values.imageCredentials.password | b64enc) | b64enc }}
{{- end }}
imageCredentials:
registry: quay.io
username: someone
password: sillyness
apiVersion: v1
kind: Secret
metadata:
name: imageSecretKey
type: kubernetes.io/dockerconfigjson
data:
.dockerconfigjson: {{ template "imagePullSecret" . }}
values.yaml
templates/_helper.tpl
templates/imageSecretKey.yaml

View Slide

Autodeploy
21
apiVersion: apps/v1
kind: Deployment
spec:
template:
metadata:
annotations:
checksum/config: {{ include (print $.Template.BasePath "/configmap.yaml") . | sha256sum }}
deployment.yaml
or 
$ helm upgrade --recreate-pods

View Slide

Helm hook flow
22
Chart load Chart verification
Pre-install hooks
Sorted by weight lowest first
Wait hooks ready
Load chart manifests
Post-install hooks
sorted by weight lowest first
Wait hooks ready
Return
release name
Helm client exists

View Slide

Helm hook flow
23
kubernetes
manifests
pre-install
post-install
pre-delete
post-delete
pre-upgrade
post-upgrade
pre-rollback
post-rollback
crd-install
apiVersion: ...
kind: ...
metadata:
annotations:
"helm.sh/hook": "pre-install"
# ...

View Slide

Microservice with k8s manifests
24
POD POD POD
Deployment
ReplicaSet
Service
Ingress
Ingress
Proxy
Endpoints
Loadbalancer
watch
service
watch
Req/Res
selector
manage
manage
Req/Res
manage
group of containers
contaner
contaner
contaner
contaner
contaner
contaner
pod of whales
manage
:-)

View Slide

Voting app
25
Web Ingress
Web Service Vote
Vote Pod Vote Pod
Redis Master Service
Redis Pod
Vote Result Pod Vote Result Pod
PostgreSQL Service
PostgreSQL Pod
ReplicaSet
Deployment
ReplicaSet
Deployment
Web Service Vote Result
Vote Worker Pod
ReplicaSet
Deployment
ReplicaSet
Deployment
ReplicaSet
Deployment
DNS & LB
/-vote /-result
Redis Master Pod
StatefulSet
PersistenceVolumeClaim
Redis Slave Service
Volume (data)

View Slide

Example: voting app
voting-app-reference
├── Chart.yaml
├── charts
│ ├── postgresql-0.13.1.tgz
│ ├── redis-3.3.6.tgz
│ ├── result-0.1.6.tgz
│ ├── vote-0.1.6.tgz
│ └── worker-0.1.5.tgz
├── requirements.lock
├── requirements.yaml 
├── components
│ ├── result
│ │ ├── Chart.yaml
│ │ ├── templates
│ │ │ ├── _helpers.tpl
│ │ │ ├── deployment.yaml
│ │ │ └── service.yaml
│ │ └── values.yaml
│ ├── vote
│ │ ├── Chart.yaml
│ │ ├── templates
│ │ │ ├── _helpers.tpl
│ │ │ ├── deployment.yaml
│ │ │ └── service.yaml
│ │ └── values.yaml
│ └── worker
│ ├── Chart.yaml
│ ├── templates
│ │ ├── _helpers.tpl
│ │ └── deployment.yaml
│ └── values.yaml
├── templates
│ ├── _helpers.tpl
│ ├── kcr-imageSecret.yaml
│ ├── result-ingress.yaml
│ └── vote-ingress.yaml
└── values.yaml
Dependencies
Subcharts
Templates

View Slide

Demo
27
• Review app code
• Chart and Subchart review
• Insides
• Tips and Tricks
W
o
rk
hard
Dream
big

View Slide

Kubernets stacking
28
Application Services
Kubernetes extensions
Linux OS
vm or bare metal
Kubernetes minion
CRI CNI CSI CLI
wireguard
operator
controller
clusterAPI
logging tracing
RBAC
Security
Auth
network
storage
metrics
DNS LB
Pod
contaner
contaner
Servvice
Ingress
ReplicaSet
Deployment
RBAC
ServiceAccount
PodSecurityPolicy
NetWorkPolicy
Limits
Auth
Pod
contaner
contaner
Servvice
Ingress
ReplicaSet
Deployment
RBAC
ServiceAccount
PodSecurityPolicy
NetWorkPolicy
Limits
Auth
Namespace Dev Namespace PreProd
Kubernets Backplane
Your multi tenant, multistage or 
multi branch services
Kubernetes Platform

View Slide

Release a chart with security in mind
29
PodSecurityPolicy
NetworkPolicy
Deployment
ReplicaSet
ClusterRole
ClusterRoleBinding
helm client
kubeconﬁg
api-server
tiller service tiller deploy pod
tiller release
conﬁgmap
Service
Account
Service Pod
Ingress
Release Namespace X
Role
RoleBinding
Service
Account Limits
Endpoints Containers
Release
grpc
https
kube-system Namespace
ClusterRole
ClusterRoleBinding
Service
Account

View Slide

PodSecurityPolicy
30
apiVersion: policy/v1beta1
kind: PodSecurityPolicy
metadata:
name: restricted
annotations:
seccomp.security.alpha.kubernetes.io/allowedProfileNames: 'docker/default'
apparmor.security.beta.kubernetes.io/allowedProfileNames: 'runtime/default'
seccomp.security.alpha.kubernetes.io/defaultProfileName: 'docker/default'
apparmor.security.beta.kubernetes.io/defaultProfileName: 'runtime/default'
spec:
privileged: false
# Required to prevent escalations to root.
allowPrivilegeEscalation: false
# This is redundant with non-root + disallow privilege escalation,
# but we can provide it for defense in depth.
requiredDropCapabilities:
- ALL
# Allow core volume types.
volumes:
- 'configMap'
- 'emptyDir'
- 'projected'
- 'secret'
- 'downwardAPI'
# Assume that persistentVolumes set up by the cluster admin are safe to use.
- 'persistentVolumeClaim'
hostNetwork: false
hostIPC: false
hostPID: false
runAsUser:
# Require the container to run without root privileges.
rule: 'MustRunAsNonRoot'
seLinux:
# This policy assumes the nodes are using AppArmor rather than SELinux.
rule: 'RunAsAny'
supplementalGroups:
rule: 'MustRunAs'
ranges:
# Forbid adding the root group.
- min: 1
max: 65535
fsGroup:
rule: 'MustRunAs'
ranges:
# Forbid adding the root group.
- min: 1
max: 65535
readOnlyRootFilesystem: false
https://kubernetes.io/docs/concepts/policy/pod-security-policy/

View Slide

Lessons learned
31
• Use the immutable deployment pattern
• Only system pods need access to the api server
• Add PodSecurityPolicy and limits of the resource usages
• Limit the network access with a NetworkPolicy
• Use multiple tiller with limit RBAC roles and track your k8s cluster users
• Check your container images with CVE Checkers (Clair, NeuVector, AquaSec, … )
• Add more Know How to your teams: Teach your Ops and Dev colleagues
• Share your Knowledge
• Think about Chaos Engineering: Replace your system components every time
• Automate all and never stop this process

View Slide

32
Container System Blueprint: beehive
Orchestration
Provisioning
Security
Conﬁg
Metrics
Alarming
Logging
Traceing
Artefact
Registry
Vault
Auth
Source Code
CI/CD Pipeline
Storage
Network
API Gateway Routing
Loadbalancer, DNS and Firewall
On Premise Cloud
Operating Systems
Backends
VM machines
Databases
Bare metal
Container machines
Container machines

View Slide

CSSC: Container System Supply Chain
33
CI/CD
Git
DevOps
commit trigger
commit
Jobs
ci trigger
build
test
tag
push
helm
install
CI
Registry
Runner
K8s
cluster
push image
deploy
manifests
pull jobs
pull
images
K8s
resources
Monitor
Tiller
metrics, logs, tracing
status
Company
Registry
Hubs &
Mirrors
Package
Repos
pull image
use
packages
execute

View Slide

34

View Slide

35
We hiring :-)
 
https://bit.ly/2K8DtRu  
[email protected] 
@bee42solutions

View Slide

bee42 Trainings
36
25.06.2018 in Berlin
15.10.2018 in München
19.11.2018 in Essen
06.07.2018 in Essen
03.09.2018 in Essen
17.09.2018 in Essen
22.10.2018 in Essen
19.09.2018 in Essen
22.11.2018 in Essen
Rabatt code
meetup-berlin-20
Date of Expiry 31/07/2018
03.08.2018 in Essen
24.10.2018 in Essen

View Slide

K8s poster
pre registration started

https://bit.ly/2JN0hWI
37
We

View Slide

Cloud Native System Architect
&
bee42 founder
Peter Roßbach

@PRossbach

[email protected]

https://bee42.com

https://devops-gathering.io

38
Save the date…
#DOG19 11.-13.3.2019 at Bochum

View Slide

References
39
[Sprig library](https://godoc.org/github.com/Masterminds/sprig
[Deis Workflow](https://github.com/deis/workflow/tree/master/charts/workflow))
[OpenStack chart](https://github.com/sapcc/openstack-helm)
[helm.sh](https://helm.sh)
[bitnami charts](https://github.com/bitnami/charts)
[kubernetes charts](https://github.com/kubernetes/charts)

View Slide

Helm: Build fashionable container systems with Kubernetes

Helm: Build fashionable container systems with Kubernetes

Peter Rossbach

More Decks by Peter Rossbach

Other Decks in Programming

Featured

Transcript