Tweet
Tweet

Slide 1

Slide 1 text

Kubernetes - the abstract cloud Jörg Müller - @joergm 11.6.2018
 München / Microservice Summit

Slide 2

Slide 2 text

Jörg Müller
 Principal Consultant innoQ Deutschland GmbH [email protected] @joergm - architecture, development, devOps - focus on platform & infrastructure

Slide 3

Slide 3 text

What to expect? • Overview, Ways to get Kubernetes and basic concepts • Core abstractions • Internal Architecture • Deploying complex applications • Production readiness

Slide 4

Slide 4 text

Timeslots • 9:30 - 10:30 Slot 1 • 10:30 - 11:00 Coffee break • 11:00 - 12:30 Slot 2 • 12:30 - 13:30 Lunch • 13:30 - 15:00 Slot 3 • 15:00 - 15:30 Coffee break • 15:30 - 17:00 Slot 4

Slide 5

Slide 5 text

Prerequisites & rules • Kubernetes know-how not necessary, but it doesn’t hurt • Basic knowledge about Docker is assumed • Demos can be followed but don’t have to • github.com/JoergM/kubernetes_workshop_demos • Please ask questions!

Slide 6

Slide 6 text

Kubernetes Overview

Slide 7

Slide 7 text

Docker Recap

Slide 8

Slide 8 text

Docker container at runtime • Isolated process • Separate file system • Own network address and port space

Slide 9

Slide 9 text

Docker container - advantages • Better isolation than package management on same machine • e.g. multiple versions of core libraries • not necessary to coordinate available ports • Faster startup than virtual machine images • Better resource usage compared to VMs

Slide 10

Slide 10 text

Docker images • Standardized format • Container hierarchies and difference file system • Registries • Unique name format 
 (Registry/username/imagename:version) • Simple Text-Format to create new images (Dockerfile)

Slide 11

Slide 11 text

Docker images - advantages • Deployment format independent of implementation technology • Same deliverable in all stages (Development, CI, Tests, Production) • Container hierarchies allow simpler patch management • Definition simpler than most package manager definitions

Slide 12

Slide 12 text

What is Kubernetes?

Slide 13

Slide 13 text

Kubernetes — adds to Docker • Handling of multiple servers • Scheduling of containers • Networking • Failure handling • Service Discovery features • Many other useful abstractions for container interactions

Slide 14

Slide 14 text

Kubernetes - executive summary • Kubernetes (K8s) is an open-source system for automating deployment, scaling, and management of containerized applications • Marketing claim: • Planet Scale • Never Outgrow • Run Anywhere

Slide 15

Slide 15 text

Kubernetes — brief history • Designed by Google, later donated to Cloud Native Computing Foundation • Heavily influenced by Google's internal Borg system • Code name: Project Seven • Initial release: 7 June 2014 / 15 December 2015 (first stable version)

Slide 16

Slide 16 text

Why abstract cloud?

Slide 17

Slide 17 text

Why do we need one? • Working on local machines • Prevent Vendor lock in • Less specific Know How necessary • Easier to move • Common way to automate complex setups • For inhouse applications • Also for software vendors

Slide 18

Slide 18 text

Kubernetes • You define resources needed not machines or implementations • Kubernetes manages resources • Has a large base of runtime environments Application Resource abstraction Resource management Runtime environment

Slide 19

Slide 19 text

We tried that before … • Virtual machines • Configuration management (Puppet, Ansible, Chef) • Terraform, CloudFormation • PaaS • …

Slide 20

Slide 20 text

Standing on Shoulders • Container abstractions • Googles experiences running Borg • Focus on immutable infrastructure

Slide 21

Slide 21 text

K8s for microservices

Slide 22

Slide 22 text

Challenges • Deployment • Configuration • Service Discovery • Load Balancing • Routing • Resilience

Slide 23

Slide 23 text

Kubernetes • Provides solutions for those challenges • Is available everywhere • Becomes more and more widespread • So developers know how to solve those challenges • Operations accepts and knows the solution • Microservices infrastructure looses a lot of its horror

Slide 24

Slide 24 text

Ways to get Kubernetes

Slide 25

Slide 25 text

Local installation Installing a simple Kubernetes on your notebook. • Minikube • Docker native

Slide 26

Slide 26 text

Online Tryout Try Kubernetes without installing anything. • https://www.katacoda.com/courses/kubernetes/ playground • https://labs.play-with-k8s.com/

Slide 27

Slide 27 text

By Cloud providers Managed Kubernetes is now offered by all major cloud providers. • Google Kubernetes Engine (GKE) • Azure AKS • IBM Cloud Kubernetes Services • Amazon EKS (GA just started in us-east and us-west) • Digital Ocean Kubernetes (coming soon)

Slide 28

Slide 28 text

Specialised Kubernetes providers Offering managed Kubernetes on different plattforms. Often including Support and On-Premise install. • GiantSwarm • Rancher • Tectonic • Kontena Pharos • …

Slide 29

Slide 29 text

PaaS Solutions Plattform as a Service built on Kubernetes or offering Kubernetes services. • RedHat OpenShift • CloudFoundry Container runtime

Slide 30

Slide 30 text

Self install Finally a lot of options to install Kubernetes yourself on Cloud Providers or On-Premise. • kubeadm • KOPS • https://github.com/kelseyhightower/kubernetes-the- hard-way

Slide 31

Slide 31 text

Basic concepts

Slide 32

Slide 32 text

Cluster overview Cluster Node Node Node Master Master Master Node …

Slide 33

Slide 33 text

Master Components Cluster Node Node Node Master Master Master Node … Master api-server etcd scheduler controller-manager

Slide 34

Slide 34 text

Node Components Cluster Node Node Node Master Master Master Node … Node container-runtime kubelet kube-proxy network

Slide 35

Slide 35 text

API Objects • Persistent (in etcd) • represent the desired state of the cluster • Have • Spec • Status

Slide 36

Slide 36 text

Cluster Master API Objects interaction $ kubectl run … api-server etcd Node kubelet docker scheduler 1 2 3 4

Slide 37

Slide 37 text

Core abstractions

Slide 38

Slide 38 text

github.com/JoergM/ kubernetes_workshop_demos

Slide 39

Slide 39 text

Pods

Slide 40

Slide 40 text

Pod • Deployment-Unit in Kubernetes • A pod consists of one or more containers • Containers in a pod share network • Containers in a pod can share volumes • Each pod receives its own cluster-wide and cluster internal IP address

Slide 41

Slide 41 text

Pod with a single container Pod C

Slide 42

Slide 42 text

Sharing network Pod C C localhost

Slide 43

Slide 43 text

Sharing volumes Pod C C

Slide 44

Slide 44 text

Pods with init containers Pod C 1

Slide 45

Slide 45 text

Complex pod patterns Pod init ssl auth app

Slide 46

Slide 46 text

github.com/JoergM/kubernetes_workshop_demos/pods DEMO

Slide 47

Slide 47 text

Deployments

Slide 48

Slide 48 text

Deployment • Declares a state of Pods • Is used for scaling up N instances of the same pod • Is used to deploy old or new revisions of a pod

Slide 49

Slide 49 text

Deployment Deployment Replicas: 4 ... Pod Pod Pod Pod

Slide 50

Slide 50 text

Deployment Pod Pod Pod Deployment Replicas: 4 ... Pod

Slide 51

Slide 51 text

Deployment Deployment Replicas: 4 ... Pod Pod Pod Pod replica set handles self healing Pod

Slide 52

Slide 52 text

Deploying new versions • Rolling update • Recreate • Blue Green • Canary

Slide 53

Slide 53 text

Rolling Update • Default variant • No service downtime • Both versions get traffic at the same time • consider setting maxUnavailable/ maxSurge

Slide 54

Slide 54 text

Recreate • Activated setting type • Involves downtime • no version conflicts

Slide 55

Slide 55 text

Prelude Service • IP and DNS for multiple Pods • Loadbalancing • Uses Labels to find Pods Service Pod Pod order-service 172.30.0.1:80 Label A Label A

Slide 56

Slide 56 text

Blue / Green • No version conflicts • No downtime • High resource usage • Involves custom handling by Switching service labels Service

Slide 57

Slide 57 text

Canary • Slowly testing new versions • No Downtime • Both versions get traffic at the same time • Some custom handling of multiple deployments

Slide 58

Slide 58 text

github.com/JoergM/kubernetes_workshop_demos/deployments DEMO

Slide 59

Slide 59 text

Services

Slide 60

Slide 60 text

Service Overview Service Pod Pod order-service 172.30.0.1:80 Label A Label A

Slide 61

Slide 61 text

Service • Is an abstraction which defines a logical set of pods and a policy by which to access them • Usually represents a micro-service • Different types of services possible • Discovery inside Cluster via DNS • It’s not a physical LoadBalancer (more later)

Slide 62

Slide 62 text

Service type ClusterIP Cluster ClusterIP 172.30.0.1 Node 1 Node 2 Node 3 Consumer Pod Pod

Slide 63

Slide 63 text

Service type NodePort Cluster Node 1 Node 2 Node 3 Pod Pod :80 :80 :80 Consumer

Slide 64

Slide 64 text

Service type LoadBalancer Cluster Node 1 Node 2 Pod Pod Consumer LoadBalancer

Slide 65

Slide 65 text

Service type External… Cluster internal service name/IP Node 1 Consumer External Service

Slide 66

Slide 66 text

github.com/JoergM/kubernetes_workshop_demos/services DEMO

Slide 67

Slide 67 text

Configuration

Slide 68

Slide 68 text

Config Maps • Provide Pods with configuration data • from • literal values • files • directories

Slide 69

Slide 69 text

Config Maps • In Pods as • environment variables • files • directories

Slide 70

Slide 70 text

github.com/JoergM/kubernetes_workshop_demos/configuration DEMO

Slide 71

Slide 71 text

Ingress

Slide 72

Slide 72 text

Ingress Cluster Pod Ingress-Controller :80 Pod Service A Service B Pod Pod default default /order /items /foo

Slide 73

Slide 73 text

Ingress Controller • Creates a LoadBalancer service that points to a pod, which runs a reverse proxy (nginx, haproxy, Apache, traefik) • Uses IngressRules to describe which DNS and/or path should point to which service • Always needs a default service

Slide 74

Slide 74 text

Ingress controller implementations • nginx • traefik • voyager • GCE ingress • Kong • …

Slide 75

Slide 75 text

github.com/JoergM/kubernetes_workshop_demos/ingress DEMO

Slide 76

Slide 76 text

Jobs

Slide 77

Slide 77 text

Jobs • Running pods until completion • Like deployment for long- running pods • supports parallelism Job Pod Pod …

Slide 78

Slide 78 text

CronJob • Regularly starting jobs • Follows typical Cron patterns: • 0 12 * * 1-5 • (weekdays at noon) Job Pod Pod CronJob

Slide 79

Slide 79 text

github.com/JoergM/kubernetes_workshop_demos/jobs DEMO

Slide 80

Slide 80 text

Persistence

Slide 81

Slide 81 text

Persistence Overview Pod Volume Persistent Volume Claim Persistent Volume Real storage satisfied by references has

Slide 82

Slide 82 text

Persistent Volume Claims • User requesting Storage • Used in pod as volume • Survives pod recreation • Certain Size (e.g. 5Gi) • Certain class (e.g. SSD) • Will be matched to persistent volumes

Slide 83

Slide 83 text

Persistent Volumes • Defines a real volume of a certain size (e.g. 5Gi) • Can be created upfront • Lots of implementations: • GCEPersistenceDisk • HostPath • AWS EBS • NFS

Slide 84

Slide 84 text

Dynamic provisioning • Creating Volumes based on Claims • Requires dynamic way of creating volumes and mounting to nodes • e.g. AWS EBS, GCE Persistent Disk • Custom provisioners for can be created too

Slide 85

Slide 85 text

github.com/JoergM/kubernetes_workshop_demos/ persistent_volumes DEMO

Slide 86

Slide 86 text

Stateful Sets

Slide 87

Slide 87 text

Overview Stateful Set Pod Template Volume Claim Template Pod Volume-1 Name-1 Pod Volume-2 Name-2 Replicas=2

Slide 88

Slide 88 text

Stateful Sets • Like Deployment but with other guarantees • Each Replica has always the same name (and DNS) • Replica and Volume Claim always come together • Most parameters not changeable after creation

Slide 89

Slide 89 text

Stateful Sets - usages • All kind of software that builds a cluster, but needs certain guarantees • e.g. Databases with fixed Follower-Leader specification • MongoDB • Zookeeper • Postgresql • Do not use if not necessary!

Slide 90

Slide 90 text

github.com/JoergM/kubernetes_workshop_demos/stateful_sets DEMO

Slide 91

Slide 91 text

Namespaces

Slide 92

Slide 92 text

Kubernetes Cluster My-namespace My-namespace Namespaces overview default kube-system Pods Services Deployments Jobs Pods Deployments … Deployment Pod Services Jobs … Services …

Slide 93

Slide 93 text

Namespaces • Scope for names of objects • Hook for service accounts and network policies • Isolation level depends on your installation • Not all objects are in namespaces (esp. low level like nodes or persistent volumes)

Slide 94

Slide 94 text

github.com/JoergM/kubernetes_workshop_demos/namespaces DEMO

Slide 95

Slide 95 text

Internal architecture

Slide 96

Slide 96 text

Master Components Cluster Node Node Node Master Master Master Node … Master api-server etcd scheduler controller-manager

Slide 97

Slide 97 text

API Server • Entry point for all interactions • Stores desired state into etcd • available from outside and inside cluster Master api-server etcd scheduler controller-manager

Slide 98

Slide 98 text

etcd • consensus based distributed key value database • interaction only via api- server Master api-server etcd scheduler controller-manager

Slide 99

Slide 99 text

Scheduler • Watches newly created pods and assigns them to nodes • Lots of criterias • resource requirements • load • specific constraints Master api-server etcd scheduler controller-manager

Slide 100

Slide 100 text

Controller-Manager • Manages / runs the controllers responsible for certain tasks in Kubernetes Master api-server etcd scheduler controller-manager

Slide 101

Slide 101 text

Kubernetes API — Controller • Watch the Api Server for changes • perform Operations on changes • Creation/ Deletion or Update on other API objects • Running a reconciliation loop

Slide 102

Slide 102 text

Controller examples • Node Controller • Replication Controller • Cloud Volume Controller • DNS Controller • (your controller)

Slide 103

Slide 103 text

Node Components Cluster Node Node Node Master Master Master Node … Node container-runtime kubelet kube-proxy network

Slide 104

Slide 104 text

Container runtime • Component to run containers on nodes • Usually Docker • Can be other implementation (e.g. rkt) Node container-runtime kubelet kube-proxy network

Slide 105

Slide 105 text

Kubelet • Reads PodSpecs from the API • Uses container-runtime to run Pods according to Spec Node container-runtime kubelet kube-proxy network

Slide 106

Slide 106 text

Cluster Master Running a Pod $ kubectl run … api-server etcd Node kubelet docker scheduler 1 2 3 4

Slide 107

Slide 107 text

Kube-Proxy • responsible for service abstraction • Classic proxy in usermode (old) • New mode uses iptables to implement routing Node container-runtime kubelet kube-proxy network

Slide 108

Slide 108 text

Kube-Proxy & Services • Service has virtual address • Kube-proxy updates IP-Tables on Node • Any packet with the virtual address/port combination will be changed to a node-ip and port combination • Overlay network will then do the rest • see IPs (172 - virtual) (10.1.x nodes)

Slide 109

Slide 109 text

Kube-Proxy iptables mode Cluster Consumer ClusterIP 172.30.0.1 iptables kube-proxy Pod1 10.1.2.1 Pod2 10.1.3.5 configures

Slide 110

Slide 110 text

Network • Making sure that Pods can connect across nodes • No NAT in Cluster • different implementations of the Container Network Interface (CNI) Node container-runtime kubelet kube-proxy network

Slide 111

Slide 111 text

Network basic example Cluster Node-1 Node-2 Pod A eth0: 10.1.1.1 Pod B eth0: 10.1.1.2 host network vethxxx vethxxx Bridge 10.1.1.0/24 Impl Pod C eth0: 10.1.2.1 Pod D eth0: 10.1.2.2 vethxxx vethxxx Bridge 10.1.2.0/24 Impl

Slide 112

Slide 112 text

Deploying complex applications

Slide 113

Slide 113 text

Helm

Slide 114

Slide 114 text

Helm • Package management for Kubernetes: update, rollback, create, version, share, and publish applications • Ready to use Kubernetes-applications (but always check the sources — like … for real, do it) • Handy for deployment*: persistent history and easy rollback for free • https://helm.sh/

Slide 115

Slide 115 text

Helm cont. • Part of Cloud Native Computing Foundation • Includes the possibility to template Kubernetes config files (e.g. for handling different clusters with the same configs)

Slide 116

Slide 116 text

github.com/JoergM/kubernetes_workshop_demos/helm DEMO

Slide 117

Slide 117 text

Operators

Slide 118

Slide 118 text

Operators • Idea to automate the knowledge of a human Operator • Not only install automated, but operate automated • The Operator itself run on Kubernetes too • Uses Kubernetes API to do his job • https://coreos.com/operators/

Slide 119

Slide 119 text

Advanced Features • Create Backups • Autoscale • Autoupdate installed Software

Slide 120

Slide 120 text

Operator examples • etcd • Vault • Prometheus • Elasticsearch • Kafka

Slide 121

Slide 121 text

Operator Framework • Published on KubeCon 2018 • Operator SDK to create your own Operators • Operator Lifecycle Manager - managing operators in a cluster • Operator Metering - gathering data on operators

Slide 122

Slide 122 text

Service meshes

Slide 123

Slide 123 text

Service Meshes • Providing common infrastructure for microservices • Features • Circuit Breaking • TLS • Authentication • Tracing • …

Slide 124

Slide 124 text

Service Mesh basics Cluster Pod1 Service Sidecar Pod2 Service Sidecar Mesh Control Plane

Slide 125

Slide 125 text

Projects to look at • linkerd (https://linkerd.io/) • Envoy Proxy (https://www.envoyproxy.io/) • Istio (https://istio.io/) • Conduit (https://conduit.io/)

Slide 126

Slide 126 text

Production readiness

Slide 127

Slide 127 text

Monitoring

Slide 128

Slide 128 text

Monitoring Overview Cluster Node 1 Node 2 Node 3 Pods Pods Pods Pods Pods Pods Kubelet cAdvisor Kubelet cAdvisor Kubelet cAdvisor Heapster

Slide 129

Slide 129 text

cAdvisor & Heapster • cAdvisor is integrated into kubelet • collects performance data of containers on node • and on the node itself • Heapster is running as a pod inside the cluster • collects data from all nodes • makes them available for other tools • Command line, dashboard, Influx, Prometheus …

Slide 130

Slide 130 text

Command line $ kubectl top node NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% minikube 226m 11% 880Mi 22% $ kubectl top pods --all-namespaces NAMESPACE NAME CPU(cores) MEMORY(bytes) default nginx-5ccd64769-gn9bn 0m 1Mi kube-system influxdb-grafana-rl265 2m 84Mi kube-system kube-addon-manager-minikube 91m 50Mi kube-system kube-dns-54cccfbdf8-2r526 1m 23Mi kube-system kubernetes-dashboard-77d8b98585-md5 3m 13Mi …

Slide 131

Slide 131 text

Dashboard

Slide 132

Slide 132 text

Grafana

Slide 133

Slide 133 text

github.com/JoergM/kubernetes_workshop_demos/monitoring DEMO

Slide 134

Slide 134 text

Managing load

Slide 135

Slide 135 text

Ressource requests • Requirements stated at container level • Primarly CPU and Memory • Scheduler uses values to find best Node apiVersion: v1 kind: Pod metadata: name: example-pod spec: containers: - image: alpine name: foo resources: requests: cpu: 100m memory: 25Mi

Slide 136

Slide 136 text

Ressource limits • Limits limit the resources available to a container • If CPU exceeds limit it will be throttled • If memory exceeds limit pod will be killed apiVersion: v1 kind: Pod metadata: name: example-pod spec: containers: - image: alpine name: foo resources: limits: cpu: 100m memory: 25Mi

Slide 137

Slide 137 text

Limits and Requests • Understand how limits and requests work • Set them accordingly • Be aware of resource visibility to container processes (esp. with Java applications) • Processes see node memory and cores

Slide 138

Slide 138 text

Two levels of autoscaling • Scaling Nodes • Depending on underlying runtime enironment • Autoscaling Groups on AWS as usual • Scaling Pods • Cluster internal • Of course limited to available nodes

Slide 139

Slide 139 text

Scaling Overview Cluster Pods Pods Kubelet cAdvisor Heapster Horizontal PodAutoscaler Deployment Replicaset

Slide 140

Slide 140 text

Scaling Definition • HorizontalPodAutoscaler API Object • currently supports CPU and custom metrics apiVersion: autoscaling/v2beta1 kind: HorizontalPodAutoscaler metadata: … spec: scaleTargetRef: apiVersion: apps/v1 kind: Deployment name: … minReplicas: 1 maxReplicas: 10 metrics: - type: Resource resource: name: cpu targetAverageUtilization: 50

Slide 141

Slide 141 text

Security

Slide 142

Slide 142 text

Disclaimer • This is only scratching at the surface • To truly secure your cluster learn about the concepts, try them yourself, let somebody else look at it

Slide 143

Slide 143 text

Securing the API • Who is allowed to do what using the API • Who is „Who“? • How to identify? • How to assign rights?

Slide 144

Slide 144 text

Users in Kubernetes • Human users • Accessing the API using e.g. kubectl • several mechanisms to identify (X.509, tokens …) • managed externally • Pods accessing the API • Pods are associated to Service Accounts • Namespace default or in Spec

Slide 145

Slide 145 text

Role Based Access Control Cluster User Service Account (Cluster)Role Binding (Cluster)Role Allowed Resources Other Resources

Slide 146

Slide 146 text

Pod Security Policies • What is a Pod allowed to do? • User inside Pod? Is Root allowed? • What Kernel capabilities are allowed? • Read only filesystem • … • Assigned using ClusterRoles

Slide 147

Slide 147 text

Network Policies • Availability depending on installed network layer • By default every pod can be accessed (need to change) • can isolate single pods but also namespaces apiVersion: networking.k8s.io/v1 kind: NetworkPolicy metadata: name: postgres-netpolicy spec: podSelector: matchLabels: app: database ingress: - from: - podSelector: matchLabels: app: webserver ports: - port: 5432

Slide 148

Slide 148 text

Jörg Müller
 Principal Consultant innoQ Deutschland GmbH [email protected] @joergm - architecture, development, devOps - focus on platform & infrastructure

Slide 149

Slide 149 text

www.innoq.com OFFICES Monheim Berlin Offenbach Munich Zurich FACTS ~125 employees Privately owned Vendor-independent SERVICES Strategy & technology consulting Digital business models Software architecture & development Digital platforms & infrastructures Knowledge transfer, coaching & trainings CLIENTS Finance Telecommunications Logistics E-commerce Fortune 500 SMBs Startups