Slide 1

Slide 1 text

Sandeep Dinesh Developer Advocate @sandeepdinesh github.com/thesandlord Kubernetes Best Practices

Slide 2

Slide 2 text

Google Cloud Platform 2 Kubernetes is really flexible

Slide 3

Slide 3 text

Google Cloud Platform 3 But you might yourself in the

Slide 4

Slide 4 text

Building Containers

Slide 5

Slide 5 text

Google Cloud Platform 5 Don’t trust arbitrary base images!

Slide 6

Slide 6 text

6 Google Cloud Platform Static Analysis of Containers https://github.com/banyanops/collector https://github.com/coreos/clair

Slide 7

Slide 7 text

Google Cloud Platform 7 Use small base images

Slide 8

Slide 8 text

8 Google Cloud Platform Overhead Node.js App Your App → 5MB Your App’s Dependencies → 95MB Total App Size → 100MB Docker Base Images: node:8 → 667MB node:8-wheezy → 521MB node:8-slim → 225MB node:8-alpine → 63.7MB scratch → ~50MB

Slide 9

Slide 9 text

9 Google Cloud Platform Overhead Node.js App Your App → 5MB Your App’s Dependencies → 95MB Total App Size → 100MB Docker Base Images: node:8 → 667MB node:8-wheezy → 521MB node:8-slim → 225MB node:8-alpine → 63.7MB scratch → ~50MB ← 6.6x App Size!!

Slide 10

Slide 10 text

10 Google Cloud Platform Overhead Node.js App Your App → 5MB Your App’s Dependencies → 95MB Total App Size → 100MB Docker Base Images: node:8 → 667MB node:8-wheezy → 521MB node:8-slim → 225MB node:8-alpine → 63.7MB scratch → ~50MB ← 6.6x App Size!! ← 13.3x “min” overhead!!

Slide 11

Slide 11 text

11 Google Cloud Platform Overhead Node.js App Your App → 5MB Your App’s Dependencies → 95MB Total App Size → 100MB Docker Base Images: node:8 → 667MB node:8-wheezy → 521MB node:8-slim → 225MB node:8-alpine → 63.7MB scratch → ~50MB ← 6.6x App Size!! ← 13.3x “min” overhead!! Pros: Builds are faster Need less storage Cold starts (image pull) are faster Potentially less attack surface Cons: Less tooling inside container “Non-standard” environment

Slide 12

Slide 12 text

Google Cloud Platform 12 Use the “builder pattern”

Slide 13

Slide 13 text

13 Google Cloud Platform Code Build Container Compiler Dev Deps Unit Tests etc... Build Artifact(s) Runtime Container Runtime Env Debug/Monitor Tooling Binaries Static Files Bundles Transpiled Code

Slide 14

Slide 14 text

Google Cloud Platform 14 Docker bringing native support for multi-stage builds in Docker CE 17.05

Slide 15

Slide 15 text

Container Internals

Slide 16

Slide 16 text

Google Cloud Platform 16 Use a non-root user inside the container

Slide 17

Slide 17 text

17 Google Cloud Platform FROM node:alpine RUN apk update && apk add imagemagick RUN groupadd -r nodejs RUN useradd -m -r -g nodejs nodejs USER nodejs ADD package.json package.json RUN npm install ADD index.js index.js CMD npm start Example Dockerfile

Slide 18

Slide 18 text

18 Google Cloud Platform Enforce it! apiVersion: v1 kind: Pod metadata: name: hello-world spec: containers: # specification of the pod’s containers # ... securityContext: runAsNonRoot: true

Slide 19

Slide 19 text

Google Cloud Platform 19 Make the filesystem read-only

Slide 20

Slide 20 text

20 Google Cloud Platform apiVersion: v1 kind: Pod metadata: name: hello-world spec: containers: # specification of the pod’s containers # ... securityContext: runAsNonRoot: true readOnlyRootFilesystem: true Enforce it!

Slide 21

Slide 21 text

Google Cloud Platform 21 One process per container

Slide 22

Slide 22 text

Google Cloud Platform 22 Don’t restart on failure. Crash cleanly instead.

Slide 23

Slide 23 text

Google Cloud Platform 23 Log to stdout and stderr

Slide 24

Slide 24 text

Google Cloud Platform 24 Add “dumb-init” to prevent zombie processes

Slide 25

Slide 25 text

25 Google Cloud Platform FROM node:alpine RUN apk update && apk add imagemagick RUN groupadd -r nodejs RUN useradd -m -r -g nodejs nodejs USER nodejs ADD https://github.com/Yelp/dumb-init/releases/download/v1.2.0/dumb-init_1.2.0_amd64 \ /usr/local/bin/dumb-init RUN chmod +x /usr/local/bin/dumb-init ENTRYPOINT ["/usr/bin/dumb-init", "--"] ADD package.json package.json RUN npm install ADD index.js index.js CMD npm start Example Dockerfile

Slide 26

Slide 26 text

Google Cloud Platform 26 Good News: No need to do this in K8s 1.7

Slide 27

Slide 27 text

Deployments

Slide 28

Slide 28 text

Google Cloud Platform 28 Use the “record” option for easier rollbacks

Slide 29

Slide 29 text

29 Google Cloud Platform $ kubectl apply -f deployment.yaml --record … $ kubectl rollout history deployments my-deployment deployments "ghost-recorded" REVISION CHANGE-CAUSE 1 kubectl apply -f deployment.yaml --record 2 kubectl edit deployments my-deployment 3 kubectl set image deployment/my-deplyoment my-container=app:2.0

Slide 30

Slide 30 text

Google Cloud Platform 30 Use plenty of descriptive labels

Slide 31

Slide 31 text

31 Google Cloud Platform apiVersion: extensions/v1beta1 kind: Deployment metadata: name: web spec: replicas: 12 template: metadata: labels: name: web color: blue experimental: 'true' Labels

Slide 32

Slide 32 text

32 Google Cloud Platform App: Nifty Phase: Dev Role: FE App: Nifty Phase: Test Role: FE App: Nifty Phase: Dev Role: BE App: Nifty Phase: Test Role: BE Labels

Slide 33

Slide 33 text

33 Google Cloud Platform App: Nifty Phase: Dev Role: FE App: Nifty Phase: Test Role: FE App: Nifty Phase: Dev Role: BE App: Nifty Phase: Test Role: BE Labels App = Nifty

Slide 34

Slide 34 text

34 Google Cloud Platform App: Nifty Phase: Dev Role: FE App: Nifty Phase: Test Role: FE App: Nifty Phase: Dev Role: BE App: Nifty Phase: Test Role: BE Labels Role = BE

Slide 35

Slide 35 text

35 Google Cloud Platform App: Nifty Phase: Dev Role: FE App: Nifty Phase: Test Role: FE App: Nifty Phase: Dev Role: BE App: Nifty Phase: Test Role: BE Labels Phase = Dev

Slide 36

Slide 36 text

36 Google Cloud Platform App: Nifty Phase: Dev Role: FE App: Nifty Phase: Test Role: FE App: Nifty Phase: Dev Role: BE App: Nifty Phase: Test Role: BE Labels Phase = Dev Role = FE

Slide 37

Slide 37 text

Google Cloud Platform 37 Use sidecar containers for proxies, watchers, etc

Slide 38

Slide 38 text

38 Google Cloud Platform Examples App Database localhost App App Proxy Proxy Proxy secure connection

Slide 39

Slide 39 text

39 Google Cloud Platform Auth, Rate Limiting, etc. Examples App Incoming Requests localhost App App Proxy Proxy Proxy

Slide 40

Slide 40 text

Google Cloud Platform 40 Don’t use sidecars for bootstrapping!

Slide 41

Slide 41 text

Google Cloud Platform 41 Use init containers instead!

Slide 42

Slide 42 text

42 Google Cloud Platform apiVersion: v1 kind: Pod metadata: name: awesomeapp-pod labels: app: awesomeapp annotations: pod.beta.kubernetes.io/init-containers : '[ { "name": "init-myapp", "image": "busybox", "command": ["sh", "-c", "until nslookup myapp; do echo waiting for myapp; sleep 2; done;"] }, { "name": "init-mydb", "image": "busybox", "command": ["sh", "-c", "until nslookup mydb; do echo waiting for mydb; sleep 2; done;"] } ]' spec: containers: - name: awesomeapp-container image: busybox command: ['sh', '-c', 'echo The app is running! && sleep 3600' ]

Slide 43

Slide 43 text

Google Cloud Platform 43 Don’t use :latest or no tag

Slide 44

Slide 44 text

Google Cloud Platform 44 Readiness and Liveness probes are your friend

Slide 45

Slide 45 text

45 Google Cloud Platform Readiness → Is the app ready to start serving traffic? ● Won’t be added to a service endpoint until it passes ● Required for a “production app” in my opinion Liveness → Is the app still running? ● Default is “process is running” ● Possible that the process can be running but not working correctly ● Good to define, might not be 100% necessary These can sometimes be the same endpoint, but not always Health Checks

Slide 46

Slide 46 text

Services

Slide 47

Slide 47 text

Google Cloud Platform 47 Don’t always use type: LoadBalancer

Slide 48

Slide 48 text

Google Cloud Platform 48 Ingress is great

Slide 49

Slide 49 text

49 Google Cloud Platform Ingress Service 1 Service 2 Service 3 websocket.mydomain.com mydomain.com /foo /bar

Slide 50

Slide 50 text

Google Cloud Platform 50 type: NodePort can be “good enough”

Slide 51

Slide 51 text

Google Cloud Platform 51 Use Static IPs. They are free*!

Slide 52

Slide 52 text

52 Google Cloud Platform apiVersion: v1 kind: Service metadata: name: myservice spec: type: LoadBalancer loadBalancerIP: QQQ.ZZZ.YYY.XXX ports: - port: 80 targetPort: 3000 protocol: TCP selector: name: myapp $ gcloud compute addresses create ingress --global … $ gcloud compute addresses create myservice --region=us-west1 Created … address: QQQ.ZZZ.YYY.XXX … $ apiVersion: extensions/v1beta1 kind: Ingress metadata: name: myingress annotations: kubernetes.io/ingress.global-static-ip-name: "ingress" spec: backend: serviceName: myservice servicePort: 80

Slide 53

Slide 53 text

Google Cloud Platform 53 Map external services to internal ones

Slide 54

Slide 54 text

54 Google Cloud Platform External Services kind: Service apiVersion: v1 metadata: name: mydatabase namespace: prod spec: type: ExternalName externalName : my.database.example.com ports: - port: 12345 kind: Service apiVersion: v1 metadata: name: mydatabase spec: ports: - protocol: TCP port: 80 targetPort: 12345 kind: Endpoints apiVersion: v1 metadata: name: mydatabase subsets: - addresses: - ip: 10.128.0.2 ports: - port: 12345 Hosted Database Database outside cluster but inside network

Slide 55

Slide 55 text

Application Architecture

Slide 56

Slide 56 text

Google Cloud Platform 56 Use Helm Charts

Slide 57

Slide 57 text

Google Cloud Platform 57 ALL downstream dependencies are unreliable

Slide 58

Slide 58 text

Google Cloud Platform 58 Make sure your microservices aren’t too micro

Slide 59

Slide 59 text

Google Cloud Platform 59 Use a “Service Mesh”

Slide 60

Slide 60 text

60 Google Cloud Platform https://github.com/istio/istio https://github.com/linkerd/linkerd

Slide 61

Slide 61 text

Google Cloud Platform 61 Use a PaaS?

Slide 62

Slide 62 text

62 Google Cloud Platform

Slide 63

Slide 63 text

Cluster Management

Slide 64

Slide 64 text

Google Cloud Platform 64 Use Google Container Engine

Slide 65

Slide 65 text

Google Cloud Platform 65 Resources, Anti-Affinity, and Scheduling

Slide 66

Slide 66 text

66 Google Cloud Platform Node Affinity hostname zone region instance-type os arch custom!

Slide 67

Slide 67 text

67 Google Cloud Platform Node Taints / Tolerations special hardware dedicated hosts etc

Slide 68

Slide 68 text

68 Google Cloud Platform Pod Affinity / Anti-Affinity hostname zone region

Slide 69

Slide 69 text

Google Cloud Platform 69 Use Namespaces to split up your cluster

Slide 70

Slide 70 text

Google Cloud Platform 70 Role Based Access Control

Slide 71

Slide 71 text

Google Cloud Platform 71 Unleash the Chaos Monkey

Slide 72

Slide 72 text

72 Google Cloud Platform More Resources ● http://blog.kubernetes.io/2016/08/security-best-practices-kubernetes-deployment.html ● https://github.com/gravitational/workshop/blob/master/k8sprod.md ● https://nodesource.com/blog/8-protips-to-start-killing-it-when-dockerizing-node-js/ ● https://www.ianlewis.org/en/using-kubernetes-health-checks ● https://www.linux.com/learn/rolling-updates-and-rollbacks-using-kubernetes-deployments ● https://kubernetes.io/docs/api-reference/v1.6/

Slide 73

Slide 73 text

Questions? What best practices do you have?