No Big Deal Deploys to Kubernetes

Transcript

1. KUBERNETES NO BIG DEAL DEPLOYS TO
2. WHAT ARE WE TALKING ABOUT? ▸ The deployment landscape to Kubernetes ▸ How (and why) FireHydrant.io deploys using GitOps
3. WHO AM I? ROBERT ROSS BUT PEOPLE CALL ME BOBBY TABLES @BOBBYTABLES
4. NAMELY DIGITALOCEAN THUNDERBOLT LABS 5 OTHERS THAT WE’LL SKIP PREVIOUSLY WORKED AT
5. I am the founder and CEO of An incident response tool
6. RELIABLE DEPLOYS THE IMPORTANCE OF
7. BUT WHY PREVENTING INCIDENTS ▸ As an incident response tool, FireHydrant needs to make sure our deploys are consistent ▸ 80% of the time an incident opens, it’s due to a deploy of bad code (based on 3rd party research) ▸ If things do break, we need a quick way to mitigate either by rolling back or forward ▸ Having a reliable deployment pipeline is critical for us
8. LET’S TALK ABOUT KUBERNETES
9. KUBERNETES OBJECTS ▸ Usually represented as YAML or JSON (but usually YAML) ▸ Describe the desired state of your Kubernetes cluster apiVersion: apps/v1 kind: Deployment metadata: name: rails namespace: laddertruck labels: owner: infra spec: replicas: 20 strategy: type: RollingUpdate
10. SOME OBJECT EXAMPLES ▸ Basically everything in Kubernetes is represented with an object ▸ Deployments ▸ Pods ▸ Volumes ▸ Even IP Endpoints
11. PODS, REPLICASETS, DEPLOYS, OH MY DEPLOYMENTS ▸ It exists under the “workloads” API in Kubernetes ▸ The Kubernetes deployment object is a “wrapper” of sorts ▸ It controls the creation of replica sets, which control the creation of pods ▸ ReplicaSet’s created have an ownership ref on them https://thenewstack.io/kubernetes-deployments-work/
12. ROLLING OUT NEW POD SPECS K8S DEPLOYMENTS SOLVE
13. ROLLBACK POINTS K8S DEPLOYMENTS SOLVE
14. DEPLOYMENTS CREATE VERSIONED REPLICA SETS
15. KUBERNETES OWNERSHIP REFS Deployments create replica sets, which create pods # kubectl get rs -n laddertruck rails-5d6c47bccb -o yaml apiVersion: extensions/v1beta1 kind: ReplicaSet metadata: creationTimestamp: 2019-07-10T11:55:21Z generation: 2 labels: app: rails name: rails pod-template-hash: "1827036776" name: rails-5d6c47bccb namespace: laddertruck ownerReferences: - apiVersion: apps/v1 blockOwnerDeletion: true controller: true kind: Deployment name: rails uid: 4f4d6a6f-3c93-11e9-ad7a-4201ac1e0416
16. KUBERNETES OWNERSHIP REFS Deployments create replica sets, which create pods # kubectl get rs -n laddertruck rails-5d6c47bccb -o yaml apiVersion: extensions/v1beta1 kind: ReplicaSet metadata: creationTimestamp: 2019-07-10T11:55:21Z generation: 2 labels: app: rails name: rails pod-template-hash: "1827036776" name: rails-5d6c47bccb namespace: laddertruck ownerReferences: - apiVersion: apps/v1 blockOwnerDeletion: true controller: true kind: Deployment name: rails uid: 4f4d6a6f-3c93-11e9-ad7a-4201ac1e0416 This replica set is owned by
 this deployment
17. OUR DEPLOYMENT MANIFEST
18. apiVersion: apps/v1 kind: Deployment metadata: labels: app: rails app.kubernetes.io/component: rails app.kubernetes.io/managed-by: weave app.kubernetes.io/name: laddertruck name: rails namespace: laddertruck spec: progressDeadlineSeconds: 600 replicas: 10 revisionHistoryLimit: 10 selector: matchLabels: name: rails strategy: rollingUpdate: maxSurge: 75% maxUnavailable: 75% type: RollingUpdate template: metadata: ... OUR DEPLOYMENT
19. apiVersion: apps/v1 kind: Deployment metadata: labels: app: rails app.kubernetes.io/component: rails app.kubernetes.io/managed-by: weave app.kubernetes.io/name: laddertruck name: rails namespace: laddertruck spec: progressDeadlineSeconds: 600 replicas: 10 revisionHistoryLimit: 10 selector: matchLabels: name: rails strategy: rollingUpdate: maxSurge: 75% maxUnavailable: 75% type: RollingUpdate template: metadata: ... OUR DEPLOYMENT MANIFEST NUMBER OF PODS TO RUN NUMBER OF ROLLBACK POINTS TO KEEP
20. REVISE ME CONTROLLER REVISIONS ▸ Deployments, DaemonSets, and StatefulSets are all “controllers” ▸ These object types will maintain a history of revisions. ▸ Revisions are only created when you modify the PodSpec portion of your deploy (updating the image, env vars, volumes, etc) ▸ Revisions are stored depending on the revisionHistoryLimit parameter
21. SHOW ME REVISIONS! $ kubectl -n laddertruck rollout history deployment rails deployments "rails" REVISION CHANGE-CAUSE 217 <none> 218 <none> 219 <none> 220 <none> 221 <none>
22. SHOW ME A SPECIFIC ONE! $ kubectl -n laddertruck rollout history deployment rails --revision=227 deployments "rails" with revision #227 Pod Template: Labels: app=rails date=1556589959 name=rails pod-template-hash=2400309021 Containers: rails: Image: us.gcr.io/infrastructure-58ca87ae/ laddertruck:master-80fa93a5435fa927c36b4d1ca0a57fca8ca1d1c4 Port: 3000/TCP Host Port: 0/TCP Requests: cpu: 500m memory: 512Mi Readiness: http-get http://:3000/infra/health delay=10s timeout=1s period=5s #success=1 #failure=3 Environment Variables from: slack Secret Optional: false
23. REVISIONS ARE KEY ▸ Revisions are what enable you to rollback a bad deploy quickly ▸ They’re a stored version of your pod definition that can be referenced later when things catch on fire
24. HOW ADVENTUROUS ARE YOU? SO HOW SHOULD I DEPLOY?
25. THE WORLDS MOST ADVANCED BASH TERMINAL FIRST UP
26. DON’T HATE THE JENK JENKINS IS FINE FOR SIMPLE DEPLOYS HONESTLY ▸ Jenkins to execute kubectl commands is ok in a lot of cases ▸ I would wager most deploys to Kubernetes are run this way
27. DON’T HATE THE JENK JENKINS IS FINE FOR SIMPLE DEPLOYS HONESTLY ▸ A simple kubectl command from Jenkins could look like: kubectl set image deployment.v1.apps/nginx-deployment nginx=nginx:1.9.1 ▸ Then wait for it to finish * $ kubectl wait -n laddertruck deploy/rails --for condition=available deployment.extensions/rails condition met * This won’t catch pods that didn’t actually come online
28. SECOND ON DECK
29. THE CADILLAC OF CONTINUOUS DELIVERY SUPER ADVANCED ▸ Spinnaker can do everything you need for a continuous deployment process ▸ Deploys, Rollbacks, Scale ups, Advanced Pipelines ▸ Authentication ▸ Web UI ▸ So. Many. Things.
30. ONE PROBLEM, THOUGH
31. HAVE FUN SETTING IT UP
32. TRUST ME
33. I GAVE A TALK ABOUT IT
34. AT WHAT COST THE POWER COMES WITH A STEEP COST ▸ Spinnaker is a completely new tool that your engineering team will have to learn in order to deploy ▸ It’s very complicated to setup and run effectively 
 (especially across multiple environments) ▸ Configuration is difficult to keep as code ▸ You will need a dedicated team of at least 2 people to install and get it going for your organization in a timely fashion ▸ If you’re still interested, check out armory.io - They’re a managed Spinnaker consultancy and super smart folks
35. WEAVE FLUX OUR CHOICE
36. FLUX ▸ A GitOps Kubernetes Controller ▸ Relies on a git repository containing your configuration ▸ Can watch a Docker registry for new images and apply the changes ▸ Can deploy any k8s object (CRD, Deployments, CronJobs, etc)
37. YAML AS A SERVICE DEDICATED YAML REPO ▸ We store all of our Kubernetes manifests in repo called “k8s- manifests” ▸ We separate environments with a folder ▸ Folders for the separate applications and related manifest files
38. DEPLOY IT LIVE LET’S
39. CIRCLECI GOOGLE CONTAINER REGISTRY FLUX GITHUB ‣ CircleCI Runs our tests, builds a Docker image and pushes it to Google Container Registry (GCR) ‣ Tags are formatted as us.gcr.io/infrastructure-58ca87ae/ laddertruck:staging-123517023ede043c0d1987c106b822d9ac0ef26a ‣ Name of the branch and commit SHA ‣ Flux is looking for new images that match a pattern in our registry it doesn’t know about ‣ If a new image is pushed that matches, Flux clones and updates our GitHub repository of Kubernetes objects (YAML files) ‣ It then applies the change to the cluster it is running on K8S
40. DEPLOYMENT FILES ▸ Flux relies on annotations in your Kubernetes objects to perform it’s deployment operations ▸ For example, we inform Flux which images we care about by using the annotation: flux.weave.works/tag.rails: glob:master-* ▸ Any images that match this pattern on the registry will be
 registered for a deployment in Flux ‣ us.gcr.io/infrastructure-58ca87ae/ laddertruck:master-123517023ede043c0d1987c106b822d9ac0ef26a
41. WEAVE ANNOTATIONS MAPPING TAGS TO CONTAINERS ▸ The annotation 
 “flux.weave.works/tag.rails” maps to the container “rails” in our pod specification ▸ This is how Weave knows which container in the PodSpec it needs to update
42. WEIRD FLUX BUT OK THEN MAGIC HAPPENS ▸ If an image is pushed to our registry that matches our tag annotation, Weave updates the image value in the spec
43. CHECK IF IT WORKED LET’S
44. CLOSING UP ▸ Simple, Consistent Deploys = Safer ▸ Deploying code should be the most boring part of your infrastructure ▸ Using tools you’re already know (Git for example) makes the barrier to entry much smaller
45. STAY IN TOUCH bobby@firehydrant.io @bobbytables github.com/bobbytables github.com/firehydrant .io