The (long) road to Kubernetes

Slide 1

Slide 1 text

2/1/2019 The (long) road to Kubernetes http://localhost:8000/pres.html?print-pdf#/sec-title-slide 1/67 THE (LONG) ROAD TO THE (LONG) ROAD TO KUBERNETES KUBERNETES 1

Slide 2

Slide 2 text

2/1/2019 The (long) road to Kubernetes http://localhost:8000/pres.html?print-pdf#/sec-title-slide 2/67 @PYR @PYR CTO, co-founder at Exoscale Open source developer 2 . 1

Slide 3

Slide 3 text

2/1/2019 The (long) road to Kubernetes http://localhost:8000/pres.html?print-pdf#/sec-title-slide 3/67 OUTLINE OUTLINE A small intro for context The road to distributed systems Keeping promises in a containerized world 3 . 1

Slide 4

Slide 4 text

2/1/2019 The (long) road to Kubernetes http://localhost:8000/pres.html?print-pdf#/sec-title-slide 4/67 EXOSCALE EXOSCALE Infrastructure as a service Part of A1 Digital Zones in Frankfurt, Vienna, Zürich, Geneva, more in 2019! 4 . 1

Slide 5

Slide 5 text

2/1/2019 The (long) road to Kubernetes http://localhost:8000/pres.html?print-pdf#/sec-title-slide 5/67 EXOSCALE EXOSCALE 5 . 1

Slide 6

Slide 6 text

2/1/2019 The (long) road to Kubernetes http://localhost:8000/pres.html?print-pdf#/sec-title-slide 6/67 EXOSCALE EXOSCALE provider "exoscale" { api_key = "${var.exoscale_api_key}" secret_key = "${var.exoscale_secret_key}" } resource "exoscale_instance" "web" { template = "Ubuntu 18.04" disk_size = "50g" profile = "medium" ssh_key = "production" } 6 . 1

Slide 7

Slide 7 text

2/1/2019 The (long) road to Kubernetes http://localhost:8000/pres.html?print-pdf#/sec-title-slide 7/67 I THOUGHT THIS WAS A CONTAINER I THOUGHT THIS WAS A CONTAINER CONFERENCE! CONFERENCE! 7 . 1

Slide 8

Slide 8 text

2/1/2019 The (long) road to Kubernetes http://localhost:8000/pres.html?print-pdf#/sec-title-slide 8/67 WHAT'S IN A CLOUD PROVIDER WHAT'S IN A CLOUD PROVIDER Datacenter operations So ware development 8 . 1

Slide 9

Slide 9 text

2/1/2019 The (long) road to Kubernetes http://localhost:8000/pres.html?print-pdf#/sec-title-slide 9/67 SOFTWARE AT EXOSCALE SOFTWARE AT EXOSCALE Object storage controller Network controller Internal SDN Customer management Metering system Billing Web portal 9 . 1

Slide 10

Slide 10 text

2/1/2019 The (long) road to Kubernetes http://localhost:8000/pres.html?print-pdf#/sec-title-slide 10/67 LANGUAGES AT EXOSCALE LANGUAGES AT EXOSCALE C & Go Clojure Python ClojureScript & JS 10 . 1

Slide 11

Slide 11 text

2/1/2019 The (long) road to Kubernetes http://localhost:8000/pres.html?print-pdf#/sec-title-slide 11/67 THE ROAD TO DISTRIBUTED SYSTEMS THE ROAD TO DISTRIBUTED SYSTEMS 11 . 1

Slide 12

Slide 12 text

2/1/2019 The (long) road to Kubernetes http://localhost:8000/pres.html?print-pdf#/sec-title-slide 12/67 IT ALWAYS STARTS WITH A SIMPLE PRODUCT IT ALWAYS STARTS WITH A SIMPLE PRODUCT You want to change the world by disrupting the job board industry Standard three-tier, self-contained app Does not fall into the usual definition of distributed systems 12 . 1

Slide 13

Slide 13 text

2/1/2019 The (long) road to Kubernetes http://localhost:8000/pres.html?print-pdf#/sec-title-slide 13/67 IT ALWAYS STARTS WITH A SIMPLE PRODUCT IT ALWAYS STARTS WITH A SIMPLE PRODUCT 13 . 1

Slide 14

Slide 14 text

2/1/2019 The (long) road to Kubernetes http://localhost:8000/pres.html?print-pdf#/sec-title-slide 14/67 FIRST SIGNS OF SUCCESS FIRST SIGNS OF SUCCESS Your single server is not suﬀicient anymore Database gets its own machines, adding new web servers fixes the issue Logging becomes a bit harder You switch to a centralized logging solution 14 . 1

Slide 15

Slide 15 text

2/1/2019 The (long) road to Kubernetes http://localhost:8000/pres.html?print-pdf#/sec-title-slide 15/67 FIRST SIGNS OF SUCCESS FIRST SIGNS OF SUCCESS 15 . 1

Slide 16

Slide 16 text

2/1/2019 The (long) road to Kubernetes http://localhost:8000/pres.html?print-pdf#/sec-title-slide 16/67 FEATURES GET ADDED FEATURES GET ADDED Subscriptions emails Doing it synchronously is impossible Let's add a worker (and thus a queueing mechanism) You start switching from pets to cattle 16 . 1

Slide 17

Slide 17 text

2/1/2019 The (long) road to Kubernetes http://localhost:8000/pres.html?print-pdf#/sec-title-slide 17/67 FEATURES GET ADDED FEATURES GET ADDED 17 . 1

Slide 18

Slide 18 text

2/1/2019 The (long) road to Kubernetes http://localhost:8000/pres.html?print-pdf#/sec-title-slide 18/67 SEED MONEY RUNS OUT SEED MONEY RUNS OUT Let's try other monetization techniques Freemium model with analytics Where do I run these batch jobs? You partner with another company to exchange data They have this weird legacy system and the only client lib is in PHP :-( 18 . 1

Slide 19

Slide 19 text

2/1/2019 The (long) road to Kubernetes http://localhost:8000/pres.html?print-pdf#/sec-title-slide 19/67 SEED MONEY RUNS OUT SEED MONEY RUNS OUT 19 . 1

Slide 20

Slide 20 text

2/1/2019 The (long) road to Kubernetes http://localhost:8000/pres.html?print-pdf#/sec-title-slide 20/67 AS THE PRODUCT GROWS, SO DOES AS THE PRODUCT GROWS, SO DOES INFRASTRUCTURE INFRASTRUCTURE You're now at 3 jenkins workers You had to split metrics and monitoring on separate machines You introduce a command and control solution to perform your regular operations It's time to use puppet (You're really starting to feel like an ops person now) 20 . 1

Slide 21

Slide 21 text

2/1/2019 The (long) road to Kubernetes http://localhost:8000/pres.html?print-pdf#/sec-title-slide 21/67 LET'S TAKE A STEP BACK LET'S TAKE A STEP BACK You're ticking all the boxes: CI, Infrastructure as Code, DevOps 21 . 1

Slide 22

Slide 22 text

2/1/2019 The (long) road to Kubernetes http://localhost:8000/pres.html?print-pdf#/sec-title-slide 22/67 BUT RESOURCE UTILIZATION IS LOW BUT RESOURCE UTILIZATION IS LOW Most of it is articifial (agents on every nodes) But you still have peak induced regular contention 22 . 1

Slide 23

Slide 23 text

2/1/2019 The (long) road to Kubernetes http://localhost:8000/pres.html?print-pdf#/sec-title-slide 23/67 ADDING NEW SERVICES OR COMPONENTS IS HARD ADDING NEW SERVICES OR COMPONENTS IS HARD Should your most active git repository really be the puppet one? You constantly have to make allocation decisions 23 . 1

Slide 24

Slide 24 text

2/1/2019 The (long) road to Kubernetes http://localhost:8000/pres.html?print-pdf#/sec-title-slide 24/67 HANDLING FAILURE IS HARD HANDLING FAILURE IS HARD Your monitoring system tells you when something breaks You have to recreate machines manually, update configuration all over the place 24 . 1

Slide 25

Slide 25 text

2/1/2019 The (long) road to Kubernetes http://localhost:8000/pres.html?print-pdf#/sec-title-slide 25/67 FROM A SERVICE POINT OF VIEW IT ALL MAKES FROM A SERVICE POINT OF VIEW IT ALL MAKES SENSE SENSE 25 . 1

Slide 26

Slide 26 text

2/1/2019 The (long) road to Kubernetes http://localhost:8000/pres.html?print-pdf#/sec-title-slide 26/67 WHAT WOULD BE NICE WHAT WOULD BE NICE 26 . 1

Slide 27

Slide 27 text

2/1/2019 The (long) road to Kubernetes http://localhost:8000/pres.html?print-pdf#/sec-title-slide 27/67 WHAT WOULD BE NICE WHAT WOULD BE NICE 27 . 1

Slide 28

Slide 28 text

2/1/2019 The (long) road to Kubernetes http://localhost:8000/pres.html?print-pdf#/sec-title-slide 28/67 WHERE TO FROM HERE WHERE TO FROM HERE How do you get out of the business of shuﬀling configuration and apps around? 28 . 1

Slide 29

Slide 29 text

2/1/2019 The (long) road to Kubernetes http://localhost:8000/pres.html?print-pdf#/sec-title-slide 29/67 ALTERNATIVES CONSIDERED ALTERNATIVES CONSIDERED Mesos Docker LXD Kubernetes 29 . 1

Slide 30

Slide 30 text

2/1/2019 The (long) road to Kubernetes http://localhost:8000/pres.html?print-pdf#/sec-title-slide 30/67 ALTERNATIVES CONSIDERED ALTERNATIVES CONSIDERED Mesos Docker LXD Kubernetes 30 . 1

Slide 31

Slide 31 text

2/1/2019 The (long) road to Kubernetes http://localhost:8000/pres.html?print-pdf#/sec-title-slide 31/67 KUBERNETES PROMISES KUBERNETES PROMISES Utilization optimization Reducing the pain of adding new components Command and control Preserves a service boundaries 31 . 1

Slide 32

Slide 32 text

2/1/2019 The (long) road to Kubernetes http://localhost:8000/pres.html?print-pdf#/sec-title-slide 32/67 ADDITIONAL PROMISES ADDITIONAL PROMISES A good substrate for creating resources on- demand Lingua-franca for infrastructure concepts Initial learning-curve Most likely smaller than ad-hoc solutions Eating our own dog food 32 . 1

Slide 33

Slide 33 text

2/1/2019 The (long) road to Kubernetes http://localhost:8000/pres.html?print-pdf#/sec-title-slide 33/67 LET'S INSTALL KUBERNETES AND GO LET'S INSTALL KUBERNETES AND GO HOME? HOME? 33 . 1

Slide 34

Slide 34 text

2/1/2019 The (long) road to Kubernetes http://localhost:8000/pres.html?print-pdf#/sec-title-slide 34/67 REMAINING WORRIES REMAINING WORRIES Security Monitoring Deployment Process Networking 34 . 1

Slide 35

Slide 35 text

2/1/2019 The (long) road to Kubernetes http://localhost:8000/pres.html?print-pdf#/sec-title-slide 35/67 DEPLOYMENT PROCESS DEPLOYMENT PROCESS Reproducibility Traceability Security Checkpoints 35 . 1

Slide 36

Slide 36 text

2/1/2019 The (long) road to Kubernetes http://localhost:8000/pres.html?print-pdf#/sec-title-slide 36/67 REPRODUCIBILITY REPRODUCIBILITY Building once (and in chroots) ensures clean packages Reproducible builds make wide changes easier We need staging deploys and production deploys to be identical 36 . 1

Slide 37

Slide 37 text

2/1/2019 The (long) road to Kubernetes http://localhost:8000/pres.html?print-pdf#/sec-title-slide 37/67 TRACEABILITY TRACEABILITY When did we last build this? What did the output look like? What commit did it correspond to? 37 . 1

Slide 38

Slide 38 text

2/1/2019 The (long) road to Kubernetes http://localhost:8000/pres.html?print-pdf#/sec-title-slide 38/67 SECURITY SECURITY No code download on production hosts Signed packages 38 . 1

Slide 39

Slide 39 text

2/1/2019 The (long) road to Kubernetes http://localhost:8000/pres.html?print-pdf#/sec-title-slide 39/67 CHECKPOINTS CHECKPOINTS CD is great for test and staging We are wary of unattended production deploys There should be a clear (but simple) trigger 39 . 1

Slide 40

Slide 40 text

2/1/2019 The (long) road to Kubernetes http://localhost:8000/pres.html?print-pdf#/sec-title-slide 40/67 BEFORE KUBERNETES BEFORE KUBERNETES 40 . 1

Slide 41

Slide 41 text

2/1/2019 The (long) road to Kubernetes http://localhost:8000/pres.html?print-pdf#/sec-title-slide 41/67 BEFORE KUBERNETES BEFORE KUBERNETES 41 . 1

Slide 42

Slide 42 text

2/1/2019 The (long) road to Kubernetes http://localhost:8000/pres.html?print-pdf#/sec-title-slide 42/67 BEFORE KUBERNETES BEFORE KUBERNETES 42 . 1

Slide 43

Slide 43 text

2/1/2019 The (long) road to Kubernetes http://localhost:8000/pres.html?print-pdf#/sec-title-slide 43/67 BEFORE KUBERNETES BEFORE KUBERNETES 43 . 1

Slide 44

Slide 44 text

2/1/2019 The (long) road to Kubernetes http://localhost:8000/pres.html?print-pdf#/sec-title-slide 44/67 BEFORE KUBERNETES BEFORE KUBERNETES 44 . 1

Slide 45

Slide 45 text

2/1/2019 The (long) road to Kubernetes http://localhost:8000/pres.html?print-pdf#/sec-title-slide 45/67 BEFORE KUBERNETES BEFORE KUBERNETES 45 . 1

Slide 46

Slide 46 text

2/1/2019 The (long) road to Kubernetes http://localhost:8000/pres.html?print-pdf#/sec-title-slide 46/67 BEFORE KUBERNETES BEFORE KUBERNETES 46 . 1

Slide 47

Slide 47 text

2/1/2019 The (long) road to Kubernetes http://localhost:8000/pres.html?print-pdf#/sec-title-slide 47/67 AFTER KUBERNETES AFTER KUBERNETES 47 . 1

Slide 48

Slide 48 text

2/1/2019 The (long) road to Kubernetes http://localhost:8000/pres.html?print-pdf#/sec-title-slide 48/67 AFTER KUBERNETES AFTER KUBERNETES Reproducibility ✔ Traceability ✔ Security ✔ Checkpoints ✔ 48 . 1

Slide 49

Slide 49 text

2/1/2019 The (long) road to Kubernetes http://localhost:8000/pres.html?print-pdf#/sec-title-slide 49/67 AFTER KUBERNETES: PACKAGING AFTER KUBERNETES: PACKAGING Building is faster, easier, and gives developers more autonomy Docker registries forced us to move from enforcement to convention 49 . 1

Slide 50

Slide 50 text

2/1/2019 The (long) road to Kubernetes http://localhost:8000/pres.html?print-pdf#/sec-title-slide 50/67 AFTER KUBERNETES: CONFIGURATION AFTER KUBERNETES: CONFIGURATION The split across environment services, variables, configmaps, and secrets makes separation easy Greatly reduces the need for config management Configuration can be colocated with the so ware Removes the code, build, and configuration-management impedance mismatch Kept things simple No helm 50 . 1

Slide 51

Slide 51 text

2/1/2019 The (long) road to Kubernetes http://localhost:8000/pres.html?print-pdf#/sec-title-slide 51/67 AFTER KUBERNETES: ON-DEMAND AFTER KUBERNETES: ON-DEMAND RESOURCES RESOURCES CRDs provide great integration High cardinality or complex queries can be tedious to work with 51 . 1

Slide 52

Slide 52 text

2/1/2019 The (long) road to Kubernetes http://localhost:8000/pres.html?print-pdf#/sec-title-slide 52/67 AFTER KUBERNETES: SECURITY AFTER KUBERNETES: SECURITY RBAC policies are powerful but tedious to write (and error prone) Certificate management leaves a lot to be improved 52 . 1

Slide 53

Slide 53 text

2/1/2019 The (long) road to Kubernetes http://localhost:8000/pres.html?print-pdf#/sec-title-slide 53/67 GOING FORWARD GOING FORWARD Going back from convention to enforcement for the registry Providing more Paas-like encoding of our common cases 53 . 1

Slide 54

Slide 54 text

2/1/2019 The (long) road to Kubernetes http://localhost:8000/pres.html?print-pdf#/sec-title-slide 54/67 NETWORKING NETWORKING Security Scalability 54 . 1

Slide 55

Slide 55 text

2/1/2019 The (long) road to Kubernetes http://localhost:8000/pres.html?print-pdf#/sec-title-slide 55/67 BEFORE KUBERNETES BEFORE KUBERNETES A public IP per VM Security groups for firewall management 55 . 1

Slide 56

Slide 56 text

2/1/2019 The (long) road to Kubernetes http://localhost:8000/pres.html?print-pdf#/sec-title-slide 56/67 BEFORE KUBERNETES BEFORE KUBERNETES A boring, solid network 56 . 1

Slide 57

Slide 57 text

2/1/2019 The (long) road to Kubernetes http://localhost:8000/pres.html?print-pdf#/sec-title-slide 57/67 KUBERNETES NETWORKING 101 KUBERNETES NETWORKING 101 57 . 1

Slide 58

Slide 58 text

2/1/2019 The (long) road to Kubernetes http://localhost:8000/pres.html?print-pdf#/sec-title-slide 58/67 KUBERNETES NETWORKING 101 KUBERNETES NETWORKING 101 58 . 1

Slide 59

Slide 59 text

2/1/2019 The (long) road to Kubernetes http://localhost:8000/pres.html?print-pdf#/sec-title-slide 59/67 KUBERNETES NETWORKING 101 KUBERNETES NETWORKING 101 59 . 1

Slide 60

Slide 60 text

2/1/2019 The (long) road to Kubernetes http://localhost:8000/pres.html?print-pdf#/sec-title-slide 60/67 KUBERNETES NETWORKING 101 KUBERNETES NETWORKING 101 A boring, solid network 60 . 1

Slide 61

Slide 61 text

2/1/2019 The (long) road to Kubernetes http://localhost:8000/pres.html?print-pdf#/sec-title-slide 61/67 WHAT ABOUT EXTERNAL SERVICES WHAT ABOUT EXTERNAL SERVICES By default NodePort services run on all worker nodes Traﬀic is source-nat'd to the destination Losing source IP information is unviable for most use- cases Performance impact 61 . 1

Slide 62

Slide 62 text

2/1/2019 The (long) road to Kubernetes http://localhost:8000/pres.html?print-pdf#/sec-title-slide 62/67 NETWORKING AT EXOSCALE NETWORKING AT EXOSCALE A layer3 all-the-way design A BGP first design VM Public IPs advertised by BGP from hypervisors Private network VXLAN membership advertised through BGP- eVPN Best performance is on the public interfaces 62 . 1

Slide 63

Slide 63 text

2/1/2019 The (long) road to Kubernetes http://localhost:8000/pres.html?print-pdf#/sec-title-slide 63/67 NETWORKING: AFTER KUBERNETES NETWORKING: AFTER KUBERNETES Security ✔ Scalability ✔ 63 . 1

Slide 64

Slide 64 text

2/1/2019 The (long) road to Kubernetes http://localhost:8000/pres.html?print-pdf#/sec-title-slide 64/67 IN-CLUSTER NETWORKING IN-CLUSTER NETWORKING Kept with a BGP-based CNI Avoids additional encapsulation 64 . 1

Slide 65

Slide 65 text

2/1/2019 The (long) road to Kubernetes http://localhost:8000/pres.html?print-pdf#/sec-title-slide 65/67 EXTERNAL SERVICES EXTERNAL SERVICES Needs additional development We went for IPIP load-balancing Needs node-local decapsulation Makes for a hacky setup eBPF/XDP to the rescue! 65 . 1

Slide 66

Slide 66 text

2/1/2019 The (long) road to Kubernetes http://localhost:8000/pres.html?print-pdf#/sec-title-slide 66/67 KUBERNETES AT EXOSCALE TODAY KUBERNETES AT EXOSCALE TODAY Most production critical services Some things we are in no hurry to containerize :-) Basis for as-a-service oﬀerings Private network management For customers exo lab kube Cluster API OpenShi On-demand cluster in the Exoscale API 66 . 1

Slide 67

Slide 67 text

2/1/2019 The (long) road to Kubernetes http://localhost:8000/pres.html?print-pdf#/sec-title-slide 67/67 QUESTIONS? QUESTIONS? We're productizing all of this as we speak! 67 . 1