Upgrade to Pro — share decks privately, control downloads, hide ads and more …

The (long) road to Kubernetes

The (long) road to Kubernetes

Pierre-Yves Ritschard

January 31, 2019
Tweet

More Decks by Pierre-Yves Ritschard

Other Decks in Technology

Transcript

  1. 2/1/2019 The (long) road to Kubernetes http://localhost:8000/pres.html?print-pdf#/sec-title-slide 3/67 OUTLINE OUTLINE

    A small intro for context The road to distributed systems Keeping promises in a containerized world 3 . 1
  2. 2/1/2019 The (long) road to Kubernetes http://localhost:8000/pres.html?print-pdf#/sec-title-slide 4/67 EXOSCALE EXOSCALE

    Infrastructure as a service Part of A1 Digital Zones in Frankfurt, Vienna, Zürich, Geneva, more in 2019! 4 . 1
  3. 2/1/2019 The (long) road to Kubernetes http://localhost:8000/pres.html?print-pdf#/sec-title-slide 6/67 EXOSCALE EXOSCALE

    provider "exoscale" { api_key = "${var.exoscale_api_key}" secret_key = "${var.exoscale_secret_key}" } resource "exoscale_instance" "web" { template = "Ubuntu 18.04" disk_size = "50g" profile = "medium" ssh_key = "production" } 6 . 1
  4. 2/1/2019 The (long) road to Kubernetes http://localhost:8000/pres.html?print-pdf#/sec-title-slide 7/67 I THOUGHT

    THIS WAS A CONTAINER I THOUGHT THIS WAS A CONTAINER CONFERENCE! CONFERENCE! 7 . 1
  5. 2/1/2019 The (long) road to Kubernetes http://localhost:8000/pres.html?print-pdf#/sec-title-slide 8/67 WHAT'S IN

    A CLOUD PROVIDER WHAT'S IN A CLOUD PROVIDER Datacenter operations So ware development 8 . 1
  6. 2/1/2019 The (long) road to Kubernetes http://localhost:8000/pres.html?print-pdf#/sec-title-slide 9/67 SOFTWARE AT

    EXOSCALE SOFTWARE AT EXOSCALE Object storage controller Network controller Internal SDN Customer management Metering system Billing Web portal 9 . 1
  7. 2/1/2019 The (long) road to Kubernetes http://localhost:8000/pres.html?print-pdf#/sec-title-slide 10/67 LANGUAGES AT

    EXOSCALE LANGUAGES AT EXOSCALE C & Go Clojure Python ClojureScript & JS 10 . 1
  8. 2/1/2019 The (long) road to Kubernetes http://localhost:8000/pres.html?print-pdf#/sec-title-slide 12/67 IT ALWAYS

    STARTS WITH A SIMPLE PRODUCT IT ALWAYS STARTS WITH A SIMPLE PRODUCT You want to change the world by disrupting the job board industry Standard three-tier, self-contained app Does not fall into the usual definition of distributed systems 12 . 1
  9. 2/1/2019 The (long) road to Kubernetes http://localhost:8000/pres.html?print-pdf#/sec-title-slide 13/67 IT ALWAYS

    STARTS WITH A SIMPLE PRODUCT IT ALWAYS STARTS WITH A SIMPLE PRODUCT 13 . 1
  10. 2/1/2019 The (long) road to Kubernetes http://localhost:8000/pres.html?print-pdf#/sec-title-slide 14/67 FIRST SIGNS

    OF SUCCESS FIRST SIGNS OF SUCCESS Your single server is not sufficient anymore Database gets its own machines, adding new web servers fixes the issue Logging becomes a bit harder You switch to a centralized logging solution 14 . 1
  11. 2/1/2019 The (long) road to Kubernetes http://localhost:8000/pres.html?print-pdf#/sec-title-slide 16/67 FEATURES GET

    ADDED FEATURES GET ADDED Subscriptions emails Doing it synchronously is impossible Let's add a worker (and thus a queueing mechanism) You start switching from pets to cattle 16 . 1
  12. 2/1/2019 The (long) road to Kubernetes http://localhost:8000/pres.html?print-pdf#/sec-title-slide 18/67 SEED MONEY

    RUNS OUT SEED MONEY RUNS OUT Let's try other monetization techniques Freemium model with analytics Where do I run these batch jobs? You partner with another company to exchange data They have this weird legacy system and the only client lib is in PHP :-( 18 . 1
  13. 2/1/2019 The (long) road to Kubernetes http://localhost:8000/pres.html?print-pdf#/sec-title-slide 20/67 AS THE

    PRODUCT GROWS, SO DOES AS THE PRODUCT GROWS, SO DOES INFRASTRUCTURE INFRASTRUCTURE You're now at 3 jenkins workers You had to split metrics and monitoring on separate machines You introduce a command and control solution to perform your regular operations It's time to use puppet (You're really starting to feel like an ops person now) 20 . 1
  14. 2/1/2019 The (long) road to Kubernetes http://localhost:8000/pres.html?print-pdf#/sec-title-slide 21/67 LET'S TAKE

    A STEP BACK LET'S TAKE A STEP BACK You're ticking all the boxes: CI, Infrastructure as Code, DevOps 21 . 1
  15. 2/1/2019 The (long) road to Kubernetes http://localhost:8000/pres.html?print-pdf#/sec-title-slide 22/67 BUT RESOURCE

    UTILIZATION IS LOW BUT RESOURCE UTILIZATION IS LOW Most of it is articifial (agents on every nodes) But you still have peak induced regular contention 22 . 1
  16. 2/1/2019 The (long) road to Kubernetes http://localhost:8000/pres.html?print-pdf#/sec-title-slide 23/67 ADDING NEW

    SERVICES OR COMPONENTS IS HARD ADDING NEW SERVICES OR COMPONENTS IS HARD Should your most active git repository really be the puppet one? You constantly have to make allocation decisions 23 . 1
  17. 2/1/2019 The (long) road to Kubernetes http://localhost:8000/pres.html?print-pdf#/sec-title-slide 24/67 HANDLING FAILURE

    IS HARD HANDLING FAILURE IS HARD Your monitoring system tells you when something breaks You have to recreate machines manually, update configuration all over the place 24 . 1
  18. 2/1/2019 The (long) road to Kubernetes http://localhost:8000/pres.html?print-pdf#/sec-title-slide 25/67 FROM A

    SERVICE POINT OF VIEW IT ALL MAKES FROM A SERVICE POINT OF VIEW IT ALL MAKES SENSE SENSE 25 . 1
  19. 2/1/2019 The (long) road to Kubernetes http://localhost:8000/pres.html?print-pdf#/sec-title-slide 28/67 WHERE TO

    FROM HERE WHERE TO FROM HERE How do you get out of the business of shuffling configuration and apps around? 28 . 1
  20. 2/1/2019 The (long) road to Kubernetes http://localhost:8000/pres.html?print-pdf#/sec-title-slide 31/67 KUBERNETES PROMISES

    KUBERNETES PROMISES Utilization optimization Reducing the pain of adding new components Command and control Preserves a service boundaries 31 . 1
  21. 2/1/2019 The (long) road to Kubernetes http://localhost:8000/pres.html?print-pdf#/sec-title-slide 32/67 ADDITIONAL PROMISES

    ADDITIONAL PROMISES A good substrate for creating resources on- demand Lingua-franca for infrastructure concepts Initial learning-curve Most likely smaller than ad-hoc solutions Eating our own dog food 32 . 1
  22. 2/1/2019 The (long) road to Kubernetes http://localhost:8000/pres.html?print-pdf#/sec-title-slide 36/67 REPRODUCIBILITY REPRODUCIBILITY

    Building once (and in chroots) ensures clean packages Reproducible builds make wide changes easier We need staging deploys and production deploys to be identical 36 . 1
  23. 2/1/2019 The (long) road to Kubernetes http://localhost:8000/pres.html?print-pdf#/sec-title-slide 37/67 TRACEABILITY TRACEABILITY

    When did we last build this? What did the output look like? What commit did it correspond to? 37 . 1
  24. 2/1/2019 The (long) road to Kubernetes http://localhost:8000/pres.html?print-pdf#/sec-title-slide 39/67 CHECKPOINTS CHECKPOINTS

    CD is great for test and staging We are wary of unattended production deploys There should be a clear (but simple) trigger 39 . 1
  25. 2/1/2019 The (long) road to Kubernetes http://localhost:8000/pres.html?print-pdf#/sec-title-slide 48/67 AFTER KUBERNETES

    AFTER KUBERNETES Reproducibility ✔ Traceability ✔ Security ✔ Checkpoints ✔ 48 . 1
  26. 2/1/2019 The (long) road to Kubernetes http://localhost:8000/pres.html?print-pdf#/sec-title-slide 49/67 AFTER KUBERNETES:

    PACKAGING AFTER KUBERNETES: PACKAGING Building is faster, easier, and gives developers more autonomy Docker registries forced us to move from enforcement to convention 49 . 1
  27. 2/1/2019 The (long) road to Kubernetes http://localhost:8000/pres.html?print-pdf#/sec-title-slide 50/67 AFTER KUBERNETES:

    CONFIGURATION AFTER KUBERNETES: CONFIGURATION The split across environment services, variables, configmaps, and secrets makes separation easy Greatly reduces the need for config management Configuration can be colocated with the so ware Removes the code, build, and configuration-management impedance mismatch Kept things simple No helm 50 . 1
  28. 2/1/2019 The (long) road to Kubernetes http://localhost:8000/pres.html?print-pdf#/sec-title-slide 51/67 AFTER KUBERNETES:

    ON-DEMAND AFTER KUBERNETES: ON-DEMAND RESOURCES RESOURCES CRDs provide great integration High cardinality or complex queries can be tedious to work with 51 . 1
  29. 2/1/2019 The (long) road to Kubernetes http://localhost:8000/pres.html?print-pdf#/sec-title-slide 52/67 AFTER KUBERNETES:

    SECURITY AFTER KUBERNETES: SECURITY RBAC policies are powerful but tedious to write (and error prone) Certificate management leaves a lot to be improved 52 . 1
  30. 2/1/2019 The (long) road to Kubernetes http://localhost:8000/pres.html?print-pdf#/sec-title-slide 53/67 GOING FORWARD

    GOING FORWARD Going back from convention to enforcement for the registry Providing more Paas-like encoding of our common cases 53 . 1
  31. 2/1/2019 The (long) road to Kubernetes http://localhost:8000/pres.html?print-pdf#/sec-title-slide 55/67 BEFORE KUBERNETES

    BEFORE KUBERNETES A public IP per VM Security groups for firewall management 55 . 1
  32. 2/1/2019 The (long) road to Kubernetes http://localhost:8000/pres.html?print-pdf#/sec-title-slide 61/67 WHAT ABOUT

    EXTERNAL SERVICES WHAT ABOUT EXTERNAL SERVICES By default NodePort services run on all worker nodes Traffic is source-nat'd to the destination Losing source IP information is unviable for most use- cases Performance impact 61 . 1
  33. 2/1/2019 The (long) road to Kubernetes http://localhost:8000/pres.html?print-pdf#/sec-title-slide 62/67 NETWORKING AT

    EXOSCALE NETWORKING AT EXOSCALE A layer3 all-the-way design A BGP first design VM Public IPs advertised by BGP from hypervisors Private network VXLAN membership advertised through BGP- eVPN Best performance is on the public interfaces 62 . 1
  34. 2/1/2019 The (long) road to Kubernetes http://localhost:8000/pres.html?print-pdf#/sec-title-slide 64/67 IN-CLUSTER NETWORKING

    IN-CLUSTER NETWORKING Kept with a BGP-based CNI Avoids additional encapsulation 64 . 1
  35. 2/1/2019 The (long) road to Kubernetes http://localhost:8000/pres.html?print-pdf#/sec-title-slide 65/67 EXTERNAL SERVICES

    EXTERNAL SERVICES Needs additional development We went for IPIP load-balancing Needs node-local decapsulation Makes for a hacky setup eBPF/XDP to the rescue! 65 . 1
  36. 2/1/2019 The (long) road to Kubernetes http://localhost:8000/pres.html?print-pdf#/sec-title-slide 66/67 KUBERNETES AT

    EXOSCALE TODAY KUBERNETES AT EXOSCALE TODAY Most production critical services Some things we are in no hurry to containerize :-) Basis for as-a-service offerings Private network management For customers exo lab kube Cluster API OpenShi On-demand cluster in the Exoscale API 66 . 1