The (long) road to Kubernetes

Transcript

1. 2/1/2019 The (long) road to Kubernetes http://localhost:8000/pres.html?print-pdf#/sec-title-slide 1/67 THE (LONG)

   ROAD TO THE (LONG) ROAD TO KUBERNETES KUBERNETES 1

2. 2/1/2019 The (long) road to Kubernetes http://localhost:8000/pres.html?print-pdf#/sec-title-slide 2/67 @PYR @PYR

   CTO, co-founder at Exoscale Open source developer 2 . 1

3. 2/1/2019 The (long) road to Kubernetes http://localhost:8000/pres.html?print-pdf#/sec-title-slide 3/67 OUTLINE OUTLINE

   A small intro for context The road to distributed systems Keeping promises in a containerized world 3 . 1

4. 2/1/2019 The (long) road to Kubernetes http://localhost:8000/pres.html?print-pdf#/sec-title-slide 4/67 EXOSCALE EXOSCALE

   Infrastructure as a service Part of A1 Digital Zones in Frankfurt, Vienna, Zürich, Geneva, more in 2019! 4 . 1

5. 2/1/2019 The (long) road to Kubernetes http://localhost:8000/pres.html?print-pdf#/sec-title-slide 5/67 EXOSCALE EXOSCALE

   5 . 1

6. 2/1/2019 The (long) road to Kubernetes http://localhost:8000/pres.html?print-pdf#/sec-title-slide 6/67 EXOSCALE EXOSCALE

   provider "exoscale" { api_key = "${var.exoscale_api_key}" secret_key = "${var.exoscale_secret_key}" } resource "exoscale_instance" "web" { template = "Ubuntu 18.04" disk_size = "50g" profile = "medium" ssh_key = "production" } 6 . 1

7. 2/1/2019 The (long) road to Kubernetes http://localhost:8000/pres.html?print-pdf#/sec-title-slide 7/67 I THOUGHT

   THIS WAS A CONTAINER I THOUGHT THIS WAS A CONTAINER CONFERENCE! CONFERENCE! 7 . 1

8. 2/1/2019 The (long) road to Kubernetes http://localhost:8000/pres.html?print-pdf#/sec-title-slide 8/67 WHAT'S IN

   A CLOUD PROVIDER WHAT'S IN A CLOUD PROVIDER Datacenter operations So ware development 8 . 1

9. 2/1/2019 The (long) road to Kubernetes http://localhost:8000/pres.html?print-pdf#/sec-title-slide 9/67 SOFTWARE AT

   EXOSCALE SOFTWARE AT EXOSCALE Object storage controller Network controller Internal SDN Customer management Metering system Billing Web portal 9 . 1

10. 2/1/2019 The (long) road to Kubernetes http://localhost:8000/pres.html?print-pdf#/sec-title-slide 10/67 LANGUAGES AT

    EXOSCALE LANGUAGES AT EXOSCALE C & Go Clojure Python ClojureScript & JS 10 . 1

11. 2/1/2019 The (long) road to Kubernetes http://localhost:8000/pres.html?print-pdf#/sec-title-slide 11/67 THE ROAD

    TO DISTRIBUTED SYSTEMS THE ROAD TO DISTRIBUTED SYSTEMS 11 . 1

12. 2/1/2019 The (long) road to Kubernetes http://localhost:8000/pres.html?print-pdf#/sec-title-slide 12/67 IT ALWAYS

    STARTS WITH A SIMPLE PRODUCT IT ALWAYS STARTS WITH A SIMPLE PRODUCT You want to change the world by disrupting the job board industry Standard three-tier, self-contained app Does not fall into the usual definition of distributed systems 12 . 1

13. 2/1/2019 The (long) road to Kubernetes http://localhost:8000/pres.html?print-pdf#/sec-title-slide 13/67 IT ALWAYS

    STARTS WITH A SIMPLE PRODUCT IT ALWAYS STARTS WITH A SIMPLE PRODUCT 13 . 1

14. 2/1/2019 The (long) road to Kubernetes http://localhost:8000/pres.html?print-pdf#/sec-title-slide 14/67 FIRST SIGNS

    OF SUCCESS FIRST SIGNS OF SUCCESS Your single server is not sufficient anymore Database gets its own machines, adding new web servers fixes the issue Logging becomes a bit harder You switch to a centralized logging solution 14 . 1

15. 2/1/2019 The (long) road to Kubernetes http://localhost:8000/pres.html?print-pdf#/sec-title-slide 15/67 FIRST SIGNS

    OF SUCCESS FIRST SIGNS OF SUCCESS 15 . 1

16. 2/1/2019 The (long) road to Kubernetes http://localhost:8000/pres.html?print-pdf#/sec-title-slide 16/67 FEATURES GET

    ADDED FEATURES GET ADDED Subscriptions emails Doing it synchronously is impossible Let's add a worker (and thus a queueing mechanism) You start switching from pets to cattle 16 . 1

17. 2/1/2019 The (long) road to Kubernetes http://localhost:8000/pres.html?print-pdf#/sec-title-slide 17/67 FEATURES GET

    ADDED FEATURES GET ADDED 17 . 1

18. 2/1/2019 The (long) road to Kubernetes http://localhost:8000/pres.html?print-pdf#/sec-title-slide 18/67 SEED MONEY

    RUNS OUT SEED MONEY RUNS OUT Let's try other monetization techniques Freemium model with analytics Where do I run these batch jobs? You partner with another company to exchange data They have this weird legacy system and the only client lib is in PHP :-( 18 . 1

19. 2/1/2019 The (long) road to Kubernetes http://localhost:8000/pres.html?print-pdf#/sec-title-slide 19/67 SEED MONEY

    RUNS OUT SEED MONEY RUNS OUT 19 . 1

20. 2/1/2019 The (long) road to Kubernetes http://localhost:8000/pres.html?print-pdf#/sec-title-slide 20/67 AS THE

    PRODUCT GROWS, SO DOES AS THE PRODUCT GROWS, SO DOES INFRASTRUCTURE INFRASTRUCTURE You're now at 3 jenkins workers You had to split metrics and monitoring on separate machines You introduce a command and control solution to perform your regular operations It's time to use puppet (You're really starting to feel like an ops person now) 20 . 1

21. 2/1/2019 The (long) road to Kubernetes http://localhost:8000/pres.html?print-pdf#/sec-title-slide 21/67 LET'S TAKE

    A STEP BACK LET'S TAKE A STEP BACK You're ticking all the boxes: CI, Infrastructure as Code, DevOps 21 . 1

22. 2/1/2019 The (long) road to Kubernetes http://localhost:8000/pres.html?print-pdf#/sec-title-slide 22/67 BUT RESOURCE

    UTILIZATION IS LOW BUT RESOURCE UTILIZATION IS LOW Most of it is articifial (agents on every nodes) But you still have peak induced regular contention 22 . 1

23. 2/1/2019 The (long) road to Kubernetes http://localhost:8000/pres.html?print-pdf#/sec-title-slide 23/67 ADDING NEW

    SERVICES OR COMPONENTS IS HARD ADDING NEW SERVICES OR COMPONENTS IS HARD Should your most active git repository really be the puppet one? You constantly have to make allocation decisions 23 . 1

24. 2/1/2019 The (long) road to Kubernetes http://localhost:8000/pres.html?print-pdf#/sec-title-slide 24/67 HANDLING FAILURE

    IS HARD HANDLING FAILURE IS HARD Your monitoring system tells you when something breaks You have to recreate machines manually, update configuration all over the place 24 . 1

25. 2/1/2019 The (long) road to Kubernetes http://localhost:8000/pres.html?print-pdf#/sec-title-slide 25/67 FROM A

    SERVICE POINT OF VIEW IT ALL MAKES FROM A SERVICE POINT OF VIEW IT ALL MAKES SENSE SENSE 25 . 1

26. 2/1/2019 The (long) road to Kubernetes http://localhost:8000/pres.html?print-pdf#/sec-title-slide 26/67 WHAT WOULD

    BE NICE WHAT WOULD BE NICE 26 . 1

27. 2/1/2019 The (long) road to Kubernetes http://localhost:8000/pres.html?print-pdf#/sec-title-slide 27/67 WHAT WOULD

    BE NICE WHAT WOULD BE NICE 27 . 1

28. 2/1/2019 The (long) road to Kubernetes http://localhost:8000/pres.html?print-pdf#/sec-title-slide 28/67 WHERE TO

    FROM HERE WHERE TO FROM HERE How do you get out of the business of shuffling configuration and apps around? 28 . 1

29. 2/1/2019 The (long) road to Kubernetes http://localhost:8000/pres.html?print-pdf#/sec-title-slide 29/67 ALTERNATIVES CONSIDERED

    ALTERNATIVES CONSIDERED Mesos Docker LXD Kubernetes 29 . 1

30. 2/1/2019 The (long) road to Kubernetes http://localhost:8000/pres.html?print-pdf#/sec-title-slide 30/67 ALTERNATIVES CONSIDERED

    ALTERNATIVES CONSIDERED Mesos Docker LXD Kubernetes 30 . 1

31. 2/1/2019 The (long) road to Kubernetes http://localhost:8000/pres.html?print-pdf#/sec-title-slide 31/67 KUBERNETES PROMISES

    KUBERNETES PROMISES Utilization optimization Reducing the pain of adding new components Command and control Preserves a service boundaries 31 . 1

32. 2/1/2019 The (long) road to Kubernetes http://localhost:8000/pres.html?print-pdf#/sec-title-slide 32/67 ADDITIONAL PROMISES

    ADDITIONAL PROMISES A good substrate for creating resources on- demand Lingua-franca for infrastructure concepts Initial learning-curve Most likely smaller than ad-hoc solutions Eating our own dog food 32 . 1

33. 2/1/2019 The (long) road to Kubernetes http://localhost:8000/pres.html?print-pdf#/sec-title-slide 33/67 LET'S INSTALL

    KUBERNETES AND GO LET'S INSTALL KUBERNETES AND GO HOME? HOME? 33 . 1

34. 2/1/2019 The (long) road to Kubernetes http://localhost:8000/pres.html?print-pdf#/sec-title-slide 34/67 REMAINING WORRIES

    REMAINING WORRIES Security Monitoring Deployment Process Networking 34 . 1

35. 2/1/2019 The (long) road to Kubernetes http://localhost:8000/pres.html?print-pdf#/sec-title-slide 35/67 DEPLOYMENT PROCESS

    DEPLOYMENT PROCESS Reproducibility Traceability Security Checkpoints 35 . 1

36. 2/1/2019 The (long) road to Kubernetes http://localhost:8000/pres.html?print-pdf#/sec-title-slide 36/67 REPRODUCIBILITY REPRODUCIBILITY

    Building once (and in chroots) ensures clean packages Reproducible builds make wide changes easier We need staging deploys and production deploys to be identical 36 . 1

37. 2/1/2019 The (long) road to Kubernetes http://localhost:8000/pres.html?print-pdf#/sec-title-slide 37/67 TRACEABILITY TRACEABILITY

    When did we last build this? What did the output look like? What commit did it correspond to? 37 . 1

38. 2/1/2019 The (long) road to Kubernetes http://localhost:8000/pres.html?print-pdf#/sec-title-slide 38/67 SECURITY SECURITY

    No code download on production hosts Signed packages 38 . 1

39. 2/1/2019 The (long) road to Kubernetes http://localhost:8000/pres.html?print-pdf#/sec-title-slide 39/67 CHECKPOINTS CHECKPOINTS

    CD is great for test and staging We are wary of unattended production deploys There should be a clear (but simple) trigger 39 . 1

40. 2/1/2019 The (long) road to Kubernetes http://localhost:8000/pres.html?print-pdf#/sec-title-slide 40/67 BEFORE KUBERNETES

    BEFORE KUBERNETES 40 . 1

41. 2/1/2019 The (long) road to Kubernetes http://localhost:8000/pres.html?print-pdf#/sec-title-slide 41/67 BEFORE KUBERNETES

    BEFORE KUBERNETES 41 . 1

42. 2/1/2019 The (long) road to Kubernetes http://localhost:8000/pres.html?print-pdf#/sec-title-slide 42/67 BEFORE KUBERNETES

    BEFORE KUBERNETES 42 . 1

43. 2/1/2019 The (long) road to Kubernetes http://localhost:8000/pres.html?print-pdf#/sec-title-slide 43/67 BEFORE KUBERNETES

    BEFORE KUBERNETES 43 . 1

44. 2/1/2019 The (long) road to Kubernetes http://localhost:8000/pres.html?print-pdf#/sec-title-slide 44/67 BEFORE KUBERNETES

    BEFORE KUBERNETES 44 . 1

45. 2/1/2019 The (long) road to Kubernetes http://localhost:8000/pres.html?print-pdf#/sec-title-slide 45/67 BEFORE KUBERNETES

    BEFORE KUBERNETES 45 . 1

46. 2/1/2019 The (long) road to Kubernetes http://localhost:8000/pres.html?print-pdf#/sec-title-slide 46/67 BEFORE KUBERNETES

    BEFORE KUBERNETES 46 . 1

47. 2/1/2019 The (long) road to Kubernetes http://localhost:8000/pres.html?print-pdf#/sec-title-slide 47/67 AFTER KUBERNETES

    AFTER KUBERNETES 47 . 1

48. 2/1/2019 The (long) road to Kubernetes http://localhost:8000/pres.html?print-pdf#/sec-title-slide 48/67 AFTER KUBERNETES

    AFTER KUBERNETES Reproducibility ✔ Traceability ✔ Security ✔ Checkpoints ✔ 48 . 1

49. 2/1/2019 The (long) road to Kubernetes http://localhost:8000/pres.html?print-pdf#/sec-title-slide 49/67 AFTER KUBERNETES:

    PACKAGING AFTER KUBERNETES: PACKAGING Building is faster, easier, and gives developers more autonomy Docker registries forced us to move from enforcement to convention 49 . 1

50. 2/1/2019 The (long) road to Kubernetes http://localhost:8000/pres.html?print-pdf#/sec-title-slide 50/67 AFTER KUBERNETES:

    CONFIGURATION AFTER KUBERNETES: CONFIGURATION The split across environment services, variables, configmaps, and secrets makes separation easy Greatly reduces the need for config management Configuration can be colocated with the so ware Removes the code, build, and configuration-management impedance mismatch Kept things simple No helm 50 . 1

51. 2/1/2019 The (long) road to Kubernetes http://localhost:8000/pres.html?print-pdf#/sec-title-slide 51/67 AFTER KUBERNETES:

    ON-DEMAND AFTER KUBERNETES: ON-DEMAND RESOURCES RESOURCES CRDs provide great integration High cardinality or complex queries can be tedious to work with 51 . 1

52. 2/1/2019 The (long) road to Kubernetes http://localhost:8000/pres.html?print-pdf#/sec-title-slide 52/67 AFTER KUBERNETES:

    SECURITY AFTER KUBERNETES: SECURITY RBAC policies are powerful but tedious to write (and error prone) Certificate management leaves a lot to be improved 52 . 1

53. 2/1/2019 The (long) road to Kubernetes http://localhost:8000/pres.html?print-pdf#/sec-title-slide 53/67 GOING FORWARD

    GOING FORWARD Going back from convention to enforcement for the registry Providing more Paas-like encoding of our common cases 53 . 1

54. 2/1/2019 The (long) road to Kubernetes http://localhost:8000/pres.html?print-pdf#/sec-title-slide 54/67 NETWORKING NETWORKING

    Security Scalability 54 . 1

55. 2/1/2019 The (long) road to Kubernetes http://localhost:8000/pres.html?print-pdf#/sec-title-slide 55/67 BEFORE KUBERNETES

    BEFORE KUBERNETES A public IP per VM Security groups for firewall management 55 . 1

56. 2/1/2019 The (long) road to Kubernetes http://localhost:8000/pres.html?print-pdf#/sec-title-slide 56/67 BEFORE KUBERNETES

    BEFORE KUBERNETES A boring, solid network 56 . 1

57. 2/1/2019 The (long) road to Kubernetes http://localhost:8000/pres.html?print-pdf#/sec-title-slide 57/67 KUBERNETES NETWORKING

    101 KUBERNETES NETWORKING 101 57 . 1

58. 2/1/2019 The (long) road to Kubernetes http://localhost:8000/pres.html?print-pdf#/sec-title-slide 58/67 KUBERNETES NETWORKING

    101 KUBERNETES NETWORKING 101 58 . 1

59. 2/1/2019 The (long) road to Kubernetes http://localhost:8000/pres.html?print-pdf#/sec-title-slide 59/67 KUBERNETES NETWORKING

    101 KUBERNETES NETWORKING 101 59 . 1

60. 2/1/2019 The (long) road to Kubernetes http://localhost:8000/pres.html?print-pdf#/sec-title-slide 60/67 KUBERNETES NETWORKING

    101 KUBERNETES NETWORKING 101 A boring, solid network 60 . 1

61. 2/1/2019 The (long) road to Kubernetes http://localhost:8000/pres.html?print-pdf#/sec-title-slide 61/67 WHAT ABOUT

    EXTERNAL SERVICES WHAT ABOUT EXTERNAL SERVICES By default NodePort services run on all worker nodes Traffic is source-nat'd to the destination Losing source IP information is unviable for most use- cases Performance impact 61 . 1

62. 2/1/2019 The (long) road to Kubernetes http://localhost:8000/pres.html?print-pdf#/sec-title-slide 62/67 NETWORKING AT

    EXOSCALE NETWORKING AT EXOSCALE A layer3 all-the-way design A BGP first design VM Public IPs advertised by BGP from hypervisors Private network VXLAN membership advertised through BGP- eVPN Best performance is on the public interfaces 62 . 1

63. 2/1/2019 The (long) road to Kubernetes http://localhost:8000/pres.html?print-pdf#/sec-title-slide 63/67 NETWORKING: AFTER

    KUBERNETES NETWORKING: AFTER KUBERNETES Security ✔ Scalability ✔ 63 . 1

64. 2/1/2019 The (long) road to Kubernetes http://localhost:8000/pres.html?print-pdf#/sec-title-slide 64/67 IN-CLUSTER NETWORKING

    IN-CLUSTER NETWORKING Kept with a BGP-based CNI Avoids additional encapsulation 64 . 1

65. 2/1/2019 The (long) road to Kubernetes http://localhost:8000/pres.html?print-pdf#/sec-title-slide 65/67 EXTERNAL SERVICES

    EXTERNAL SERVICES Needs additional development We went for IPIP load-balancing Needs node-local decapsulation Makes for a hacky setup eBPF/XDP to the rescue! 65 . 1

66. 2/1/2019 The (long) road to Kubernetes http://localhost:8000/pres.html?print-pdf#/sec-title-slide 66/67 KUBERNETES AT

    EXOSCALE TODAY KUBERNETES AT EXOSCALE TODAY Most production critical services Some things we are in no hurry to containerize :-) Basis for as-a-service offerings Private network management For customers exo lab kube Cluster API OpenShi On-demand cluster in the Exoscale API 66 . 1

67. 2/1/2019 The (long) road to Kubernetes http://localhost:8000/pres.html?print-pdf#/sec-title-slide 67/67 QUESTIONS? QUESTIONS?

    We're productizing all of this as we speak! 67 . 1