Highly available cross-region deployments with Kubernetes

Highly available cross-region deployments with Kubernetes

Out-of-the-box Kubernetes makes it easy to deploy and scale your applications within one Kubernetes cluster in one single region. But with the Kubernetes Federation API and the use of service meshes like Istio, it’s also possible to deploy an application over multiple clusters in different regions, so that it becomes truly highly available even if a complete region fails.

Bastian Hofmann explains how to set the Kubernetes cluster federation up and how to deploy your applications to make best use of it, especially regarding monitoring, persistent storage, latencies, and data integrity.

Ded87c77266697ee6981c2277bb97633?s=128

Bastian Hofmann

November 06, 2019
Tweet

Transcript

  1. @BastianHofmann Highly available cross-region deployments with Kubernetes Bastian Hofmann

  2. None
  3. Container orchestration platform

  4. Deploy, run and scale your services in isolated containers

  5. No vendor lock in

  6. Standardized APIs

  7. Runs on

  8. Your laptop

  9. Bare metal

  10. Cloud Providers

  11. And if you don't want to install and maintain Kubernetes

    yourself
  12. Managed Kubernetes

  13. None
  14. Self-healing built in

  15. But what happens when a complete datacenter is not available

  16. Multi-Region Kubernetes Setups

  17. Why

  18. Availability

  19. Scalability

  20. More Points-of-Presence

  21. Reduce dependencies on one single cloud provider

  22. Because of the standardised API across providers Kubernetes can help

  23. Features

  24. Scheduling by Node Labels

  25. failure- domain.beta.kubernetes.io/ region=dbl failure- domain.beta.kubernetes.io/ zone=dbl1

  26. Regions vs availability zone

  27. failure- domain.beta.kubernetes.io/ region=dbl failure- domain.beta.kubernetes.io/ zone=dbl1

  28. apiVersion: apps/v1 kind: Deployment metadata: name: nginx spec: template: spec:

    containers: - image: nginx name: nginx nodeSelector: failure-domain.beta.kubernetes.io/region: dbl
  29. Affinities

  30. spec: affinity: podAntiAffinity: requiredDuringSchedulingIgnoredDuringExecution: - topologyKey: "failure-domain.beta.kubernetes.io/ region" labelSelector: matchLabels:

    app: nginx
  31. Service discovery with built in DNS

  32. apiVersion: v1 kind: Service metadata: name: nginx spec: type: ClusterIP

    ports: - port: 80 targetPort: 80 selector: app: nginx
  33. External LoadBalancers

  34. apiVersion: v1 kind: Service metadata: name: nginx spec: type: LoadBalancer

    ports: - port: 80 targetPort: 80 selector: app: nginx
  35. external-dns

  36. StorageClasses

  37. Some storage providers support dynamic volume provisioning

  38. apiVersion: storage.k8s.io/v1 kind: StorageClass metadata: name: standard provisioner: kubernetes.io/gce-pd parameters:

    type: pd-standard volumeBindingMode: WaitForFirstConsumer allowedTopologies: - matchLabelExpressions: - key: failure-domain.beta.kubernetes.io/zone values: - us-central1-a - us-central1-b
  39. Challenges

  40. Especially over different regions

  41. External load balancing and traffic management

  42. Internal load balancing

  43. Connectivity between private networks

  44. Kubernetes DNS across multiple clusters

  45. Latencies

  46. Beskow, Paul & Vik, Knut-Helge & Halvorsen, Pål & Griwodz,

    Carsten. (2009). The partial migration of game state and dynamic server selection to reduce latency. Multimedia Tools Appl.. 45. 83-107. 10.1007/s11042-009-0287-7.
  47. Synchronising Deployments across multiple clusters

  48. Storage

  49. Data replication

  50. Split-brain problem

  51. Possible setups

  52. One cluster across AZs in one region

  53. One cluster across regions

  54. Multiple clusters connected via VPN

  55. One cluster setup

  56. One Kubernetes cluster across multiple availability zones

  57. Zone 1 Zone 2 Kubernetes Master Components Nginx Nginx Nginx

    Service Curl
  58. Easiest option

  59. Low latencies

  60. You can use one network

  61. All pods and services can talk with each other

  62. One cluster => one internal DNS

  63. Service Discovery and internal load balancing works

  64. External LoadBalancers work

  65. Zone 1 Zone 2 Kubernetes Master Components Nginx Nginx Nginx

    Service Curl LB
  66. You can mount storage across AZ

  67. One cluster state => One deployment

  68. But what if the whole region is down?

  69. One Kubernetes cluster across multiple regions and VPN connection between

    networks
  70. Region 1 Region 2 WireGuard VPN Kubernetes Master Components Nginx

    Nginx Nginx Service Curl
  71. Latencies get higher by distance

  72. You need a VPN to connect the networks

  73. WireGuard as a VPN

  74. https:/ /github.com/squat/kilo

  75. Kilo gets all Nodes from Kubernetes

  76. Discovers regions and zones by Kubernetes Node labels

  77. Configures WireGuard

  78. Routing tables

  79. All pods and services can talk with each other

  80. One cluster => one internal DNS

  81. Service Discovery and internal load balancing works

  82. You need a solution for external load balancing

  83. Region 1 Region 2 WireGuard VPN Kubernetes Master Components Nginx

    Nginx Nginx Service Curl LB LB
  84. You have to replicate storage yourself

  85. Not every Storage provider supports dynamic volume provisioning

  86. One cluster state => One deployment

  87. Demo

  88. Multi cluster setup

  89. Connecting multiple clusters with a VPN

  90. Cluster 1 Cluster 2 WireGuard VPN Kubernetes Master Components Kubernetes

    Master Components Nginx Nginx Nginx Service Nginx Service Curl
  91. WireGuard as a VPN

  92. https:/ /github.com/squat/kilo

  93. All pods and services can talk with each other

  94. Separate clusters => separate internal DNS

  95. For service discovery configure each internal DNS to resolve to

    other clusters
  96. cluster.region2:53 { forward . 10.10.11.10 } .:53 { kubernetes cluster.local

    cluster.region1 in-addr.arpa ip6.arpa { pods insecure upstream fallthrough in-addr.arpa ip6.arpa } forward . /etc/resolv.conf loop loadbalance }
  97. cluster.region1:53 { forward . 10.10.10.10 } .:53 { kubernetes cluster.local

    cluster.region2 in-addr.arpa ip6.arpa { pods insecure upstream fallthrough in-addr.arpa ip6.arpa } forward . /etc/resolv.conf loop loadbalance }
  98. Internal load balancing works

  99. You have to replicate storage yourself

  100. Every cluster has their own StorageClass that works on all

    nodes
  101. Separate clusters have separate state

  102. Management of deployments across clusters

  103. Kubefed https://github.com/kubernetes-sigs/kubefed

  104. FederatedNamespaces, FederatedDeployments, FederatedConfigMaps, FederatedServices, ...

  105. Cluster aware controller that manages resources in all connected clusters

  106. Cluster 1 Cluster 2 Kubernetes Master Components Kubernetes Master Components

    Kubefed Controller
  107. Cluster 1 Cluster 2 Kubernetes Master Components Kubernetes Master Components

    Kubefed Controller FederatedService FederatedDeployment
  108. Cluster 1 Cluster 2 Kubernetes Master Components Kubernetes Master Components

    Nginx Service Nginx Service Kubefed Controller Kubefed Controller Nginx Nginx
  109. Demo

  110. Solution for external traffic

  111. Cluster 1 Cluster 2 WireGuard VPN Kubernetes Master Components Kubernetes

    Master Components Nginx Nginx Nginx Service Nginx Service Curl LB LB
  112. Multi cluster external-dns

  113. None
  114. service.namespace.domain.svc.example.com service.namespace.domain.svc.region1.example.com service.namespace.domain.svc.region2.example.com

  115. Demo

  116. More options

  117. Multiple clusters connected via Service Mesh (Istio)

  118. Conclusion

  119. Kubernetes makes it easier to create multi region setups

  120. There are still challenges you have to overcome

  121. Often just physics

  122. Federation Tooling is just getting started

  123. Test it 30 days For free Visit us at our

    booth
  124. None
  125. https:/ /github.com/bashofmann/ kubernetes-multicluster-demos

  126. mail@bastianhofmann.de https:/ /twitter.com/BastianHofmann

  127. Connecting multiple clusters with a Service Mesh Gateway

  128. None
  129. None
  130. Cluster 1 Cluster 2 Istio Gateway Istio Control Plane Kubernetes

    Master Components Kubernetes Master Components Istio Nginx Nginx Nginx Service Nginx Service Istio Gateway Curl
  131. No VPN necessary

  132. Fairly easy to set up

  133. Pods from different clusters communicate over public Ips

  134. Traffic encrypted and authenticated with mutual TLS

  135. Communication is only possible through Istio proxies

  136. Easy service discovery

  137. Flexible, location aware traffic management

  138. Telemetry and tracing included

  139. Demo