Highly available cross-region deployments with Kubernetes

Out-of-the-box Kubernetes makes it easy to deploy and scale your applications within one Kubernetes cluster in one single region. But with the Kubernetes Federation API and the use of service meshes like Istio, it’s also possible to deploy an application over multiple clusters in different regions, so that it becomes truly highly available even if a complete region fails.

Bastian Hofmann explains how to set the Kubernetes cluster federation up and how to deploy your applications to make best use of it, especially regarding monitoring, persistent storage, latencies, and data integrity.

Transcript

1. @BastianHofmann Highly available cross-region deployments with Kubernetes Bastian Hofmann
3. Container orchestration platform
4. Deploy, run and scale your services in isolated containers
5. No vendor lock in
6. Standardized APIs
7. Runs on
8. Your laptop
9. Bare metal
10. Cloud Providers
11. And if you don't want to install and maintain Kubernetes yourself
12. Managed Kubernetes
14. Self-healing built in
15. But what happens when a complete datacenter is not available
16. Multi-Region Kubernetes Setups
17. Why
18. Availability
19. Scalability
20. More Points-of-Presence
21. Reduce dependencies on one single cloud provider
22. Because of the standardised API across providers Kubernetes can help
23. Features
24. Scheduling by Node Labels
25. failure- domain.beta.kubernetes.io/ region=dbl failure- domain.beta.kubernetes.io/ zone=dbl1
26. Regions vs availability zone
27. failure- domain.beta.kubernetes.io/ region=dbl failure- domain.beta.kubernetes.io/ zone=dbl1
28. apiVersion: apps/v1 kind: Deployment metadata: name: nginx spec: template: spec: containers: - image: nginx name: nginx nodeSelector: failure-domain.beta.kubernetes.io/region: dbl
29. Affinities
30. spec: affinity: podAntiAffinity: requiredDuringSchedulingIgnoredDuringExecution: - topologyKey: "failure-domain.beta.kubernetes.io/ region" labelSelector: matchLabels: app: nginx
31. Service discovery with built in DNS
32. apiVersion: v1 kind: Service metadata: name: nginx spec: type: ClusterIP ports: - port: 80 targetPort: 80 selector: app: nginx
33. External LoadBalancers
34. apiVersion: v1 kind: Service metadata: name: nginx spec: type: LoadBalancer ports: - port: 80 targetPort: 80 selector: app: nginx
35. external-dns
36. StorageClasses
37. Some storage providers support dynamic volume provisioning
38. apiVersion: storage.k8s.io/v1 kind: StorageClass metadata: name: standard provisioner: kubernetes.io/gce-pd parameters: type: pd-standard volumeBindingMode: WaitForFirstConsumer allowedTopologies: - matchLabelExpressions: - key: failure-domain.beta.kubernetes.io/zone values: - us-central1-a - us-central1-b
39. Challenges
40. Especially over different regions
41. External load balancing and traffic management
42. Internal load balancing
43. Connectivity between private networks
44. Kubernetes DNS across multiple clusters
45. Latencies
46. Beskow, Paul & Vik, Knut-Helge & Halvorsen, Pål & Griwodz, Carsten. (2009). The partial migration of game state and dynamic server selection to reduce latency. Multimedia Tools Appl.. 45. 83-107. 10.1007/s11042-009-0287-7.
47. Synchronising Deployments across multiple clusters
48. Storage
49. Data replication
50. Split-brain problem
51. Possible setups
52. One cluster across AZs in one region
53. One cluster across regions
54. Multiple clusters connected via VPN
55. One cluster setup
56. One Kubernetes cluster across multiple availability zones
57. Zone 1 Zone 2 Kubernetes Master Components Nginx Nginx Nginx Service Curl
58. Easiest option
59. Low latencies
60. You can use one network
61. All pods and services can talk with each other
62. One cluster => one internal DNS
63. Service Discovery and internal load balancing works
64. External LoadBalancers work
65. Zone 1 Zone 2 Kubernetes Master Components Nginx Nginx Nginx Service Curl LB
66. You can mount storage across AZ
67. One cluster state => One deployment
68. But what if the whole region is down?
69. One Kubernetes cluster across multiple regions and VPN connection between networks
70. Region 1 Region 2 WireGuard VPN Kubernetes Master Components Nginx Nginx Nginx Service Curl
71. Latencies get higher by distance
72. You need a VPN to connect the networks
73. WireGuard as a VPN
74. https:/ /github.com/squat/kilo
75. Kilo gets all Nodes from Kubernetes
76. Discovers regions and zones by Kubernetes Node labels
77. Configures WireGuard
78. Routing tables
79. All pods and services can talk with each other
80. One cluster => one internal DNS
81. Service Discovery and internal load balancing works
82. You need a solution for external load balancing
83. Region 1 Region 2 WireGuard VPN Kubernetes Master Components Nginx Nginx Nginx Service Curl LB LB
84. You have to replicate storage yourself
85. Not every Storage provider supports dynamic volume provisioning
86. One cluster state => One deployment
87. Demo
88. Multi cluster setup
89. Connecting multiple clusters with a VPN
90. Cluster 1 Cluster 2 WireGuard VPN Kubernetes Master Components Kubernetes Master Components Nginx Nginx Nginx Service Nginx Service Curl
91. WireGuard as a VPN
92. https:/ /github.com/squat/kilo
93. All pods and services can talk with each other
94. Separate clusters => separate internal DNS
95. For service discovery configure each internal DNS to resolve to other clusters
96. cluster.region2:53 { forward . 10.10.11.10 } .:53 { kubernetes cluster.local cluster.region1 in-addr.arpa ip6.arpa { pods insecure upstream fallthrough in-addr.arpa ip6.arpa } forward . /etc/resolv.conf loop loadbalance }
97. cluster.region1:53 { forward . 10.10.10.10 } .:53 { kubernetes cluster.local cluster.region2 in-addr.arpa ip6.arpa { pods insecure upstream fallthrough in-addr.arpa ip6.arpa } forward . /etc/resolv.conf loop loadbalance }
98. Internal load balancing works
99. You have to replicate storage yourself
100. Every cluster has their own StorageClass that works on all nodes
101. Separate clusters have separate state
102. Management of deployments across clusters
103. Kubefed https://github.com/kubernetes-sigs/kubefed
104. FederatedNamespaces, FederatedDeployments, FederatedConfigMaps, FederatedServices, ...
105. Cluster aware controller that manages resources in all connected clusters
106. Cluster 1 Cluster 2 Kubernetes Master Components Kubernetes Master Components Kubefed Controller
107. Cluster 1 Cluster 2 Kubernetes Master Components Kubernetes Master Components Kubefed Controller FederatedService FederatedDeployment
108. Cluster 1 Cluster 2 Kubernetes Master Components Kubernetes Master Components Nginx Service Nginx Service Kubefed Controller Kubefed Controller Nginx Nginx
109. Demo
110. Solution for external traffic
111. Cluster 1 Cluster 2 WireGuard VPN Kubernetes Master Components Kubernetes Master Components Nginx Nginx Nginx Service Nginx Service Curl LB LB
112. Multi cluster external-dns
114. service.namespace.domain.svc.example.com service.namespace.domain.svc.region1.example.com service.namespace.domain.svc.region2.example.com
115. Demo
116. More options
117. Multiple clusters connected via Service Mesh (Istio)
118. Conclusion
119. Kubernetes makes it easier to create multi region setups
120. There are still challenges you have to overcome
121. Often just physics
122. Federation Tooling is just getting started
123. Test it 30 days For free Visit us at our booth
125. https:/ /github.com/bashofmann/ kubernetes-multicluster-demos
126. mail@bastianhofmann.de https:/ /twitter.com/BastianHofmann
127. Connecting multiple clusters with a Service Mesh Gateway
130. Cluster 1 Cluster 2 Istio Gateway Istio Control Plane Kubernetes Master Components Kubernetes Master Components Istio Nginx Nginx Nginx Service Nginx Service Istio Gateway Curl
131. No VPN necessary
132. Fairly easy to set up
133. Pods from different clusters communicate over public Ips
134. Traffic encrypted and authenticated with mutual TLS
135. Communication is only possible through Istio proxies
136. Easy service discovery
137. Flexible, location aware traffic management
138. Telemetry and tracing included
139. Demo