Highly available cross-region deployments with Kubernetes

Out-of-the-box Kubernetes makes it easy to deploy and scale your applications within one Kubernetes cluster in one single region. But with the Kubernetes Federation API and the use of service meshes like Istio, it’s also possible to deploy an application over multiple clusters in different regions, so that it becomes truly highly available even if a complete region fails.

Bastian Hofmann explains how to set the Kubernetes cluster federation up and how to deploy your applications to make best use of it, especially regarding monitoring, persistent storage, latencies, and data integrity.

Transcript

1. @BastianHofmann
   Highly available
   cross-region deployments
   with Kubernetes
   Bastian Hofmann
   

   View Slide

2. View Slide

3. Container orchestration platform
   

   View Slide

4. Deploy, run and scale your services
   in isolated containers
   

   View Slide

5. No vendor lock in
   

   View Slide

6. Standardized APIs
   

   View Slide

7. Runs on
   

   View Slide

8. Your laptop
   

   View Slide

9. Bare metal
   

   View Slide

10. Cloud Providers
    

    View Slide

11. And if you don't want to install and
    maintain Kubernetes yourself
    

    View Slide

12. Managed Kubernetes
    

    View Slide

13. View Slide

14. Self-healing built in
    

    View Slide

15. But what happens when a complete
    datacenter is not available
    

    View Slide

16. Multi-Region Kubernetes Setups
    

    View Slide

17. Why
    

    View Slide

18. Availability
    

    View Slide

19. Scalability
    

    View Slide

20. More Points-of-Presence
    

    View Slide

21. Reduce dependencies on one single
    cloud provider
    

    View Slide

22. Because of the standardised API
    across providers Kubernetes can
    help
    

    View Slide

23. Features
    

    View Slide

24. Scheduling by Node Labels
    

    View Slide

25. failure-
    domain.beta.kubernetes.io/
    region=dbl
    failure-
    domain.beta.kubernetes.io/
    zone=dbl1
    

    View Slide

26. Regions vs availability zone
    

    View Slide

27. failure-
    domain.beta.kubernetes.io/
    region=dbl
    failure-
    domain.beta.kubernetes.io/
    zone=dbl1
    

    View Slide

28. apiVersion: apps/v1
    kind: Deployment
    metadata:
    name: nginx
    spec:
    template:
    spec:
    containers:
    - image: nginx
    name: nginx
    nodeSelector:
    failure-domain.beta.kubernetes.io/region: dbl
    

    View Slide

29. Affinities
    

    View Slide

30. spec:
    affinity:
    podAntiAffinity:
    requiredDuringSchedulingIgnoredDuringExecution:
    - topologyKey: "failure-domain.beta.kubernetes.io/
    region"
    labelSelector:
    matchLabels:
    app: nginx
    

    View Slide

31. Service discovery with built in DNS
    

    View Slide

32. apiVersion: v1
    kind: Service
    metadata:
    name: nginx
    spec:
    type: ClusterIP
    ports:
    - port: 80
    targetPort: 80
    selector:
    app: nginx
    

    View Slide

33. External LoadBalancers
    

    View Slide

34. apiVersion: v1
    kind: Service
    metadata:
    name: nginx
    spec:
    type: LoadBalancer
    ports:
    - port: 80
    targetPort: 80
    selector:
    app: nginx
    

    View Slide

35. external-dns
    

    View Slide

36. StorageClasses
    

    View Slide

37. Some storage providers support
    dynamic volume provisioning
    

    View Slide

38. apiVersion: storage.k8s.io/v1
    kind: StorageClass
    metadata:
    name: standard
    provisioner: kubernetes.io/gce-pd
    parameters:
    type: pd-standard
    volumeBindingMode: WaitForFirstConsumer
    allowedTopologies:
    - matchLabelExpressions:
    - key: failure-domain.beta.kubernetes.io/zone
    values:
    - us-central1-a
    - us-central1-b
    

    View Slide

39. Challenges
    

    View Slide

40. Especially over different regions
    

    View Slide

41. External load balancing and traffic
    management
    

    View Slide

42. Internal load balancing
    

    View Slide

43. Connectivity between private
    networks
    

    View Slide

44. Kubernetes DNS across multiple
    clusters
    

    View Slide

45. Latencies
    

    View Slide

46. Beskow, Paul & Vik, Knut-Helge & Halvorsen, Pål & Griwodz, Carsten. (2009).
    The partial migration of game state and dynamic server selection to reduce latency. Multimedia Tools Appl..
    45. 83-107. 10.1007/s11042-009-0287-7.
    

    View Slide

47. Synchronising Deployments across
    multiple clusters
    

    View Slide

48. Storage
    

    View Slide

49. Data replication
    

    View Slide

50. Split-brain problem
    

    View Slide

51. Possible setups
    

    View Slide

52. One cluster across AZs in one region
    

    View Slide

53. One cluster across regions
    

    View Slide

54. Multiple clusters connected via VPN
    

    View Slide

55. One cluster setup
    

    View Slide

56. One Kubernetes cluster across
    multiple availability zones
    

    View Slide

57. Zone 1 Zone 2
    Kubernetes Master Components
    Nginx
    Nginx
    Nginx Service
    Curl
    

    View Slide

58. Easiest option
    

    View Slide

59. Low latencies
    

    View Slide

60. You can use one network
    

    View Slide

61. All pods and services can talk with
    each other
    

    View Slide

62. One cluster => one internal DNS
    

    View Slide

63. Service Discovery and internal load
    balancing works
    

    View Slide

64. External LoadBalancers work
    

    View Slide

65. Zone 1 Zone 2
    Kubernetes Master Components
    Nginx
    Nginx
    Nginx Service
    Curl
    LB
    

    View Slide

66. You can mount storage across AZ
    

    View Slide

67. One cluster state => One deployment
    

    View Slide

68. But what if the whole region is
    down?
    

    View Slide

69. One Kubernetes cluster across
    multiple regions and VPN connection
    between networks
    

    View Slide

70. Region 1 Region 2
    WireGuard VPN
    Kubernetes Master Components
    Nginx
    Nginx
    Nginx Service
    Curl
    

    View Slide

71. Latencies get higher by distance
    

    View Slide

72. You need a VPN to connect the
    networks
    

    View Slide

73. WireGuard as a VPN
    

    View Slide

74. https:/
    /github.com/squat/kilo
    

    View Slide

75. Kilo gets all Nodes from Kubernetes
    

    View Slide

76. Discovers regions and zones by
    Kubernetes Node labels
    

    View Slide

77. Configures WireGuard
    

    View Slide

78. Routing tables
    

    View Slide

79. All pods and services can talk with
    each other
    

    View Slide

80. One cluster => one internal DNS
    

    View Slide

81. Service Discovery and internal load
    balancing works
    

    View Slide

82. You need a solution for external load
    balancing
    

    View Slide

83. Region 1 Region 2
    WireGuard VPN
    Kubernetes Master Components
    Nginx
    Nginx
    Nginx Service
    Curl
    LB LB
    

    View Slide

84. You have to replicate storage
    yourself
    

    View Slide

85. Not every Storage provider supports
    dynamic volume provisioning
    

    View Slide

86. One cluster state => One deployment
    

    View Slide

87. Demo
    

    View Slide

88. Multi cluster setup
    

    View Slide

89. Connecting multiple clusters with a
    VPN
    

    View Slide

90. Cluster 1 Cluster 2
    WireGuard VPN
    Kubernetes Master
    Components
    Kubernetes Master
    Components
    Nginx
    Nginx
    Nginx Service Nginx Service
    Curl
    

    View Slide

91. WireGuard as a VPN
    

    View Slide

92. https:/
    /github.com/squat/kilo
    

    View Slide

93. All pods and services can talk with
    each other
    

    View Slide

94. Separate clusters => separate
    internal DNS
    

    View Slide

95. For service discovery configure each
    internal DNS to resolve to other
    clusters
    

    View Slide

96. cluster.region2:53 {
    forward . 10.10.11.10
    }
    .:53 {
    kubernetes cluster.local cluster.region1 in-addr.arpa
    ip6.arpa {
    pods insecure
    upstream
    fallthrough in-addr.arpa ip6.arpa
    }
    forward . /etc/resolv.conf
    loop
    loadbalance
    }
    

    View Slide

97. cluster.region1:53 {
    forward . 10.10.10.10
    }
    .:53 {
    kubernetes cluster.local cluster.region2 in-addr.arpa
    ip6.arpa {
    pods insecure
    upstream
    fallthrough in-addr.arpa ip6.arpa
    }
    forward . /etc/resolv.conf
    loop
    loadbalance
    }
    

    View Slide

98. Internal load balancing works
    

    View Slide

99. You have to replicate storage
    yourself
    

    View Slide

100. Every cluster has their own
     StorageClass that works on all nodes
     

     View Slide

101. Separate clusters have separate
     state
     

     View Slide

102. Management of deployments across
     clusters
     

     View Slide

103. Kubefed
     https://github.com/kubernetes-sigs/kubefed
     

     View Slide

104. FederatedNamespaces,
     FederatedDeployments,
     FederatedConfigMaps,
     FederatedServices,
     ...
     

     View Slide

105. Cluster aware controller that
     manages resources in all connected
     clusters
     

     View Slide

106. Cluster 1 Cluster 2
     Kubernetes Master
     Components
     Kubernetes Master
     Components
     Kubefed Controller
     

     View Slide

107. Cluster 1 Cluster 2
     Kubernetes Master
     Components
     Kubernetes Master
     Components
     Kubefed Controller
     FederatedService
     FederatedDeployment
     

     View Slide

108. Cluster 1 Cluster 2
     Kubernetes Master
     Components
     Kubernetes Master
     Components
     Nginx Service Nginx Service
     Kubefed Controller Kubefed Controller
     Nginx Nginx
     

     View Slide

109. Demo
     

     View Slide

110. Solution for external traffic
     

     View Slide

111. Cluster 1 Cluster 2
     WireGuard VPN
     Kubernetes Master
     Components
     Kubernetes Master
     Components
     Nginx
     Nginx
     Nginx Service Nginx Service
     Curl
     LB LB
     

     View Slide

112. Multi cluster external-dns
     

     View Slide

113. View Slide

114. service.namespace.domain.svc.example.com
     service.namespace.domain.svc.region1.example.com
     service.namespace.domain.svc.region2.example.com
     

     View Slide

115. Demo
     

     View Slide

116. More options
     

     View Slide

117. Multiple clusters connected via
     Service Mesh (Istio)
     

     View Slide

118. Conclusion
     

     View Slide

119. Kubernetes makes it easier to create
     multi region setups
     

     View Slide

120. There are still challenges you have to
     overcome
     

     View Slide

121. Often just physics
     

     View Slide

122. Federation Tooling is just getting
     started
     

     View Slide

123. Test it
     30 days
     For free
     Visit us at our
     booth
     

     View Slide

124. View Slide

125. https:/
     /github.com/bashofmann/
     kubernetes-multicluster-demos
     

     View Slide

126. mail@bastianhofmann.de
     https:/
     /twitter.com/BastianHofmann
     

     View Slide

127. Connecting multiple clusters with a
     Service Mesh Gateway
     

     View Slide

128. View Slide

129. View Slide

130. Cluster 1 Cluster 2
     Istio Gateway
     Istio Control Plane
     Kubernetes Master
     Components
     Kubernetes Master
     Components
     Istio
     Nginx
     Nginx
     Nginx Service Nginx Service
     Istio Gateway
     Curl
     

     View Slide

131. No VPN necessary
     

     View Slide

132. Fairly easy to set up
     

     View Slide

133. Pods from different clusters
     communicate over public Ips
     

     View Slide

134. Traffic encrypted and authenticated
     with mutual TLS
     

     View Slide

135. Communication is only possible
     through Istio proxies
     

     View Slide

136. Easy service discovery
     

     View Slide

137. Flexible, location aware traffic
     management
     

     View Slide

138. Telemetry and tracing included
     

     View Slide

139. Demo
     

     View Slide