Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Highly available cross-region deployments with Kubernetes

Highly available cross-region deployments with Kubernetes

Out-of-the-box Kubernetes makes it easy to deploy and scale your applications within one Kubernetes cluster in one single region. But with the Kubernetes Federation API and the use of service meshes like Istio, it’s also possible to deploy an application over multiple clusters in different regions, so that it becomes truly highly available even if a complete region fails.

Bastian Hofmann explains how to set the Kubernetes cluster federation up and how to deploy your applications to make best use of it, especially regarding monitoring, persistent storage, latencies, and data integrity.

Bastian Hofmann

November 06, 2019
Tweet

More Decks by Bastian Hofmann

Other Decks in Programming

Transcript

  1. @BastianHofmann
    Highly available
    cross-region deployments
    with Kubernetes
    Bastian Hofmann

    View Slide

  2. View Slide

  3. Container orchestration platform

    View Slide

  4. Deploy, run and scale your services
    in isolated containers

    View Slide

  5. No vendor lock in

    View Slide

  6. Standardized APIs

    View Slide

  7. Runs on

    View Slide

  8. Your laptop

    View Slide

  9. Bare metal

    View Slide

  10. Cloud Providers

    View Slide

  11. And if you don't want to install and
    maintain Kubernetes yourself

    View Slide

  12. Managed Kubernetes

    View Slide

  13. View Slide

  14. Self-healing built in

    View Slide

  15. But what happens when a complete
    datacenter is not available

    View Slide

  16. Multi-Region Kubernetes Setups

    View Slide

  17. Why

    View Slide

  18. Availability

    View Slide

  19. Scalability

    View Slide

  20. More Points-of-Presence

    View Slide

  21. Reduce dependencies on one single
    cloud provider

    View Slide

  22. Because of the standardised API
    across providers Kubernetes can
    help

    View Slide

  23. Features

    View Slide

  24. Scheduling by Node Labels

    View Slide

  25. failure-
    domain.beta.kubernetes.io/
    region=dbl
    failure-
    domain.beta.kubernetes.io/
    zone=dbl1

    View Slide

  26. Regions vs availability zone

    View Slide

  27. failure-
    domain.beta.kubernetes.io/
    region=dbl
    failure-
    domain.beta.kubernetes.io/
    zone=dbl1

    View Slide

  28. apiVersion: apps/v1
    kind: Deployment
    metadata:
    name: nginx
    spec:
    template:
    spec:
    containers:
    - image: nginx
    name: nginx
    nodeSelector:
    failure-domain.beta.kubernetes.io/region: dbl

    View Slide

  29. Affinities

    View Slide

  30. spec:
    affinity:
    podAntiAffinity:
    requiredDuringSchedulingIgnoredDuringExecution:
    - topologyKey: "failure-domain.beta.kubernetes.io/
    region"
    labelSelector:
    matchLabels:
    app: nginx

    View Slide

  31. Service discovery with built in DNS

    View Slide

  32. apiVersion: v1
    kind: Service
    metadata:
    name: nginx
    spec:
    type: ClusterIP
    ports:
    - port: 80
    targetPort: 80
    selector:
    app: nginx

    View Slide

  33. External LoadBalancers

    View Slide

  34. apiVersion: v1
    kind: Service
    metadata:
    name: nginx
    spec:
    type: LoadBalancer
    ports:
    - port: 80
    targetPort: 80
    selector:
    app: nginx

    View Slide

  35. external-dns

    View Slide

  36. StorageClasses

    View Slide

  37. Some storage providers support
    dynamic volume provisioning

    View Slide

  38. apiVersion: storage.k8s.io/v1
    kind: StorageClass
    metadata:
    name: standard
    provisioner: kubernetes.io/gce-pd
    parameters:
    type: pd-standard
    volumeBindingMode: WaitForFirstConsumer
    allowedTopologies:
    - matchLabelExpressions:
    - key: failure-domain.beta.kubernetes.io/zone
    values:
    - us-central1-a
    - us-central1-b

    View Slide

  39. Challenges

    View Slide

  40. Especially over different regions

    View Slide

  41. External load balancing and traffic
    management

    View Slide

  42. Internal load balancing

    View Slide

  43. Connectivity between private
    networks

    View Slide

  44. Kubernetes DNS across multiple
    clusters

    View Slide

  45. Latencies

    View Slide

  46. Beskow, Paul & Vik, Knut-Helge & Halvorsen, Pål & Griwodz, Carsten. (2009).
    The partial migration of game state and dynamic server selection to reduce latency. Multimedia Tools Appl..
    45. 83-107. 10.1007/s11042-009-0287-7.

    View Slide

  47. Synchronising Deployments across
    multiple clusters

    View Slide

  48. Storage

    View Slide

  49. Data replication

    View Slide

  50. Split-brain problem

    View Slide

  51. Possible setups

    View Slide

  52. One cluster across AZs in one region

    View Slide

  53. One cluster across regions

    View Slide

  54. Multiple clusters connected via VPN

    View Slide

  55. One cluster setup

    View Slide

  56. One Kubernetes cluster across
    multiple availability zones

    View Slide

  57. Zone 1 Zone 2
    Kubernetes Master Components
    Nginx
    Nginx
    Nginx Service
    Curl

    View Slide

  58. Easiest option

    View Slide

  59. Low latencies

    View Slide

  60. You can use one network

    View Slide

  61. All pods and services can talk with
    each other

    View Slide

  62. One cluster => one internal DNS

    View Slide

  63. Service Discovery and internal load
    balancing works

    View Slide

  64. External LoadBalancers work

    View Slide

  65. Zone 1 Zone 2
    Kubernetes Master Components
    Nginx
    Nginx
    Nginx Service
    Curl
    LB

    View Slide

  66. You can mount storage across AZ

    View Slide

  67. One cluster state => One deployment

    View Slide

  68. But what if the whole region is
    down?

    View Slide

  69. One Kubernetes cluster across
    multiple regions and VPN connection
    between networks

    View Slide

  70. Region 1 Region 2
    WireGuard VPN
    Kubernetes Master Components
    Nginx
    Nginx
    Nginx Service
    Curl

    View Slide

  71. Latencies get higher by distance

    View Slide

  72. You need a VPN to connect the
    networks

    View Slide

  73. WireGuard as a VPN

    View Slide

  74. https:/
    /github.com/squat/kilo

    View Slide

  75. Kilo gets all Nodes from Kubernetes

    View Slide

  76. Discovers regions and zones by
    Kubernetes Node labels

    View Slide

  77. Configures WireGuard

    View Slide

  78. Routing tables

    View Slide

  79. All pods and services can talk with
    each other

    View Slide

  80. One cluster => one internal DNS

    View Slide

  81. Service Discovery and internal load
    balancing works

    View Slide

  82. You need a solution for external load
    balancing

    View Slide

  83. Region 1 Region 2
    WireGuard VPN
    Kubernetes Master Components
    Nginx
    Nginx
    Nginx Service
    Curl
    LB LB

    View Slide

  84. You have to replicate storage
    yourself

    View Slide

  85. Not every Storage provider supports
    dynamic volume provisioning

    View Slide

  86. One cluster state => One deployment

    View Slide

  87. Demo

    View Slide

  88. Multi cluster setup

    View Slide

  89. Connecting multiple clusters with a
    VPN

    View Slide

  90. Cluster 1 Cluster 2
    WireGuard VPN
    Kubernetes Master
    Components
    Kubernetes Master
    Components
    Nginx
    Nginx
    Nginx Service Nginx Service
    Curl

    View Slide

  91. WireGuard as a VPN

    View Slide

  92. https:/
    /github.com/squat/kilo

    View Slide

  93. All pods and services can talk with
    each other

    View Slide

  94. Separate clusters => separate
    internal DNS

    View Slide

  95. For service discovery configure each
    internal DNS to resolve to other
    clusters

    View Slide

  96. cluster.region2:53 {
    forward . 10.10.11.10
    }
    .:53 {
    kubernetes cluster.local cluster.region1 in-addr.arpa
    ip6.arpa {
    pods insecure
    upstream
    fallthrough in-addr.arpa ip6.arpa
    }
    forward . /etc/resolv.conf
    loop
    loadbalance
    }

    View Slide

  97. cluster.region1:53 {
    forward . 10.10.10.10
    }
    .:53 {
    kubernetes cluster.local cluster.region2 in-addr.arpa
    ip6.arpa {
    pods insecure
    upstream
    fallthrough in-addr.arpa ip6.arpa
    }
    forward . /etc/resolv.conf
    loop
    loadbalance
    }

    View Slide

  98. Internal load balancing works

    View Slide

  99. You have to replicate storage
    yourself

    View Slide

  100. Every cluster has their own
    StorageClass that works on all nodes

    View Slide

  101. Separate clusters have separate
    state

    View Slide

  102. Management of deployments across
    clusters

    View Slide

  103. Kubefed
    https://github.com/kubernetes-sigs/kubefed

    View Slide

  104. FederatedNamespaces,
    FederatedDeployments,
    FederatedConfigMaps,
    FederatedServices,
    ...

    View Slide

  105. Cluster aware controller that
    manages resources in all connected
    clusters

    View Slide

  106. Cluster 1 Cluster 2
    Kubernetes Master
    Components
    Kubernetes Master
    Components
    Kubefed Controller

    View Slide

  107. Cluster 1 Cluster 2
    Kubernetes Master
    Components
    Kubernetes Master
    Components
    Kubefed Controller
    FederatedService
    FederatedDeployment

    View Slide

  108. Cluster 1 Cluster 2
    Kubernetes Master
    Components
    Kubernetes Master
    Components
    Nginx Service Nginx Service
    Kubefed Controller Kubefed Controller
    Nginx Nginx

    View Slide

  109. Demo

    View Slide

  110. Solution for external traffic

    View Slide

  111. Cluster 1 Cluster 2
    WireGuard VPN
    Kubernetes Master
    Components
    Kubernetes Master
    Components
    Nginx
    Nginx
    Nginx Service Nginx Service
    Curl
    LB LB

    View Slide

  112. Multi cluster external-dns

    View Slide

  113. View Slide

  114. service.namespace.domain.svc.example.com
    service.namespace.domain.svc.region1.example.com
    service.namespace.domain.svc.region2.example.com

    View Slide

  115. Demo

    View Slide

  116. More options

    View Slide

  117. Multiple clusters connected via
    Service Mesh (Istio)

    View Slide

  118. Conclusion

    View Slide

  119. Kubernetes makes it easier to create
    multi region setups

    View Slide

  120. There are still challenges you have to
    overcome

    View Slide

  121. Often just physics

    View Slide

  122. Federation Tooling is just getting
    started

    View Slide

  123. Test it
    30 days
    For free
    Visit us at our
    booth

    View Slide

  124. View Slide

  125. https:/
    /github.com/bashofmann/
    kubernetes-multicluster-demos

    View Slide

  126. [email protected]
    https:/
    /twitter.com/BastianHofmann

    View Slide

  127. Connecting multiple clusters with a
    Service Mesh Gateway

    View Slide

  128. View Slide

  129. View Slide

  130. Cluster 1 Cluster 2
    Istio Gateway
    Istio Control Plane
    Kubernetes Master
    Components
    Kubernetes Master
    Components
    Istio
    Nginx
    Nginx
    Nginx Service Nginx Service
    Istio Gateway
    Curl

    View Slide

  131. No VPN necessary

    View Slide

  132. Fairly easy to set up

    View Slide

  133. Pods from different clusters
    communicate over public Ips

    View Slide

  134. Traffic encrypted and authenticated
    with mutual TLS

    View Slide

  135. Communication is only possible
    through Istio proxies

    View Slide

  136. Easy service discovery

    View Slide

  137. Flexible, location aware traffic
    management

    View Slide

  138. Telemetry and tracing included

    View Slide

  139. Demo

    View Slide