Slide 1

Slide 1 text

Setup Hybrid Clusters using Kubernetes Federation Kyle bai R&D @ inwinSTACK www.inwinstack.com

Slide 2

Slide 2 text

About Me ⽩白凱仁(Kyle Bai) • Interested in emerging technologies. • COSCUP, Kubernetes Day and OpenStack Day Speaker. • OpenStack and Kubernetes Projects Contributor(100+ PR). • Certified Kubernetes Administrator. @kairen([email protected]) https://kairen.github.io/

Slide 3

Slide 3 text

Agenda • Why Federation? • Federation Architecture • How to setup Kubernetes Federation • Bonus

Slide 4

Slide 4 text

Why Federation?

Slide 5

Slide 5 text

Dev Test Multiple Clusters Product

Slide 6

Slide 6 text

Dev Test Multiple Clusters Product US EU AP TW

Slide 7

Slide 7 text

Dev Test Multiple Clusters Product US EU AP TW

Slide 8

Slide 8 text

Multiple Clusters TW Local Dev

Slide 9

Slide 9 text

Multiple Clusters TW US Test TW Local Dev

Slide 10

Slide 10 text

Multiple Clusters US EU … AP Product TW US Test TW Local Dev

Slide 11

Slide 11 text

Multiple Clusters • Serving users from clusters closest to them. • Avoiding provider lock-in: By making it easier to migrate applications across clusters, federation prevents cluster provider lock-in. • High availability: Ability to federate clusters across different regions/ cloud providers.

Slide 12

Slide 12 text

Federation Federation makes it easy to manage multiple clusters. • Sync resources across clusters: Federation provides the ability to keep resources in multiple clusters in sync. • Cross cluster discovery: Federation provides the ability to auto-configure DNS servers and load balancers with backends from all clusters.

Slide 13

Slide 13 text

Federation Pros • Keep your app synced across clusters. • Configure network resources (services, ingress) to route traffic across clusters. • Single place to apply policies. • Policy-based Resource Placement(OPA).

Slide 14

Slide 14 text

Use cases for Federation • Geographically Distributed Deployments: Spread Deployments across clusters in different parts of the world. • Hybrid Cloud: Extend Deployments from on-premise clusters to the cloud. • Application Migration: Simplify the migration of applications from on- premise to the cloud or between cloud providers.

Slide 15

Slide 15 text

Federation Architecture

Slide 16

Slide 16 text

Overview Federation Control Plane

Slide 17

Slide 17 text

Overview Federation Control Plane Federated Resources

Slide 18

Slide 18 text

Overview Federation Control Plane Federated Resources Federated Clusters

Slide 19

Slide 19 text

Federation API Server The Kubernetes federation API server validates and configures data for the API objects which include pods, services, replicationcontrollers, and others.

Slide 20

Slide 20 text

Federation Controller Manager The federation controller manager is a daemon that embeds the core control loops shipped with federation. • Watches Federation API Server • Clusters - federation/v1beta1/cluster • API Resources - v1/foo • Watches All Kubernetes Clusters • API Resources - v1/foo • Reconciles • Compare and update • Handles cascading deletion

Slide 21

Slide 21 text

How to place? • Placement can be controlled per-resource via annotations. • Annotations supported: • federation.kubernetes.io/replica-set-preferences • federation.kubernetes.io/deployment-preferences • federation.alpha.kubernetes.io/cluster-selector • federation-controller evaluates annotations to produce final placement

Slide 22

Slide 22 text

How to setup Kubernetes Federation

Slide 23

Slide 23 text

Prerequisites(1/2) • In this demon, we will deploy clusters in different regions: • US West: Oregon (us-west) • Asia: Tokyo(ap-northeast) • My OpenStack: Taipei(os-taipei) • Install the following tools on host: • kubectl • kubefed: if os is Mac OS X, you need build from Federation source code. • kops, kubespray, kubeadm • AWS CLI

Slide 24

Slide 24 text

Prerequisites(2/2) Amazon Web Services: • IAM: Identity and Access Management. • EC2: Kubernetes cluster. • ELB: Kubernetes service load balancer. • Route53: Public domain for Kubernetes API, Service, … etc • S3: Store kops state. • VPC: Cluster Network. Godaddy: • Buy your own domain name.

Slide 25

Slide 25 text

Deme Clusters kubefed init ${FED_CONTEXT} \ --image=kairen/fcp-amd64:v1.10.0-alpha \ --host-cluster-context=${US_WEST_CONTEXT} \ --dns-provider=aws-route53 \ --dns-zone-name=${DOMAIN_NAME}

Slide 26

Slide 26 text

Deme Clusters kubefed join os-taipei \ --host-cluster-context=${US_WEST_CONTEXT} \ --cluster-context=${OS_CONTEXT}

Slide 27

Slide 27 text

Deme Topology

Slide 28

Slide 28 text

Bonus

Slide 29

Slide 29 text

Federation + OPA(Open Policy Agent ) Current Use Cases • Federated Workload Placement • Pod Scheduling • Authorization • Admission Control • Audit Future Use Cases • Storage policy • Network policy

Slide 30

Slide 30 text

Federation + OPA(Open Policy Agent )

Slide 31

Slide 31 text

Federation Cons • Increased network bandwidth and cost: The federation control plane watches all clusters to ensure that the current state is as expected. • Reduced cross cluster isolation: A bug in the federation control plane can impact all clusters. • Maturity: The federation project is relatively new and is not very mature. • Not all resources are available and many are still alpha and beta.

Slide 32

Slide 32 text

Federation support API resources • Cluster • ConfigMap • DaemonSets • Deployment • Events • HPA • Ingress • Jobs • Namespaces • ReplicaSets • Secrets • Services * Only support API version for extensions/v1beta1, v1

Slide 33

Slide 33 text

Federation v2 https://github.com/kubernetes-sigs/federation-v2 The prototype builds on the sync controller (a.k.a. push reconciler) from Federation v1. • Possible to implement a custom scheduler • Policies modeled as CRDs • Use existing RBAC • Can plug into policy engine • Implemented as an aggregated API server • Secured with ServiceAccounts https://github.com/font/k8s-example-apps/tree/master/sample-fed-v2-demo

Slide 34

Slide 34 text

Federation v2 The following abstractions support the propagation of a logical federated type: • Template: defines the representation of the resource common across clusters. • Placement: defines which clusters the resource is intended to appear in. • Override: optionally defines per-cluster field-level variation to apply to the template. https://docs.google.com/document/d/159cQGlfgXo6O4WxXyWzjZiPoIuiHVl933B43xhmqPEE/edit

Slide 35

Slide 35 text

Thanks