How to impletement Kubernetes Bare metal Load Balancer

5a84ea9a0afaca03be45b87dde58e51c?s=47 Kyle Bai
November 22, 2018
440

How to impletement Kubernetes Bare metal Load Balancer

5a84ea9a0afaca03be45b87dde58e51c?s=128

Kyle Bai

November 22, 2018
Tweet

Transcript

  1. Cloud Native Forum How to impletement Kubernetes Bare metal Load

    Balancer Kyle Bai, inwinSTACK @k2r2bai
  2. About Me ⽩白凱仁(Kyle Bai) • RDSS at inwinSTACK. • Interested

    in emerging technologies. • Kubernetes Projects Contributor(100+ PR). • Certified Kubernetes Administrator. • CNTUG(Cloud Native Taiwan User Group) co-organizer @kairen(k2r2.bai@gmail.com) https://kairen.github.io/
  3. • Why need bare-metal LB for Kubernetes? • Introduction •

    Custom Resources for Kubernetes • Custom Controllers for Kubernetes • IPVS • Implement IPVS-based LB • Live Demo Agenda Today I would like to talk about Cloud Native Forum
  4. “Kubernetes is becoming the Linux of the cloud” - Jim

    Zemlin, Linux Foundation
  5. Cloud Native Forum Why need bare-metal LB for Kubernetes?

  6. Kubernetes is very flexible to deploy • Kubernetes is very

    flexible in how you can deploy it. You can deploy to cloud environments like Google Cloud, Microsoft Azure, and Amazon AWS. • You can deploy Kubernetes on bare metal using several popular operating systems like Ubuntu Linux, CentOS. On-premises
  7. Limitations of On-Premises • Kubernetes does not offer an implementation

    of network load-balancers (Services of type LoadBalancer) for bare metal(On-Premises) clusters. • Bare metal cluster operators are left with two lesser tools to bring user traffic into their clusters, “NodePort” and “externalIPs” services. • Both of these options have significant downsides for production use, which makes bare metal clusters second class citizens in the Kubernetes ecosystem.
  8. Expose our services to the outside world Kubernetes provides several

    ways to expose these services: • HostNetworking
  9. Expose our services to the outside world Kubernetes provides several

    ways to expose these services: • HostNetworking
  10. Expose our services to the outside world Kubernetes provides several

    ways to expose these services: • HostNetworking
  11. Expose our services to the outside world Kubernetes provides several

    ways to expose these services: • HostNetworking • Service Service Client Proxy
  12. Expose our services to the outside world Kubernetes provides several

    ways to expose these services: • HostNetworking • Service
  13. Expose our services to the outside world Kubernetes provides several

    ways to expose these services: • HostNetworking • Service • NodePort
  14. Expose our services to the outside world Kubernetes provides several

    ways to expose these services: • HostNetworking • Service • NodePort • LoadBalancer
  15. Expose our services to the outside world Kubernetes provides several

    ways to expose these services: • HostNetworking • Service • NodePort • LoadBalancer P.S. Does not offer an implementation of network load-balancers for bare metal clusters.
  16. Expose our services to the outside world Kubernetes provides several

    ways to expose these services: • HostNetworking • Service • NodePort • LoadBalancer • Ingress
  17. Cloud Native Forum Introduction • Custom Resources for Kubernetes •

    Custom Controllers for Kubernetes • IPVS
  18. Kubernetes Custom Resources • A resource is an endpoint in

    the Kubernetes API that stores a collection of API objects of a certain kind. ex: Pod. • A custom resource is an extension of the Kubernetes API that is not necessarily available on every Kubernetes cluster. • Kubernetes provides two ways to add custom resources to your cluster: • CRDs • API Aggregation(custom apiserver)
  19. CRD(CustomResourceDefinition) • The CustomResourceDefinition API resource allows you to define

    custom resources. • Defining a CRD object creates a new custom resource with a name and schema that you specify. • Do not require programming
  20. API Aggregation • Require coding, built atop k8s.io/apiserver library. •

    Highly customizable, like adding a new verb, create/delete hooks. • Typed fields, validation, defaults. • Multi-versioning, supporting old clients. • Generated OpenAPI schema. • Supports protobuf. https://github.com/kubernetes/sample-apiserver https://github.com/kubernetes-sigs/kubebuilder
  21. Kubernetes Custom Controllers • Kubernetes 1.7 has added an important

    feature called Custom Controllers. • It enables developers to extend and add new functionalities, replace existent ones (like replacing kube-proxy for instance). • And of course, automate administration tasks as if they were a native Kubernetes component.
  22. Example: PA Firewall + Kubernetes • Provides Security and NAT

    custom resources. • Automatically sync PA security and NAT policies.
  23. None
  24. How to create a controller? https://github.com/kubernetes/client-go/tree/master/examples/workqueue

  25. https://github.com/kubernetes/client-go/tree/master/examples/workqueue

  26. https://github.com/kubernetes/client-go/tree/master/examples/workqueue

  27. https://github.com/kubernetes/client-go/tree/master/examples/workqueue

  28. https://github.com/kubernetes/client-go/tree/master/examples/workqueue

  29. Kubernetes Operators • An Operator is nothing more than a

    set of application-specific custom controllers. • the Operator monitors and analyzes the cluster, and based on a set of parameters, trigger a series of actions to achieve the desired state. https://coreos.com/operators/ https://github.com/operator-framework/operator-sdk
  30. Etcd Operator

  31. What is IPVS? • IPVS (IP Virtual Server) implements transport-layer

    load balancing, usually called Layer 4 LAN switching, as part of Linux kernel. • IPVS is incorporated into the LVS (Linux Virtual Server), where it runs on a host and acts as a load balancer in front of a cluster of real servers. • Same to IPTables, IPVS is built on top of Netfilter. • Support 3 load balancing mode: DNAT, DR(or DSR) and IP tunnel.
  32. 3 Load Balancing Mode

  33. Why using IPVS? • Better performance (Hashing vs. Chain) •

    More load balancing algorithm • Round robin, source/destination hashing. • Based on least load, least connection or locality, can assign weight to server. • Support server health check and connection retry • Support sticky session
  34. Cloud Native Forum Implement IPVS-based LB

  35. Architecture

  36. IP and Pool Operator Because Kubernetes cannot create IP addresses

    out of thin air, so you do have to give it pools of IP addresses that it can use. • Provides custom resources(IP and Pool). • Drive current state to desired state. • Allocate IP instance from Pool.
  37. IP Assigner IP Assigner will take care of assigning and

    unassigning individual addresses as services come and go. but it will only ever hand out IPs that are part of its configured pools.
  38. IPVS Node IPVS Node watches Kubernetes Services, calls netlink and

    libipvs to create IPVS rules accordingly and syncs ipvs rules with Kubernetes Services periodically, to make sure IPVS status is consistent with the expectation.
  39. IPVS Node Network Topology When creating a LoadBalancer type Service

    and assign external IP, IPVS Node will do the following things: • Make sure a interface exists in the node, using bind-iface flag to specify. • Call go-netlink lib to bind Service external IP(VIP) addresses to the interface. • Call go-libipvs Create IPVS virtual servers for each Service external IP address respectively(Forward VIP to cluster IP). This rule wiil be used Round-robin algorithm.
  40. Example # ip addr show dev enp6s0 2: enp6s0: <BROADCAST,MULTICAST,UP,LOWER_UP>

    mtu 1500 qdisc pfifo_fast state UP group default qlen 1000 link/ether 54:a0:50:85:d5:23 brd ff:ff:ff:ff:ff:ff inet 172.22.132.12/24 brd 172.22.132.255 scope global enp6s0 valid_lft forever preferred_lft forever inet 172.22.132.200/32 brd 172.22.132.255 scope global enp6s0:0 valid_lft forever preferred_lft forever inet6 fe80::56a0:50ff:fe85:d523/64 scope link valid_lft forever preferred_lft forever
  41. Example(Cont.) # ipvsadm -L IP Virtual Server version 1.2.1 (size=4096)

    Prot LocalAddress:Port Scheduler Flags -> RemoteAddress:Port Forward Weight ActiveConn InActConn TCP 172.22.132.200:http rr -> 10.105.177.190:http Route 1 0 0
  42. IPVS-based Kube-proxy

  43. IPVS-based Kube-proxy Network Topology When creating a ClusterIP or LoadBalancer

    type Service and assign external IP, IPVS proxier will do the following things: • Make sure a dummy interface exists in the nodes, defaults to kube-ipvs0. • Bind Service IP addresses(Cluster IP and External IP) to the dummy interface. • Create IPVS virtual servers for each Service IP address respectively.
  44. Example # ip addr show kube-ipvs0 14: kube-ipvs0: <BROADCAST,NOARP> mtu

    1500 qdisc noop state DOWN group default link/ether 3a:eb:de:45:db:92 brd ff:ff:ff:ff:ff:ff inet 10.96.30.32/32 brd 10.96.30.32 scope global kube-ipvs0 valid_lft forever preferred_lft forever inet 10.100.233.153/32 brd 10.100.233.153 scope global kube-ipvs0 valid_lft forever preferred_lft forever inet 172.22.132.200/32 brd 172.22.132.200 scope global kube-ipvs0 valid_lft forever preferred_lft forever
  45. Example(Cont.) # ipvsadm -L IP Virtual Server version 1.2.1 (size=4096)

    Prot LocalAddress:Port Scheduler Flags -> RemoteAddress:Port Forward Weight ActiveConn InActConn TCP 172.22.132.200:http rr -> 10.244.3.10:http Masq 1 0 0
  46. Using kube-proxy replace IPVS Node

  47. Live Demo Cloud Native Forum

  48. MetalLB MetalLB hooks into your Kubernetes cluster, and provides a

    network load- balancer implementation. • Address allocation. • External announcement. • Layer 2 mode (ARP for IPv4, NDP for IPv6). • Layer 3 mode (BGP). https://metallb.universe.tf/concepts/
  49. Refers • https://metallb.universe.tf/concepts/ • https://kubernetes.io/docs/concepts/extend-kubernetes/api-extension/ • http://www.linuxvirtualserver.org/software/ipvs.html • https://kubernetes.io/blog/2018/07/09/ipvs-based-in-cluster-load-balancing-deep- dive/

    • https://blog.couchbase.com/kubernetes-operators-game-changer/ • https://github.com/kubernetes/kubernetes/tree/master/pkg/proxy/ipvs • https://bestsamina.github.io/posts/2018-10-19-ipvs-based-kube-proxy-4-scaled-k8s-lb/ • https://github.com/vishvananda/netlink
  50. Thanks you!!! Cloud Native Forum