Cloud Native Forum How to impletement Kubernetes Bare metal Load Balancer Kyle Bai, inwinSTACK @k2r2bai

About Me ⽩白凱仁(Kyle Bai) • RDSS at inwinSTACK. • Interested in emerging technologies. • Kubernetes Projects Contributor(100+ PR). • Certified Kubernetes Administrator. • CNTUG(Cloud Native Taiwan User Group) co-organizer @kairen(k2r2.bai@gmail.com) https://kairen.github.io/

• Why need bare-metal LB for Kubernetes? • Introduction • Custom Resources for Kubernetes • Custom Controllers for Kubernetes • IPVS • Implement IPVS-based LB • Live Demo Agenda Today I would like to talk about Cloud Native Forum

“Kubernetes is becoming the Linux of the cloud” - Jim Zemlin, Linux Foundation

Cloud Native Forum Why need bare-metal LB for Kubernetes?

Kubernetes is very flexible to deploy • Kubernetes is very flexible in how you can deploy it. You can deploy to cloud environments like Google Cloud, Microsoft Azure, and Amazon AWS. • You can deploy Kubernetes on bare metal using several popular operating systems like Ubuntu Linux, CentOS. On-premises

Limitations of On-Premises • Kubernetes does not offer an implementation of network load-balancers (Services of type LoadBalancer) for bare metal(On-Premises) clusters. • Bare metal cluster operators are left with two lesser tools to bring user traffic into their clusters, “NodePort” and “externalIPs” services. • Both of these options have significant downsides for production use, which makes bare metal clusters second class citizens in the Kubernetes ecosystem.

Expose our services to the outside world Kubernetes provides several ways to expose these services: • HostNetworking

Expose our services to the outside world Kubernetes provides several ways to expose these services: • HostNetworking

Expose our services to the outside world Kubernetes provides several ways to expose these services: • HostNetworking

Expose our services to the outside world Kubernetes provides several ways to expose these services: • HostNetworking • Service Service Client Proxy

Expose our services to the outside world Kubernetes provides several ways to expose these services: • HostNetworking • Service

Expose our services to the outside world Kubernetes provides several ways to expose these services: • HostNetworking • Service • NodePort

Expose our services to the outside world Kubernetes provides several ways to expose these services: • HostNetworking • Service • NodePort • LoadBalancer

Expose our services to the outside world Kubernetes provides several ways to expose these services: • HostNetworking • Service • NodePort • LoadBalancer P.S. Does not offer an implementation of network load-balancers for bare metal clusters.

Expose our services to the outside world Kubernetes provides several ways to expose these services: • HostNetworking • Service • NodePort • LoadBalancer • Ingress

Cloud Native Forum Introduction • Custom Resources for Kubernetes • Custom Controllers for Kubernetes • IPVS

Kubernetes Custom Resources • A resource is an endpoint in the Kubernetes API that stores a collection of API objects of a certain kind. ex: Pod. • A custom resource is an extension of the Kubernetes API that is not necessarily available on every Kubernetes cluster. • Kubernetes provides two ways to add custom resources to your cluster: • CRDs • API Aggregation(custom apiserver)

CRD(CustomResourceDefinition) • The CustomResourceDefinition API resource allows you to define custom resources. • Defining a CRD object creates a new custom resource with a name and schema that you specify. • Do not require programming

API Aggregation • Require coding, built atop k8s.io/apiserver library. • Highly customizable, like adding a new verb, create/delete hooks. • Typed fields, validation, defaults. • Multi-versioning, supporting old clients. • Generated OpenAPI schema. • Supports protobuf. https://github.com/kubernetes/sample-apiserver https://github.com/kubernetes-sigs/kubebuilder

Kubernetes Custom Controllers • Kubernetes 1.7 has added an important feature called Custom Controllers. • It enables developers to extend and add new functionalities, replace existent ones (like replacing kube-proxy for instance). • And of course, automate administration tasks as if they were a native Kubernetes component.

Example: PA Firewall + Kubernetes • Provides Security and NAT custom resources. • Automatically sync PA security and NAT policies.

No content

How to create a controller? https://github.com/kubernetes/client-go/tree/master/examples/workqueue

https://github.com/kubernetes/client-go/tree/master/examples/workqueue

https://github.com/kubernetes/client-go/tree/master/examples/workqueue

https://github.com/kubernetes/client-go/tree/master/examples/workqueue

https://github.com/kubernetes/client-go/tree/master/examples/workqueue

Kubernetes Operators • An Operator is nothing more than a set of application-specific custom controllers. • the Operator monitors and analyzes the cluster, and based on a set of parameters, trigger a series of actions to achieve the desired state. https://coreos.com/operators/ https://github.com/operator-framework/operator-sdk

Etcd Operator

What is IPVS? • IPVS (IP Virtual Server) implements transport-layer load balancing, usually called Layer 4 LAN switching, as part of Linux kernel. • IPVS is incorporated into the LVS (Linux Virtual Server), where it runs on a host and acts as a load balancer in front of a cluster of real servers. • Same to IPTables, IPVS is built on top of Netfilter. • Support 3 load balancing mode: DNAT, DR(or DSR) and IP tunnel.

3 Load Balancing Mode

Why using IPVS? • Better performance (Hashing vs. Chain) • More load balancing algorithm • Round robin, source/destination hashing. • Based on least load, least connection or locality, can assign weight to server. • Support server health check and connection retry • Support sticky session

Cloud Native Forum Implement IPVS-based LB

Architecture

IP and Pool Operator Because Kubernetes cannot create IP addresses out of thin air, so you do have to give it pools of IP addresses that it can use. • Provides custom resources(IP and Pool). • Drive current state to desired state. • Allocate IP instance from Pool.

IP Assigner IP Assigner will take care of assigning and unassigning individual addresses as services come and go. but it will only ever hand out IPs that are part of its configured pools.

IPVS Node IPVS Node watches Kubernetes Services, calls netlink and libipvs to create IPVS rules accordingly and syncs ipvs rules with Kubernetes Services periodically, to make sure IPVS status is consistent with the expectation.

IPVS Node Network Topology When creating a LoadBalancer type Service and assign external IP, IPVS Node will do the following things: • Make sure a interface exists in the node, using bind-iface flag to specify. • Call go-netlink lib to bind Service external IP(VIP) addresses to the interface. • Call go-libipvs Create IPVS virtual servers for each Service external IP address respectively(Forward VIP to cluster IP). This rule wiil be used Round-robin algorithm.

Example # ip addr show dev enp6s0 2: enp6s0: mtu 1500 qdisc pfifo_fast state UP group default qlen 1000 link/ether 54:a0:50:85:d5:23 brd ff:ff:ff:ff:ff:ff inet 172.22.132.12/24 brd 172.22.132.255 scope global enp6s0 valid_lft forever preferred_lft forever inet 172.22.132.200/32 brd 172.22.132.255 scope global enp6s0:0 valid_lft forever preferred_lft forever inet6 fe80::56a0:50ff:fe85:d523/64 scope link valid_lft forever preferred_lft forever

Example(Cont.) # ipvsadm -L IP Virtual Server version 1.2.1 (size=4096) Prot LocalAddress:Port Scheduler Flags -> RemoteAddress:Port Forward Weight ActiveConn InActConn TCP 172.22.132.200:http rr -> 10.105.177.190:http Route 1 0 0

IPVS-based Kube-proxy

IPVS-based Kube-proxy Network Topology When creating a ClusterIP or LoadBalancer type Service and assign external IP, IPVS proxier will do the following things: • Make sure a dummy interface exists in the nodes, defaults to kube-ipvs0. • Bind Service IP addresses(Cluster IP and External IP) to the dummy interface. • Create IPVS virtual servers for each Service IP address respectively.

Example # ip addr show kube-ipvs0 14: kube-ipvs0: mtu 1500 qdisc noop state DOWN group default link/ether 3a:eb:de:45:db:92 brd ff:ff:ff:ff:ff:ff inet 10.96.30.32/32 brd 10.96.30.32 scope global kube-ipvs0 valid_lft forever preferred_lft forever inet 10.100.233.153/32 brd 10.100.233.153 scope global kube-ipvs0 valid_lft forever preferred_lft forever inet 172.22.132.200/32 brd 172.22.132.200 scope global kube-ipvs0 valid_lft forever preferred_lft forever

Example(Cont.) # ipvsadm -L IP Virtual Server version 1.2.1 (size=4096) Prot LocalAddress:Port Scheduler Flags -> RemoteAddress:Port Forward Weight ActiveConn InActConn TCP 172.22.132.200:http rr -> 10.244.3.10:http Masq 1 0 0

Using kube-proxy replace IPVS Node

Live Demo Cloud Native Forum

MetalLB MetalLB hooks into your Kubernetes cluster, and provides a network load- balancer implementation. • Address allocation. • External announcement. • Layer 2 mode (ARP for IPv4, NDP for IPv6). • Layer 3 mode (BGP). https://metallb.universe.tf/concepts/

Refers • https://metallb.universe.tf/concepts/ • https://kubernetes.io/docs/concepts/extend-kubernetes/api-extension/ • http://www.linuxvirtualserver.org/software/ipvs.html • https://kubernetes.io/blog/2018/07/09/ipvs-based-in-cluster-load-balancing-deep- dive/ • https://blog.couchbase.com/kubernetes-operators-game-changer/ • https://github.com/kubernetes/kubernetes/tree/master/pkg/proxy/ipvs • https://bestsamina.github.io/posts/2018-10-19-ipvs-based-kube-proxy-4-scaled-k8s-lb/ • https://github.com/vishvananda/netlink

Thanks you!!! Cloud Native Forum