Slide 1

Slide 1 text

Cloud Native Forum How to impletement Kubernetes Bare metal Load Balancer Kyle Bai, inwinSTACK @k2r2bai

Slide 2

Slide 2 text

About Me ⽩白凱仁(Kyle Bai) • RDSS at inwinSTACK. • Interested in emerging technologies. • Kubernetes Projects Contributor(100+ PR). • Certified Kubernetes Administrator. • CNTUG(Cloud Native Taiwan User Group) co-organizer @kairen([email protected]) https://kairen.github.io/

Slide 3

Slide 3 text

• Why need bare-metal LB for Kubernetes? • Introduction • Custom Resources for Kubernetes • Custom Controllers for Kubernetes • IPVS • Implement IPVS-based LB • Live Demo Agenda Today I would like to talk about Cloud Native Forum

Slide 4

Slide 4 text

“Kubernetes is becoming the Linux of the cloud” - Jim Zemlin, Linux Foundation

Slide 5

Slide 5 text

Cloud Native Forum Why need bare-metal LB for Kubernetes?

Slide 6

Slide 6 text

Kubernetes is very flexible to deploy • Kubernetes is very flexible in how you can deploy it. You can deploy to cloud environments like Google Cloud, Microsoft Azure, and Amazon AWS. • You can deploy Kubernetes on bare metal using several popular operating systems like Ubuntu Linux, CentOS. On-premises

Slide 7

Slide 7 text

Limitations of On-Premises • Kubernetes does not offer an implementation of network load-balancers (Services of type LoadBalancer) for bare metal(On-Premises) clusters. • Bare metal cluster operators are left with two lesser tools to bring user traffic into their clusters, “NodePort” and “externalIPs” services. • Both of these options have significant downsides for production use, which makes bare metal clusters second class citizens in the Kubernetes ecosystem.

Slide 8

Slide 8 text

Expose our services to the outside world Kubernetes provides several ways to expose these services: • HostNetworking

Slide 9

Slide 9 text

Expose our services to the outside world Kubernetes provides several ways to expose these services: • HostNetworking

Slide 10

Slide 10 text

Expose our services to the outside world Kubernetes provides several ways to expose these services: • HostNetworking

Slide 11

Slide 11 text

Expose our services to the outside world Kubernetes provides several ways to expose these services: • HostNetworking • Service Service Client Proxy

Slide 12

Slide 12 text

Expose our services to the outside world Kubernetes provides several ways to expose these services: • HostNetworking • Service

Slide 13

Slide 13 text

Expose our services to the outside world Kubernetes provides several ways to expose these services: • HostNetworking • Service • NodePort

Slide 14

Slide 14 text

Expose our services to the outside world Kubernetes provides several ways to expose these services: • HostNetworking • Service • NodePort • LoadBalancer

Slide 15

Slide 15 text

Expose our services to the outside world Kubernetes provides several ways to expose these services: • HostNetworking • Service • NodePort • LoadBalancer P.S. Does not offer an implementation of network load-balancers for bare metal clusters.

Slide 16

Slide 16 text

Expose our services to the outside world Kubernetes provides several ways to expose these services: • HostNetworking • Service • NodePort • LoadBalancer • Ingress

Slide 17

Slide 17 text

Cloud Native Forum Introduction • Custom Resources for Kubernetes • Custom Controllers for Kubernetes • IPVS

Slide 18

Slide 18 text

Kubernetes Custom Resources • A resource is an endpoint in the Kubernetes API that stores a collection of API objects of a certain kind. ex: Pod. • A custom resource is an extension of the Kubernetes API that is not necessarily available on every Kubernetes cluster. • Kubernetes provides two ways to add custom resources to your cluster: • CRDs • API Aggregation(custom apiserver)

Slide 19

Slide 19 text

CRD(CustomResourceDefinition) • The CustomResourceDefinition API resource allows you to define custom resources. • Defining a CRD object creates a new custom resource with a name and schema that you specify. • Do not require programming

Slide 20

Slide 20 text

API Aggregation • Require coding, built atop k8s.io/apiserver library. • Highly customizable, like adding a new verb, create/delete hooks. • Typed fields, validation, defaults. • Multi-versioning, supporting old clients. • Generated OpenAPI schema. • Supports protobuf. https://github.com/kubernetes/sample-apiserver https://github.com/kubernetes-sigs/kubebuilder

Slide 21

Slide 21 text

Kubernetes Custom Controllers • Kubernetes 1.7 has added an important feature called Custom Controllers. • It enables developers to extend and add new functionalities, replace existent ones (like replacing kube-proxy for instance). • And of course, automate administration tasks as if they were a native Kubernetes component.

Slide 22

Slide 22 text

Example: PA Firewall + Kubernetes • Provides Security and NAT custom resources. • Automatically sync PA security and NAT policies.

Slide 23

Slide 23 text

No content

Slide 24

Slide 24 text

How to create a controller? https://github.com/kubernetes/client-go/tree/master/examples/workqueue

Slide 25

Slide 25 text

https://github.com/kubernetes/client-go/tree/master/examples/workqueue

Slide 26

Slide 26 text

https://github.com/kubernetes/client-go/tree/master/examples/workqueue

Slide 27

Slide 27 text

https://github.com/kubernetes/client-go/tree/master/examples/workqueue

Slide 28

Slide 28 text

https://github.com/kubernetes/client-go/tree/master/examples/workqueue

Slide 29

Slide 29 text

Kubernetes Operators • An Operator is nothing more than a set of application-specific custom controllers. • the Operator monitors and analyzes the cluster, and based on a set of parameters, trigger a series of actions to achieve the desired state. https://coreos.com/operators/ https://github.com/operator-framework/operator-sdk

Slide 30

Slide 30 text

Etcd Operator

Slide 31

Slide 31 text

What is IPVS? • IPVS (IP Virtual Server) implements transport-layer load balancing, usually called Layer 4 LAN switching, as part of Linux kernel. • IPVS is incorporated into the LVS (Linux Virtual Server), where it runs on a host and acts as a load balancer in front of a cluster of real servers. • Same to IPTables, IPVS is built on top of Netfilter. • Support 3 load balancing mode: DNAT, DR(or DSR) and IP tunnel.

Slide 32

Slide 32 text

3 Load Balancing Mode

Slide 33

Slide 33 text

Why using IPVS? • Better performance (Hashing vs. Chain) • More load balancing algorithm • Round robin, source/destination hashing. • Based on least load, least connection or locality, can assign weight to server. • Support server health check and connection retry • Support sticky session

Slide 34

Slide 34 text

Cloud Native Forum Implement IPVS-based LB

Slide 35

Slide 35 text

Architecture

Slide 36

Slide 36 text

IP and Pool Operator Because Kubernetes cannot create IP addresses out of thin air, so you do have to give it pools of IP addresses that it can use. • Provides custom resources(IP and Pool). • Drive current state to desired state. • Allocate IP instance from Pool.

Slide 37

Slide 37 text

IP Assigner IP Assigner will take care of assigning and unassigning individual addresses as services come and go. but it will only ever hand out IPs that are part of its configured pools.

Slide 38

Slide 38 text

IPVS Node IPVS Node watches Kubernetes Services, calls netlink and libipvs to create IPVS rules accordingly and syncs ipvs rules with Kubernetes Services periodically, to make sure IPVS status is consistent with the expectation.

Slide 39

Slide 39 text

IPVS Node Network Topology When creating a LoadBalancer type Service and assign external IP, IPVS Node will do the following things: • Make sure a interface exists in the node, using bind-iface flag to specify. • Call go-netlink lib to bind Service external IP(VIP) addresses to the interface. • Call go-libipvs Create IPVS virtual servers for each Service external IP address respectively(Forward VIP to cluster IP). This rule wiil be used Round-robin algorithm.

Slide 40

Slide 40 text

Example # ip addr show dev enp6s0 2: enp6s0: mtu 1500 qdisc pfifo_fast state UP group default qlen 1000 link/ether 54:a0:50:85:d5:23 brd ff:ff:ff:ff:ff:ff inet 172.22.132.12/24 brd 172.22.132.255 scope global enp6s0 valid_lft forever preferred_lft forever inet 172.22.132.200/32 brd 172.22.132.255 scope global enp6s0:0 valid_lft forever preferred_lft forever inet6 fe80::56a0:50ff:fe85:d523/64 scope link valid_lft forever preferred_lft forever

Slide 41

Slide 41 text

Example(Cont.) # ipvsadm -L IP Virtual Server version 1.2.1 (size=4096) Prot LocalAddress:Port Scheduler Flags -> RemoteAddress:Port Forward Weight ActiveConn InActConn TCP 172.22.132.200:http rr -> 10.105.177.190:http Route 1 0 0

Slide 42

Slide 42 text

IPVS-based Kube-proxy

Slide 43

Slide 43 text

IPVS-based Kube-proxy Network Topology When creating a ClusterIP or LoadBalancer type Service and assign external IP, IPVS proxier will do the following things: • Make sure a dummy interface exists in the nodes, defaults to kube-ipvs0. • Bind Service IP addresses(Cluster IP and External IP) to the dummy interface. • Create IPVS virtual servers for each Service IP address respectively.

Slide 44

Slide 44 text

Example # ip addr show kube-ipvs0 14: kube-ipvs0: mtu 1500 qdisc noop state DOWN group default link/ether 3a:eb:de:45:db:92 brd ff:ff:ff:ff:ff:ff inet 10.96.30.32/32 brd 10.96.30.32 scope global kube-ipvs0 valid_lft forever preferred_lft forever inet 10.100.233.153/32 brd 10.100.233.153 scope global kube-ipvs0 valid_lft forever preferred_lft forever inet 172.22.132.200/32 brd 172.22.132.200 scope global kube-ipvs0 valid_lft forever preferred_lft forever

Slide 45

Slide 45 text

Example(Cont.) # ipvsadm -L IP Virtual Server version 1.2.1 (size=4096) Prot LocalAddress:Port Scheduler Flags -> RemoteAddress:Port Forward Weight ActiveConn InActConn TCP 172.22.132.200:http rr -> 10.244.3.10:http Masq 1 0 0

Slide 46

Slide 46 text

Using kube-proxy replace IPVS Node

Slide 47

Slide 47 text

Live Demo Cloud Native Forum

Slide 48

Slide 48 text

MetalLB MetalLB hooks into your Kubernetes cluster, and provides a network load- balancer implementation. • Address allocation. • External announcement. • Layer 2 mode (ARP for IPv4, NDP for IPv6). • Layer 3 mode (BGP). https://metallb.universe.tf/concepts/

Slide 49

Slide 49 text

Refers • https://metallb.universe.tf/concepts/ • https://kubernetes.io/docs/concepts/extend-kubernetes/api-extension/ • http://www.linuxvirtualserver.org/software/ipvs.html • https://kubernetes.io/blog/2018/07/09/ipvs-based-in-cluster-load-balancing-deep- dive/ • https://blog.couchbase.com/kubernetes-operators-game-changer/ • https://github.com/kubernetes/kubernetes/tree/master/pkg/proxy/ipvs • https://bestsamina.github.io/posts/2018-10-19-ipvs-based-kube-proxy-4-scaled-k8s-lb/ • https://github.com/vishvananda/netlink

Slide 50

Slide 50 text

Thanks you!!! Cloud Native Forum