IPVS-based Kube-Proxy for Scaled Kubernetes Load Balancing

Slide 1

Slide 1 text

IPVS-based Kube-Proxy for Scaled Kubernetes Load Balancing 2018/10/19, Kubernetes seminar, Taiwan Presenter: Shan-Jung Fu (Samina)

Slide 2

Slide 2 text

Slide 3

Slide 3 text

Copyright 2015 ITRI 工業技術研究院 Who am I ● A master student in computer science at NCTU ● Intern at ITRI ● Interested in Could, Network & Open Source Technical ● CDNJS member, TGmeetup author ● You can find me at ○ GitHub, Telegram: @sufuf3 ○ Twitter: @sufuf3149 3

Slide 4

Slide 4 text

Slide 5

Slide 5 text

Slide 6

Slide 6 text

Slide 7

Slide 7 text

Slide 8

Slide 8 text

Slide 9

Slide 9 text

Slide 10

Slide 10 text

Slide 11

Slide 11 text

Slide 12

Slide 12 text

Slide 13

Slide 13 text

Copyright 2015 ITRI 工業技術研究院 Outline ● Preface ● Introduction ● kube-proxy ○ What is kube-proxy ○ kube-Proxy mode ● IPVS ● IPVS-based kube-proxy ● Implement IPVS-based K8s service load balancing ● Conclusion 13

Slide 14

Slide 14 text

Slide 15

Slide 15 text

Copyright 2015 ITRI 工業技術研究院 What is Kube-proxy (Cont.) ● Service ○ An abstraction ○ Defines a logical set of Pods & a policy by which to access them kube-proxy is responsible for implementing a form of virtual IP for Services of type (ClusterIP, LoadBalancer, NodePort) ● What is Kube-proxy ● Kube-Proxy mode Source: https://kubernetes.io/docs/tutorials/kubernetes-basics/expose/expose-intro/ 15

Slide 16

Slide 16 text

Slide 17

Slide 17 text

Copyright 2015 ITRI 工業技術研究院 Source: https://kubernetes.io/docs/concepts/services-networking/service/#virtual-ips-and-service-proxies kube-proxy mode ● What is Kube-proxy ● Kube-proxy mode userspace iptables 17

Slide 18

Slide 18 text

Slide 19

Slide 19 text

Copyright 2015 ITRI 工業技術研究院 Outline ● Preface ● Introduction ● Kube-proxy ● IPVS ○ LVS ○ What is IPVS ○ IPVS vs iptables ● IPVS-based Kube-proxy ● Implement IPVS-based K8s service load balancing ● Conclusion 19

Slide 20

Slide 20 text

Copyright 2015 ITRI 工業技術研究院 ● Linux Virtual Server ● Highly scalable & highly available server w/ load balancer ● Provides good ○ Scalability ○ Reliability ○ Serviceability LVS Source: http://www.linuxvirtualserver.org/index.html ● LVS ● What is IPVS ● IPVS vs iptables 20

Slide 21

Slide 21 text

Copyright 2015 ITRI 工業技術研究院 LVS Framework ● LVS ● What is IPVS ● IPVS vs iptables Source: http://www.linuxvirtualserver.org/about.html E-Commerce General Network Services Cluster Management KTCPVS IPVS ● An advanced layer-4 load balancing software ● Implemented inside the Linux kernel 21

Slide 22

Slide 22 text

Copyright 2015 ITRI 工業技術研究院 IPVS ● IP Virtual Server ● Implements transport-layer load balancing ● Called Layer-4 switching ● Running on a host at the front of a cluster of real servers ● Direct requests for TCP/UDP based services to the real servers ● Makes services of the real servers to appear as a virtual service on a single IP address ● LVS ● What is IPVS ● IPVS vs iptables 22

Slide 23

Slide 23 text

Copyright 2015 ITRI 工業技術研究院 IPVS (Cont.) ● Implemented as a module over the Netfilter framework ● Based on in-kernel hash tables ● Kernel source code: net/netfilter/ipvs ● ipvsadm: Linux Virtual Server administration ● Supported features: ○ 2 protocols (TCP and UDP) ○ 3 packet-forwarding methods (NAT, tunneling, and direct routing) ○ 8 load balancing algorithms (round robin, weighted round robin, least-connection, weighted east-connection, locality-based least-connection, locality-based least-connection with replication, destination-hashing, and source-hashing) ● LVS ● What is IPVS ● IPVS vs iptables 23

Slide 24

Slide 24 text

Copyright 2015 ITRI 工業技術研究院 IPVS with Netfilter ● LVS ● What is IPVS ● IPVS vs iptables http://www.austintek.com/LVS/LVS-HOWTO/HOWTO/LVS-HOWTO.filter_rules.html PREROUTING LOCAL_IN POSTROUTING LOCAL_OUT FORWARD Route ip_vs_in Network Kernel Space User Space local process - ipvsadm local process 24

Slide 25

Slide 25 text

Copyright 2015 ITRI 工業技術研究院 IPVS vs iptables IPVS ● LVS ● What is IPVS ● IPVS vs iptables • Build on top of Netfilter • Support Load balance • Better performance with Hash table( vs Chain) iptables • Build on top of Netfilter • As Linux firewall - more operations • Not better performance with Chain & rule ▪ Latency to add/remove rule ▪ Routing latency Ref: https://www.slideshare.net/LCChina/scale-kubernetes-to-support-50000-services 25

Slide 26

Slide 26 text

Copyright 2015 ITRI 工業技術研究院 IPVS vs iptables IPVS (hash table) ● LVS ● What is IPVS ● IPVS vs iptables iptables Source: https://en.wikipedia.org/wiki/Hash_table , https://www.thegeekstuff.com/2011/01/iptables-fundamentals/ 26

Slide 27

Slide 27 text

Slide 28

Slide 28 text

Copyright 2015 ITRI 工業技術研究院 IPVS vs iptables (Cont.) ● LVS ● What is IPVS ● IPVS vs iptables From:https://www.slideshare.net/LCChina/scale-kubernetes-to-support-50000-services Network Bandwidth ith service first first last first last first last first last first last # of services 1 1000 1000 5000 5000 10000 10000 25000 25000 50000 50000 Bandwidth, IPTables (MB/S) 66.6 64 56 50 38.6 15 6 0 0 0 0 Bandwidth, IPVS (MB/S) 65.3 61.7 55.3 53.5 53.8 43 43.5 30 28.5 24 23.8 28

Slide 29

Slide 29 text

Copyright 2015 ITRI 工業技術研究院 Outline ● Preface ● Introduction ● Kube-proxy ● IPVS ● IPVS-based Kube-proxy ○ Why using IPVS? ○ How IPVS-based Kube-proxy work? ○ Run Kube-proxy in IPVS mode ○ IPVS Service Network Topology ○ Example ● Implement IPVS-based K8s service load balancing ● Conclusion 29

Slide 30

Slide 30 text

Copyright 2015 ITRI 工業技術研究院 Why using IPVS? ● Better performance (Hashing vs. Chain) ● More load balancing algorithm ○ Round robin, source/destination hashing. ○ Based on least load, least connection or locality, can assign weight to server. ● Support server health check and connection retry ● Support sticky session ● Iptables operations slow down dramatically in large scale cluster From: https://www.cncf.io/wp-content/uploads/2018/08/CNCF-Webinar_-Kubernetes-1.11-1.pdf ● Why using IPVS? ● IPVS-based Kube-proxy work ● Run IPVS-based Kube-proxy ● IPVS Service Network Topology ● Example 30

Slide 31

Slide 31 text

Slide 32

Slide 32 text

Copyright 2015 ITRI 工業技術研究院 ● Load required kernel modules ○ ip_vs, ip_vs_rr, ip_vs_wrr, ip_vs_sh, nf_conntrack_ipv4 ● Switch proxy mode to IPVS ○ --proxy-mode=ipvs ● Enable feature gateway before v1.10 ○ --feature-gates=SupportIPVSProxyMode=true From:https://www.cncf.io/wp-content/uploads/2018/08/CNCF-Webinar_-Kubernetes-1.11-1.pdf Run Kube-proxy in IPVS mode 32

Slide 33

Slide 33 text

Copyright 2015 ITRI 工業技術研究院 IPVS Service Network Topology ● When creating a ClusterIP type Service, IPVS proxier will do the following 3 things: ○ Make sure a dummy interface exists in the node, defaults to kube-ipvs0 ○ Bind Service IP addresses to the dummy interface ○ Create IPVS virtual servers for each Service IP address respectively From: https://www.cncf.io/wp-content/uploads/2018/08/CNCF-Webinar_-Kubernetes-1.11-1.pdf 33

Slide 34

Slide 34 text

Copyright 2015 ITRI 工業技術研究院 # kubectl describe svc nginx -n a-ns Name: nginx Namespace: a-ns Labels: run=nginx Annotations: Selector: run=nginx Type: ClusterIP IP: 10.105.12.124 External IPs: 100.67.151.9 Port: 80/TCP TargetPort: 80/TCP Endpoints: 10.244.241.156:80,10.244.241.158:80 Session Affinity: None Events: Example ● Why using IPVS? ● IPVS-based Kube-proxy work ● Run IPVS-based Kube-proxy ● IPVS Service Network Topology ● Example 34

Slide 35

Slide 35 text

Copyright 2015 ITRI 工業技術研究院 kube-proxy@v1.10.x # ip a 2: eth0: mtu 1500 qdisc mq state UP group default qlen 1000 link/ether 00:26:2d:08:03:a4 brd ff:ff:ff:ff:ff:ff inet 100.67.151.2/16 brd 100.67.255.255 scope global noprefixroute eth0 valid_lft forever preferred_lft forever inet 100.67.151.9/16 brd 100.67.255.255 scope global secondary eth0:1 valid_lft forever preferred_lft forever 18: kube-ipvs0: mtu 1500 qdisc noop state DOWN group default link/ether e6:f5:f6:9f:0b:9a brd ff:ff:ff:ff:ff:ff inet 10.96.0.1/32 brd 10.96.0.1 scope global kube-ipvs0 valid_lft forever preferred_lft forever inet 10.96.0.10/32 brd 10.96.0.10 scope global kube-ipvs0 valid_lft forever preferred_lft forever inet 10.105.12.124/32 brd 10.105.12.124 scope global kube-ipvs0 valid_lft forever preferred_lft forever Example (Cont.) ● Why using IPVS? ● IPVS-based Kube-proxy work ● Run IPVS-based Kube-proxy ● IPVS Service Network Topology ● Example 35

Slide 36

Slide 36 text

Copyright 2015 ITRI 工業技術研究院 kube-proxy@v1.11.x # ip a 2: eth0: mtu 1500 qdisc mq state UP group default qlen 1000 link/ether 00:26:2d:08:03:a4 brd ff:ff:ff:ff:ff:ff inet 100.67.151.2/16 brd 100.67.255.255 scope global noprefixroute eth0 valid_lft forever preferred_lft forever 18: kube-ipvs0: mtu 1500 qdisc noop state DOWN group default link/ether e6:f5:f6:9f:0b:9a brd ff:ff:ff:ff:ff:ff inet 10.96.0.1/32 brd 10.96.0.1 scope global kube-ipvs0 valid_lft forever preferred_lft forever inet 10.96.0.10/32 brd 10.96.0.10 scope global kube-ipvs0 valid_lft forever preferred_lft forever inet 10.105.12.124/32 brd 10.105.12.124 scope global kube-ipvs0 valid_lft forever preferred_lft forever inet 100.67.151.9/16 brd 100.67.255.255 scope global kube-ipvs0 valid_lft forever preferred_lft forever Example (Cont.) ● Why using IPVS? ● IPVS-based Kube-proxy work ● Run IPVS-based Kube-proxy ● IPVS Service Network Topology ● Example 36

Slide 37

Slide 37 text

Copyright 2015 ITRI 工業技術研究院 # ipvsadm -ln IP Virtual Server version 1.2.1 (size=4096) Prot LocalAddress:Port Scheduler Flags -> RemoteAddress:Port Forward Weight ActiveConn InActConn TCP 10.105.12.124:80 rr -> 10.244.241.156:80 Masq 1 0 0 -> 10.244.241.158:80 Masq 1 0 0 TCP 100.67.151.9:80 rr -> 10.244.241.156:80 Masq 1 0 0 -> 10.244.241.158:80 Masq 1 0 0 Example (Cont.) ● Why using IPVS? ● IPVS-based Kube-proxy work ● Run IPVS-based Kube-proxy ● IPVS Service Network Topology ● Example 37

Slide 38

Slide 38 text

Slide 39

Slide 39 text

Copyright 2015 ITRI 工業技術研究院 Implement IPVS-based K8s service load balancing Steps: 1. Load IPVS kernel modules 2. Deploy k8s with IPVS mode of kube-proxy 3. Create deployments 4. Create service 5. Bind External IP on network interface ○ kube-proxy@1.11.x will do this by itself ○ people bind manually with kube-proxy@v1.10.x Blog: http://bit.ly/2J1ZX33 39

Slide 40

Slide 40 text

Slide 41

Slide 41 text

Copyright 2015 ITRI 工業技術研究院 ● IPVS is a L4 load balancer in LVS ● IPVS provides ○ better scalability & performance for large clusters ○ more load balancing algorithms than iptables ○ server health checking and connection retries, etc ● We can use IPVS mode of kube-proxy ● Know how IPVS-based Kube-proxy work Conclusion 41