IPVS-based Kube-Proxy for Scaled Kubernetes Load Balancing

- IPVS Intro
- Know how to use IPVS-based kube-proxy
- Know how to implement Kubernetes service load balancing via IPVS-based kube-proxy

Blog: https://bestsamina.github.io/posts/2018-10-19-ipvs-based-kube-proxy-4-scaled-k8s-lb/

Transcript

1. IPVS-based Kube-Proxy for Scaled Kubernetes Load Balancing 2018/10/19, Kubernetes seminar, Taiwan Presenter: Shan-Jung Fu (Samina)
2. Copyright 2015 ITRI 工業技術研究院 Today's slides Slides bit.ly/1019ipvs Blog bit.ly/1019blog 2
3. Copyright 2015 ITRI 工業技術研究院 Who am I • A master student in computer science at NCTU • Intern at ITRI • Interested in Could, Network & Open Source Technical • CDNJS member, TGmeetup author • You can find me at ◦ GitHub, Telegram: @sufuf3 ◦ Twitter: @sufuf3149 3
4. Copyright 2015 ITRI 工業技術研究院 Outline • Preface • Introduction • kube-proxy • IPVS • IPVS-based kube-proxy • Implement IPVS-based K8s service load balancing • Conclusion 4
5. Copyright 2015 ITRI 工業技術研究院 Outline • Preface • Introduction • kube-proxy • IPVS • IPVS-based kube-proxy • Implement IPVS-based K8s service load balancing • Conclusion 5
6. Copyright 2015 ITRI 工業技術研究院 Preface 6
7. Copyright 2015 ITRI 工業技術研究院 Preface (Cont.) Why Tim Hockin said “Try kube-proxy via IPVS”? 7
8. Copyright 2015 ITRI 工業技術研究院 Preface (Cont.) 8
9. Copyright 2015 ITRI 工業技術研究院 Preface (Cont.) 9
10. Copyright 2015 ITRI 工業技術研究院 Preface (Cont.) • What is IPVS? • Why kube-proxy via IPVS is better than iptables mode? • How to use kube-proxy via IPVS to implement Kubernetes service load balancing? 10
11. Copyright 2015 ITRI 工業技術研究院 Outline • Preface • Introduction • kube-proxy • IPVS • IPVS-based kube-proxy • Implement IPVS-based K8s service load balancing • Conclusion 11
12. Copyright 2015 ITRI 工業技術研究院 Introduction Targets: • What is IPVS? • How to use IPVS-based kube-proxy? • How to implement Kubernetes service load balancing via IPVS-based kube-proxy? 12
13. Copyright 2015 ITRI 工業技術研究院 Outline • Preface • Introduction • kube-proxy ◦ What is kube-proxy ◦ kube-Proxy mode • IPVS • IPVS-based kube-proxy • Implement IPVS-based K8s service load balancing • Conclusion 13
14. Copyright 2015 ITRI 工業技術研究院 What is kube-proxy • What is Kube-proxy • Kube-Proxy mode Source: https://kubernetes.io/docs/concepts/architecture/cloud-controller/ 14
15. Copyright 2015 ITRI 工業技術研究院 What is Kube-proxy (Cont.) • Service ◦ An abstraction ◦ Defines a logical set of Pods & a policy by which to access them kube-proxy is responsible for implementing a form of virtual IP for Services of type (ClusterIP, LoadBalancer, NodePort) • What is Kube-proxy • Kube-Proxy mode Source: https://kubernetes.io/docs/tutorials/kubernetes-basics/expose/expose-intro/ 15
16. Copyright 2015 ITRI 工業技術研究院 kube-proxy • Runs on each node • Proxies UDP, TCP and SCTP • Provides load balancing • Is just used to reach services • What is Kube-proxy • Kube-Proxy mode 16
17. Copyright 2015 ITRI 工業技術研究院 Source: https://kubernetes.io/docs/concepts/services-networking/service/#virtual-ips-and-service-proxies kube-proxy mode • What is Kube-proxy • Kube-proxy mode userspace iptables 17
18. Copyright 2015 ITRI 工業技術研究院 https://kubernetes.io/docs/concepts/services-networking/service/#virtual-ips-and-service-proxies kube-proxy mode (Cont.) • What is Kube-proxy • Kube-proxy mode IPVS 18
19. Copyright 2015 ITRI 工業技術研究院 Outline • Preface • Introduction • Kube-proxy • IPVS ◦ LVS ◦ What is IPVS ◦ IPVS vs iptables • IPVS-based Kube-proxy • Implement IPVS-based K8s service load balancing • Conclusion 19
20. Copyright 2015 ITRI 工業技術研究院 • Linux Virtual Server • Highly scalable & highly available server w/ load balancer • Provides good ◦ Scalability ◦ Reliability ◦ Serviceability LVS Source: http://www.linuxvirtualserver.org/index.html • LVS • What is IPVS • IPVS vs iptables 20
21. Copyright 2015 ITRI 工業技術研究院 LVS Framework • LVS • What is IPVS • IPVS vs iptables Source: http://www.linuxvirtualserver.org/about.html E-Commerce General Network Services Cluster Management KTCPVS IPVS • An advanced layer-4 load balancing software • Implemented inside the Linux kernel 21
22. Copyright 2015 ITRI 工業技術研究院 IPVS • IP Virtual Server • Implements transport-layer load balancing • Called Layer-4 switching • Running on a host at the front of a cluster of real servers • Direct requests for TCP/UDP based services to the real servers • Makes services of the real servers to appear as a virtual service on a single IP address • LVS • What is IPVS • IPVS vs iptables 22
23. Copyright 2015 ITRI 工業技術研究院 IPVS (Cont.) • Implemented as a module over the Netfilter framework • Based on in-kernel hash tables • Kernel source code: net/netfilter/ipvs • ipvsadm: Linux Virtual Server administration • Supported features: ◦ 2 protocols (TCP and UDP) ◦ 3 packet-forwarding methods (NAT, tunneling, and direct routing) ◦ 8 load balancing algorithms (round robin, weighted round robin, least-connection, weighted east-connection, locality-based least-connection, locality-based least-connection with replication, destination-hashing, and source-hashing) • LVS • What is IPVS • IPVS vs iptables 23
24. Copyright 2015 ITRI 工業技術研究院 IPVS with Netfilter • LVS • What is IPVS • IPVS vs iptables http://www.austintek.com/LVS/LVS-HOWTO/HOWTO/LVS-HOWTO.filter_rules.html PREROUTING LOCAL_IN POSTROUTING LOCAL_OUT FORWARD Route ip_vs_in Network Kernel Space User Space local process - ipvsadm local process 24
25. Copyright 2015 ITRI 工業技術研究院 IPVS vs iptables IPVS • LVS • What is IPVS • IPVS vs iptables • Build on top of Netfilter • Support Load balance • Better performance with Hash table( vs Chain) iptables • Build on top of Netfilter • As Linux firewall - more operations • Not better performance with Chain & rule ▪ Latency to add/remove rule ▪ Routing latency Ref: https://www.slideshare.net/LCChina/scale-kubernetes-to-support-50000-services 25
26. Copyright 2015 ITRI 工業技術研究院 IPVS vs iptables IPVS (hash table) • LVS • What is IPVS • IPVS vs iptables iptables Source: https://en.wikipedia.org/wiki/Hash_table , https://www.thegeekstuff.com/2011/01/iptables-fundamentals/ 26
27. Copyright 2015 ITRI 工業技術研究院 IPVS vs iptables (Cont.) • LVS • What is IPVS • IPVS vs iptables From:https://www.slideshare.net/LCChina/scale-kubernetes-to-support-50000-services Latency to add rules # of Services 1 5,000 20,000 # of Rules 8 40,000 160,000 iptables 2 ms 11 min 5 hours IPVS 2 ms 2 ms 2 ms 27
28. Copyright 2015 ITRI 工業技術研究院 IPVS vs iptables (Cont.) • LVS • What is IPVS • IPVS vs iptables From:https://www.slideshare.net/LCChina/scale-kubernetes-to-support-50000-services Network Bandwidth ith service first first last first last first last first last first last # of services 1 1000 1000 5000 5000 10000 10000 25000 25000 50000 50000 Bandwidth, IPTables (MB/S) 66.6 64 56 50 38.6 15 6 0 0 0 0 Bandwidth, IPVS (MB/S) 65.3 61.7 55.3 53.5 53.8 43 43.5 30 28.5 24 23.8 28
29. Copyright 2015 ITRI 工業技術研究院 Outline • Preface • Introduction • Kube-proxy • IPVS • IPVS-based Kube-proxy ◦ Why using IPVS? ◦ How IPVS-based Kube-proxy work? ◦ Run Kube-proxy in IPVS mode ◦ IPVS Service Network Topology ◦ Example • Implement IPVS-based K8s service load balancing • Conclusion 29
30. Copyright 2015 ITRI 工業技術研究院 Why using IPVS? • Better performance (Hashing vs. Chain) • More load balancing algorithm ◦ Round robin, source/destination hashing. ◦ Based on least load, least connection or locality, can assign weight to server. • Support server health check and connection retry • Support sticky session • Iptables operations slow down dramatically in large scale cluster From: https://www.cncf.io/wp-content/uploads/2018/08/CNCF-Webinar_-Kubernetes-1.11-1.pdf • Why using IPVS? • IPVS-based Kube-proxy work • Run IPVS-based Kube-proxy • IPVS Service Network Topology • Example 30
31. Copyright 2015 ITRI 工業技術研究院 How IPVS-based Kube-proxy work? 31
32. Copyright 2015 ITRI 工業技術研究院 • Load required kernel modules ◦ ip_vs, ip_vs_rr, ip_vs_wrr, ip_vs_sh, nf_conntrack_ipv4 • Switch proxy mode to IPVS ◦ --proxy-mode=ipvs • Enable feature gateway before v1.10 ◦ --feature-gates=SupportIPVSProxyMode=true From:https://www.cncf.io/wp-content/uploads/2018/08/CNCF-Webinar_-Kubernetes-1.11-1.pdf Run Kube-proxy in IPVS mode 32
33. Copyright 2015 ITRI 工業技術研究院 IPVS Service Network Topology • When creating a ClusterIP type Service, IPVS proxier will do the following 3 things: ◦ Make sure a dummy interface exists in the node, defaults to kube-ipvs0 ◦ Bind Service IP addresses to the dummy interface ◦ Create IPVS virtual servers for each Service IP address respectively From: https://www.cncf.io/wp-content/uploads/2018/08/CNCF-Webinar_-Kubernetes-1.11-1.pdf 33
34. Copyright 2015 ITRI 工業技術研究院 # kubectl describe svc nginx -n a-ns Name: nginx Namespace: a-ns Labels: run=nginx Annotations: <none> Selector: run=nginx Type: ClusterIP IP: 10.105.12.124 External IPs: 100.67.151.9 Port: <unset> 80/TCP TargetPort: 80/TCP Endpoints: 10.244.241.156:80,10.244.241.158:80 Session Affinity: None Events: <none> Example • Why using IPVS? • IPVS-based Kube-proxy work • Run IPVS-based Kube-proxy • IPVS Service Network Topology • Example 34
35. Copyright 2015 ITRI 工業技術研究院 kube-proxy@v1.10.x # ip a 2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000 link/ether 00:26:2d:08:03:a4 brd ff:ff:ff:ff:ff:ff inet 100.67.151.2/16 brd 100.67.255.255 scope global noprefixroute eth0 valid_lft forever preferred_lft forever inet 100.67.151.9/16 brd 100.67.255.255 scope global secondary eth0:1 valid_lft forever preferred_lft forever 18: kube-ipvs0: <BROADCAST,NOARP> mtu 1500 qdisc noop state DOWN group default link/ether e6:f5:f6:9f:0b:9a brd ff:ff:ff:ff:ff:ff inet 10.96.0.1/32 brd 10.96.0.1 scope global kube-ipvs0 valid_lft forever preferred_lft forever inet 10.96.0.10/32 brd 10.96.0.10 scope global kube-ipvs0 valid_lft forever preferred_lft forever inet 10.105.12.124/32 brd 10.105.12.124 scope global kube-ipvs0 valid_lft forever preferred_lft forever Example (Cont.) • Why using IPVS? • IPVS-based Kube-proxy work • Run IPVS-based Kube-proxy • IPVS Service Network Topology • Example 35
36. Copyright 2015 ITRI 工業技術研究院 kube-proxy@v1.11.x # ip a 2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000 link/ether 00:26:2d:08:03:a4 brd ff:ff:ff:ff:ff:ff inet 100.67.151.2/16 brd 100.67.255.255 scope global noprefixroute eth0 valid_lft forever preferred_lft forever 18: kube-ipvs0: <BROADCAST,NOARP> mtu 1500 qdisc noop state DOWN group default link/ether e6:f5:f6:9f:0b:9a brd ff:ff:ff:ff:ff:ff inet 10.96.0.1/32 brd 10.96.0.1 scope global kube-ipvs0 valid_lft forever preferred_lft forever inet 10.96.0.10/32 brd 10.96.0.10 scope global kube-ipvs0 valid_lft forever preferred_lft forever inet 10.105.12.124/32 brd 10.105.12.124 scope global kube-ipvs0 valid_lft forever preferred_lft forever inet 100.67.151.9/16 brd 100.67.255.255 scope global kube-ipvs0 valid_lft forever preferred_lft forever Example (Cont.) • Why using IPVS? • IPVS-based Kube-proxy work • Run IPVS-based Kube-proxy • IPVS Service Network Topology • Example 36
37. Copyright 2015 ITRI 工業技術研究院 # ipvsadm -ln IP Virtual Server version 1.2.1 (size=4096) Prot LocalAddress:Port Scheduler Flags -> RemoteAddress:Port Forward Weight ActiveConn InActConn TCP 10.105.12.124:80 rr -> 10.244.241.156:80 Masq 1 0 0 -> 10.244.241.158:80 Masq 1 0 0 TCP 100.67.151.9:80 rr -> 10.244.241.156:80 Masq 1 0 0 -> 10.244.241.158:80 Masq 1 0 0 Example (Cont.) • Why using IPVS? • IPVS-based Kube-proxy work • Run IPVS-based Kube-proxy • IPVS Service Network Topology • Example 37
38. Copyright 2015 ITRI 工業技術研究院 Outline • Preface • Introduction • Kube-proxy • IPVS • IPVS-based Kube-proxy • Implement IPVS-based K8s service load balancing • Conclusion 38
39. Copyright 2015 ITRI 工業技術研究院 Implement IPVS-based K8s service load balancing Steps: 1. Load IPVS kernel modules 2. Deploy k8s with IPVS mode of kube-proxy 3. Create deployments 4. Create service 5. Bind External IP on network interface ◦ kube-proxy@1.11.x will do this by itself ◦ people bind manually with kube-proxy@v1.10.x Blog: http://bit.ly/2J1ZX33 39
40. Copyright 2015 ITRI 工業技術研究院 Outline • Preface • Introduction • Kube-proxy • IPVS • IPVS-based Kube-proxy • Implement IPVS-based K8s service load balancing • Conclusion 40
41. Copyright 2015 ITRI 工業技術研究院 • IPVS is a L4 load balancer in LVS • IPVS provides ◦ better scalability & performance for large clusters ◦ more load balancing algorithms than iptables ◦ server health checking and connection retries, etc • We can use IPVS mode of kube-proxy • Know how IPVS-based Kube-proxy work Conclusion 41
42. Copyright 2015 ITRI 工業技術研究院 Thank you! sufuf3[at]gmail[dot]com Telegram: @sufuf3 Twitter: @sufuf3149 fb.com/groups/cloudnative.tw 42