student in computer science at NCTU • Intern at ITRI • Interested in Could, Network & Open Source Technical • CDNJS member, TGmeetup author • You can find me at ◦ GitHub, Telegram: @sufuf3 ◦ Twitter: @sufuf3149 3
◦ An abstraction ◦ Defines a logical set of Pods & a policy by which to access them kube-proxy is responsible for implementing a form of virtual IP for Services of type (ClusterIP, LoadBalancer, NodePort) • What is Kube-proxy • Kube-Proxy mode Source: https://kubernetes.io/docs/tutorials/kubernetes-basics/expose/expose-intro/ 15
is IPVS • IPVS vs iptables Source: http://www.linuxvirtualserver.org/about.html E-Commerce General Network Services Cluster Management KTCPVS IPVS • An advanced layer-4 load balancing software • Implemented inside the Linux kernel 21
Implements transport-layer load balancing • Called Layer-4 switching • Running on a host at the front of a cluster of real servers • Direct requests for TCP/UDP based services to the real servers • Makes services of the real servers to appear as a virtual service on a single IP address • LVS • What is IPVS • IPVS vs iptables 22
What is IPVS • IPVS vs iptables http://www.austintek.com/LVS/LVS-HOWTO/HOWTO/LVS-HOWTO.filter_rules.html PREROUTING LOCAL_IN POSTROUTING LOCAL_OUT FORWARD Route ip_vs_in Network Kernel Space User Space local process - ipvsadm local process 24
• What is IPVS • IPVS vs iptables • Build on top of Netfilter • Support Load balance • Better performance with Hash table( vs Chain) iptables • Build on top of Netfilter • As Linux firewall - more operations • Not better performance with Chain & rule ▪ Latency to add/remove rule ▪ Routing latency Ref: https://www.slideshare.net/LCChina/scale-kubernetes-to-support-50000-services 25
• What is IPVS • IPVS vs iptables From:https://www.slideshare.net/LCChina/scale-kubernetes-to-support-50000-services Latency to add rules # of Services 1 5,000 20,000 # of Rules 8 40,000 160,000 iptables 2 ms 11 min 5 hours IPVS 2 ms 2 ms 2 ms 27
• What is IPVS • IPVS vs iptables From:https://www.slideshare.net/LCChina/scale-kubernetes-to-support-50000-services Network Bandwidth ith service first first last first last first last first last first last # of services 1 1000 1000 5000 5000 10000 10000 25000 25000 50000 50000 Bandwidth, IPTables (MB/S) 66.6 64 56 50 38.6 15 6 0 0 0 0 Bandwidth, IPVS (MB/S) 65.3 61.7 55.3 53.5 53.8 43 43.5 30 28.5 24 23.8 28
(Hashing vs. Chain) • More load balancing algorithm ◦ Round robin, source/destination hashing. ◦ Based on least load, least connection or locality, can assign weight to server. • Support server health check and connection retry • Support sticky session • Iptables operations slow down dramatically in large scale cluster From: https://www.cncf.io/wp-content/uploads/2018/08/CNCF-Webinar_-Kubernetes-1.11-1.pdf • Why using IPVS? • IPVS-based Kube-proxy work • Run IPVS-based Kube-proxy • IPVS Service Network Topology • Example 30
creating a ClusterIP type Service, IPVS proxier will do the following 3 things: ◦ Make sure a dummy interface exists in the node, defaults to kube-ipvs0 ◦ Bind Service IP addresses to the dummy interface ◦ Create IPVS virtual servers for each Service IP address respectively From: https://www.cncf.io/wp-content/uploads/2018/08/CNCF-Webinar_-Kubernetes-1.11-1.pdf 33
Steps: 1. Load IPVS kernel modules 2. Deploy k8s with IPVS mode of kube-proxy 3. Create deployments 4. Create service 5. Bind External IP on network interface ◦ kube-proxy@1.11.x will do this by itself ◦ people bind manually with kube-proxy@v1.10.x Blog: http://bit.ly/2J1ZX33 39
balancer in LVS • IPVS provides ◦ better scalability & performance for large clusters ◦ more load balancing algorithms than iptables ◦ server health checking and connection retries, etc • We can use IPVS mode of kube-proxy • Know how IPVS-based Kube-proxy work Conclusion 41
sufuf3
kamata_shingo
nulabinc
oracle4engineer
okepy
qsona
unchamacom
yulstat
shirayanagiryuji
mouson
sunaotabata
yukiiiiikuma
sadnessojisan
pedronauck
paulrobertlloyd
maltzj
robhawkes
marcelosomers
mojombo
lara
productmarketing
sferik
samanthasiow
mongodb
![Copyright 2015 ITRI 工業技術研究院 Thank you! sufuf3[at]gmail[dot]com Telegram: @sufuf3 Twitter:](https://files.speakerdeck.com/presentations/f37bb4704eee418cbb45999743c4d3d2/slide_41.jpg){kind=link}