Upgrade to Pro — share decks privately, control downloads, hide ads and more …

IPVS-based Kube-Proxy for Scaled Kubernetes Loa...

IPVS-based Kube-Proxy for Scaled Kubernetes Load Balancing

- IPVS Intro
- Know how to use IPVS-based kube-proxy
- Know how to implement Kubernetes service load balancing via IPVS-based kube-proxy

Blog: https://bestsamina.github.io/posts/2018-10-19-ipvs-based-kube-proxy-4-scaled-k8s-lb/

Avatar for Samina (Shan Jung Fu)

Samina (Shan Jung Fu)

October 19, 2018
Tweet

More Decks by Samina (Shan Jung Fu)

Other Decks in Technology

Transcript

  1. Copyright 2015 ITRI 工業技術研究院 Who am I • A master

    student in computer science at NCTU • Intern at ITRI • Interested in Could, Network & Open Source Technical • CDNJS member, TGmeetup author • You can find me at ◦ GitHub, Telegram: @sufuf3 ◦ Twitter: @sufuf3149 3
  2. Copyright 2015 ITRI 工業技術研究院 Outline • Preface • Introduction •

    kube-proxy • IPVS • IPVS-based kube-proxy • Implement IPVS-based K8s service load balancing • Conclusion 4
  3. Copyright 2015 ITRI 工業技術研究院 Outline • Preface • Introduction •

    kube-proxy • IPVS • IPVS-based kube-proxy • Implement IPVS-based K8s service load balancing • Conclusion 5
  4. Copyright 2015 ITRI 工業技術研究院 Preface (Cont.) • What is IPVS?

    • Why kube-proxy via IPVS is better than iptables mode? • How to use kube-proxy via IPVS to implement Kubernetes service load balancing? 10
  5. Copyright 2015 ITRI 工業技術研究院 Outline • Preface • Introduction •

    kube-proxy • IPVS • IPVS-based kube-proxy • Implement IPVS-based K8s service load balancing • Conclusion 11
  6. Copyright 2015 ITRI 工業技術研究院 Introduction Targets: • What is IPVS?

    • How to use IPVS-based kube-proxy? • How to implement Kubernetes service load balancing via IPVS-based kube-proxy? 12
  7. Copyright 2015 ITRI 工業技術研究院 Outline • Preface • Introduction •

    kube-proxy ◦ What is kube-proxy ◦ kube-Proxy mode • IPVS • IPVS-based kube-proxy • Implement IPVS-based K8s service load balancing • Conclusion 13
  8. Copyright 2015 ITRI 工業技術研究院 What is kube-proxy • What is

    Kube-proxy • Kube-Proxy mode Source: https://kubernetes.io/docs/concepts/architecture/cloud-controller/ 14
  9. Copyright 2015 ITRI 工業技術研究院 What is Kube-proxy (Cont.) • Service

    ◦ An abstraction ◦ Defines a logical set of Pods & a policy by which to access them kube-proxy is responsible for implementing a form of virtual IP for Services of type (ClusterIP, LoadBalancer, NodePort) • What is Kube-proxy • Kube-Proxy mode Source: https://kubernetes.io/docs/tutorials/kubernetes-basics/expose/expose-intro/ 15
  10. Copyright 2015 ITRI 工業技術研究院 kube-proxy • Runs on each node

    • Proxies UDP, TCP and SCTP • Provides load balancing • Is just used to reach services • What is Kube-proxy • Kube-Proxy mode 16
  11. Copyright 2015 ITRI 工業技術研究院 Outline • Preface • Introduction •

    Kube-proxy • IPVS ◦ LVS ◦ What is IPVS ◦ IPVS vs iptables • IPVS-based Kube-proxy • Implement IPVS-based K8s service load balancing • Conclusion 19
  12. Copyright 2015 ITRI 工業技術研究院 • Linux Virtual Server • Highly

    scalable & highly available server w/ load balancer • Provides good ◦ Scalability ◦ Reliability ◦ Serviceability LVS Source: http://www.linuxvirtualserver.org/index.html • LVS • What is IPVS • IPVS vs iptables 20
  13. Copyright 2015 ITRI 工業技術研究院 LVS Framework • LVS • What

    is IPVS • IPVS vs iptables Source: http://www.linuxvirtualserver.org/about.html E-Commerce General Network Services Cluster Management KTCPVS IPVS • An advanced layer-4 load balancing software • Implemented inside the Linux kernel 21
  14. Copyright 2015 ITRI 工業技術研究院 IPVS • IP Virtual Server •

    Implements transport-layer load balancing • Called Layer-4 switching • Running on a host at the front of a cluster of real servers • Direct requests for TCP/UDP based services to the real servers • Makes services of the real servers to appear as a virtual service on a single IP address • LVS • What is IPVS • IPVS vs iptables 22
  15. Copyright 2015 ITRI 工業技術研究院 IPVS (Cont.) • Implemented as a

    module over the Netfilter framework • Based on in-kernel hash tables • Kernel source code: net/netfilter/ipvs • ipvsadm: Linux Virtual Server administration • Supported features: ◦ 2 protocols (TCP and UDP) ◦ 3 packet-forwarding methods (NAT, tunneling, and direct routing) ◦ 8 load balancing algorithms (round robin, weighted round robin, least-connection, weighted east-connection, locality-based least-connection, locality-based least-connection with replication, destination-hashing, and source-hashing) • LVS • What is IPVS • IPVS vs iptables 23
  16. Copyright 2015 ITRI 工業技術研究院 IPVS with Netfilter • LVS •

    What is IPVS • IPVS vs iptables http://www.austintek.com/LVS/LVS-HOWTO/HOWTO/LVS-HOWTO.filter_rules.html PREROUTING LOCAL_IN POSTROUTING LOCAL_OUT FORWARD Route ip_vs_in Network Kernel Space User Space local process - ipvsadm local process 24
  17. Copyright 2015 ITRI 工業技術研究院 IPVS vs iptables IPVS • LVS

    • What is IPVS • IPVS vs iptables • Build on top of Netfilter • Support Load balance • Better performance with Hash table( vs Chain) iptables • Build on top of Netfilter • As Linux firewall - more operations • Not better performance with Chain & rule ▪ Latency to add/remove rule ▪ Routing latency Ref: https://www.slideshare.net/LCChina/scale-kubernetes-to-support-50000-services 25
  18. Copyright 2015 ITRI 工業技術研究院 IPVS vs iptables IPVS (hash table)

    • LVS • What is IPVS • IPVS vs iptables iptables Source: https://en.wikipedia.org/wiki/Hash_table , https://www.thegeekstuff.com/2011/01/iptables-fundamentals/ 26
  19. Copyright 2015 ITRI 工業技術研究院 IPVS vs iptables (Cont.) • LVS

    • What is IPVS • IPVS vs iptables From:https://www.slideshare.net/LCChina/scale-kubernetes-to-support-50000-services Latency to add rules # of Services 1 5,000 20,000 # of Rules 8 40,000 160,000 iptables 2 ms 11 min 5 hours IPVS 2 ms 2 ms 2 ms 27
  20. Copyright 2015 ITRI 工業技術研究院 IPVS vs iptables (Cont.) • LVS

    • What is IPVS • IPVS vs iptables From:https://www.slideshare.net/LCChina/scale-kubernetes-to-support-50000-services Network Bandwidth ith service first first last first last first last first last first last # of services 1 1000 1000 5000 5000 10000 10000 25000 25000 50000 50000 Bandwidth, IPTables (MB/S) 66.6 64 56 50 38.6 15 6 0 0 0 0 Bandwidth, IPVS (MB/S) 65.3 61.7 55.3 53.5 53.8 43 43.5 30 28.5 24 23.8 28
  21. Copyright 2015 ITRI 工業技術研究院 Outline • Preface • Introduction •

    Kube-proxy • IPVS • IPVS-based Kube-proxy ◦ Why using IPVS? ◦ How IPVS-based Kube-proxy work? ◦ Run Kube-proxy in IPVS mode ◦ IPVS Service Network Topology ◦ Example • Implement IPVS-based K8s service load balancing • Conclusion 29
  22. Copyright 2015 ITRI 工業技術研究院 Why using IPVS? • Better performance

    (Hashing vs. Chain) • More load balancing algorithm ◦ Round robin, source/destination hashing. ◦ Based on least load, least connection or locality, can assign weight to server. • Support server health check and connection retry • Support sticky session • Iptables operations slow down dramatically in large scale cluster From: https://www.cncf.io/wp-content/uploads/2018/08/CNCF-Webinar_-Kubernetes-1.11-1.pdf • Why using IPVS? • IPVS-based Kube-proxy work • Run IPVS-based Kube-proxy • IPVS Service Network Topology • Example 30
  23. Copyright 2015 ITRI 工業技術研究院 • Load required kernel modules ◦

    ip_vs, ip_vs_rr, ip_vs_wrr, ip_vs_sh, nf_conntrack_ipv4 • Switch proxy mode to IPVS ◦ --proxy-mode=ipvs • Enable feature gateway before v1.10 ◦ --feature-gates=SupportIPVSProxyMode=true From:https://www.cncf.io/wp-content/uploads/2018/08/CNCF-Webinar_-Kubernetes-1.11-1.pdf Run Kube-proxy in IPVS mode 32
  24. Copyright 2015 ITRI 工業技術研究院 IPVS Service Network Topology • When

    creating a ClusterIP type Service, IPVS proxier will do the following 3 things: ◦ Make sure a dummy interface exists in the node, defaults to kube-ipvs0 ◦ Bind Service IP addresses to the dummy interface ◦ Create IPVS virtual servers for each Service IP address respectively From: https://www.cncf.io/wp-content/uploads/2018/08/CNCF-Webinar_-Kubernetes-1.11-1.pdf 33
  25. Copyright 2015 ITRI 工業技術研究院 # kubectl describe svc nginx -n

    a-ns Name: nginx Namespace: a-ns Labels: run=nginx Annotations: <none> Selector: run=nginx Type: ClusterIP IP: 10.105.12.124 External IPs: 100.67.151.9 Port: <unset> 80/TCP TargetPort: 80/TCP Endpoints: 10.244.241.156:80,10.244.241.158:80 Session Affinity: None Events: <none> Example • Why using IPVS? • IPVS-based Kube-proxy work • Run IPVS-based Kube-proxy • IPVS Service Network Topology • Example 34
  26. Copyright 2015 ITRI 工業技術研究院 [email protected] # ip a 2: eth0:

    <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000 link/ether 00:26:2d:08:03:a4 brd ff:ff:ff:ff:ff:ff inet 100.67.151.2/16 brd 100.67.255.255 scope global noprefixroute eth0 valid_lft forever preferred_lft forever inet 100.67.151.9/16 brd 100.67.255.255 scope global secondary eth0:1 valid_lft forever preferred_lft forever 18: kube-ipvs0: <BROADCAST,NOARP> mtu 1500 qdisc noop state DOWN group default link/ether e6:f5:f6:9f:0b:9a brd ff:ff:ff:ff:ff:ff inet 10.96.0.1/32 brd 10.96.0.1 scope global kube-ipvs0 valid_lft forever preferred_lft forever inet 10.96.0.10/32 brd 10.96.0.10 scope global kube-ipvs0 valid_lft forever preferred_lft forever inet 10.105.12.124/32 brd 10.105.12.124 scope global kube-ipvs0 valid_lft forever preferred_lft forever Example (Cont.) • Why using IPVS? • IPVS-based Kube-proxy work • Run IPVS-based Kube-proxy • IPVS Service Network Topology • Example 35
  27. Copyright 2015 ITRI 工業技術研究院 [email protected] # ip a 2: eth0:

    <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000 link/ether 00:26:2d:08:03:a4 brd ff:ff:ff:ff:ff:ff inet 100.67.151.2/16 brd 100.67.255.255 scope global noprefixroute eth0 valid_lft forever preferred_lft forever 18: kube-ipvs0: <BROADCAST,NOARP> mtu 1500 qdisc noop state DOWN group default link/ether e6:f5:f6:9f:0b:9a brd ff:ff:ff:ff:ff:ff inet 10.96.0.1/32 brd 10.96.0.1 scope global kube-ipvs0 valid_lft forever preferred_lft forever inet 10.96.0.10/32 brd 10.96.0.10 scope global kube-ipvs0 valid_lft forever preferred_lft forever inet 10.105.12.124/32 brd 10.105.12.124 scope global kube-ipvs0 valid_lft forever preferred_lft forever inet 100.67.151.9/16 brd 100.67.255.255 scope global kube-ipvs0 valid_lft forever preferred_lft forever Example (Cont.) • Why using IPVS? • IPVS-based Kube-proxy work • Run IPVS-based Kube-proxy • IPVS Service Network Topology • Example 36
  28. Copyright 2015 ITRI 工業技術研究院 # ipvsadm -ln IP Virtual Server

    version 1.2.1 (size=4096) Prot LocalAddress:Port Scheduler Flags -> RemoteAddress:Port Forward Weight ActiveConn InActConn TCP 10.105.12.124:80 rr -> 10.244.241.156:80 Masq 1 0 0 -> 10.244.241.158:80 Masq 1 0 0 TCP 100.67.151.9:80 rr -> 10.244.241.156:80 Masq 1 0 0 -> 10.244.241.158:80 Masq 1 0 0 Example (Cont.) • Why using IPVS? • IPVS-based Kube-proxy work • Run IPVS-based Kube-proxy • IPVS Service Network Topology • Example 37
  29. Copyright 2015 ITRI 工業技術研究院 Outline • Preface • Introduction •

    Kube-proxy • IPVS • IPVS-based Kube-proxy • Implement IPVS-based K8s service load balancing • Conclusion 38
  30. Copyright 2015 ITRI 工業技術研究院 Implement IPVS-based K8s service load balancing

    Steps: 1. Load IPVS kernel modules 2. Deploy k8s with IPVS mode of kube-proxy 3. Create deployments 4. Create service 5. Bind External IP on network interface ◦ [email protected] will do this by itself ◦ people bind manually with [email protected] Blog: http://bit.ly/2J1ZX33 39
  31. Copyright 2015 ITRI 工業技術研究院 Outline • Preface • Introduction •

    Kube-proxy • IPVS • IPVS-based Kube-proxy • Implement IPVS-based K8s service load balancing • Conclusion 40
  32. Copyright 2015 ITRI 工業技術研究院 • IPVS is a L4 load

    balancer in LVS • IPVS provides ◦ better scalability & performance for large clusters ◦ more load balancing algorithms than iptables ◦ server health checking and connection retries, etc • We can use IPVS mode of kube-proxy • Know how IPVS-based Kube-proxy work Conclusion 41