Speaker : Yutaro Hayakawa > Joined to the LINE Network Development Team as a new graduate in this year > Working for Development and Operation for Load Balancers
A Private Cloud Service for LINE Developers Verda Compute Networking Storage K8S Kafka Redis MySQL IaaS Managed Services Function Platform … … PaaS Load Balancer
Scales the LINE Applications Verda Load Balancer As A Service (LBaaS) VM Service A VM VM VM VM Service B VM VM VM VM Service C VM VM VM Virtually Dedicated Load Balancers Shared Load Balancer Cluster
Two Types of Load Balancing For Different Requirements Server Server Server Req Req Req Req Server Server Server Req Req Req Req Layer7 Load Balancer (L7LB) a.k.a Reverse Proxy Layer4 Load Balancer (L4LB)
> Prepare Certificates > Optimize Resource Allocation L7 Load Balancing Service L4 Load Balancing Service > Efficient and Fast Data Plane > Completely Developed From Scratch API Server and Orchestration Systems > Automation Friendly Not Just “Operating” on It We Are “Developing” an LBaaS
Service B Service A In the Past From Beginning of the LINE to 2016 Service C User Traffic L4/L7 LB HW Appliances 1 + 1 Active Standby Server Server Server Server Server Server Server Server Server Server Server Server
Painful for Both of Operators and Users Operational Cost >Takes about 1 ~ 2 days for registering the backends >Cannot meet rapidly increasing demands >CLI based manual operation
Scalability and Availability Issue Session Table Exhaustion Problem > Session Table • Remember Client ⁵ Backend Server Mappings per TCP Connection
> Doesn’t Scale With Large User Traffic
> DoS Attack Causes Big Outage • TCP SYN Flood Attack Source Destination Client 1 Server A Client 2 Server D Client 3 Server A … … Client N Server X Session Table
Ride on the Shoulders of “Tech Giants” Research and Development 2010 2016 We Were Here Google Maglev[5] Microsoft Ananta[1] Facebook Talk in SRECon[4] Cloudflare Blog Post[2] MS Research Duet[3] 2019
A New Architecture Multi Tier N + 1 Load Balancer Cluster Server Server Server Server Server Server L4LB Tier L7LB Tier Router Tier N + 1 Active Active
A New Architecture Multi Tier N + 1 Load Balancer Cluster Server Server Server Server Server Server L4LB Tier L7LB Tier Router Tier Stateless No Session Table
Controller Design > Ordinary Python Web Application • Provides API To Interact With Load Balancer Clusters • Fully Automated > User Interface • GUI, CLI, or Use API Directly > Common Authentication With OpenStack > Revision Management $ openstack verda lb create … $ openstack verda lb list
Why Do We Need To Scratch the L4 Load Balancer? > The Common Problem of Software Based Load Balancer Is Performance > Performance Objective for Single L4LB Instance Was 7Mpps > Difficult To Achieve by Existing Load Balancer Software
500 LoC Fast L4 Load Balancer With XDP NIC Driver Protocol Stack XDP User App User Kernel Physical NIC XDP (eXpress Data Path) > “Fast Path” of Linux Network Stack • Hook Packets in NIC Driver > Able To Write the Packet Processing in Very Simple C Code > Statically Verifies the “Safety” of the Code C Code Packet Compile& Attach
How to “Keep” the Performance? Continuous Performance Test >Performance Is a “Value” of the Service >We Need To Continuously Make Sure We Could Keep the Performance >Like CI/CD
How To Do the Reproducible Performance Test? Fully Automated Performance Tests Traffic Generator Run Load Test PR Trigger Report Result GitHub Drone (CI/CD) Unified Test Environment Developer
How To Do the Reproducible Performance Test? Fully Automated Performance Tests Traffic Generator Run Load Test PR Trigger Report Result GitHub Drone (CI/CD) Unified Test Environment Developer
Problem of the Stateless L4LB Drawback of the Stateless Approach Packet Hash( 5tuple ) Hash Value Destination 0 Backend A 1 Backend C 2 Backend B 3 Backend D … … ServerD ServerC ServerB ServerA 1. Source IP 2. Destination IP 3. Source Port 4. Destination Port 5. Protocol Number
Problem of the Stateless L4LB Drawback of the Stateless Approach Hash Value Destination 0 Server A 1 Server C 2 Server B 3 Server D … … ServerD ServerC ServerB ServerA Hash Value Destination 0 Server D 1 Server C 2 Server B 3 Server B … … Cannot Do “Graceful” Shutdown & Difficult To Failover
Consistent Hashing > Special Type of the Hash Function Which Can Reduce the Possibility of Connection Disruption on Hash Table Update > We Use Maglev Hashing [5]
Use Cases That Consistent Hashing Is Not Enough > Media Platform • Connection Suddenly Disrupted During File Upload, Playing Video … > Ads Platform • Miss the Ads Impression due to the Connection Reset … > The Problem Was a Blocker When Media Platform Migrated to Verda
Session Caching Lookup Session cache Lookup Hash Table Update Session Cache Hash Value Destination 0 Server A 1 Server C 2 Server B 3 Server D … … Source Destination Client1 Server A Client2 Server C Client3 Server B Client4 Server D … … Miss Hit Client → Server Mapping Hash → Server Mapping Session Cache Hash Table
Solution Hybrid Approach > Detect the SYN Flood on the L4LB > Fallback to the Stateless Mode for a While When It Detects the SYN Flood Time SYN/s Threshold SYN Flood!!!
(Again) Ride on the Shoulders of “Tech Giants” Research and Development 2012 2019 We Are Here Google Magrev[5] Facebook Katran[7] Fastly Faild[6] Facebook Talk in SRECon[4] MS Research Duet[3] GitHub GLB[8] NEW! 2018
[5] Eisenbud, Daniel E., et al. "Maglev: A fast and reliable software network load balancer." 13th {USENIX} Symposium on Networked Systems Design and Implementation ({NSDI} 16). 2016.
[6] Araújo, João Taveira, et al. "Balancing on the edge: Transport affinity without network state." 15th {USENIX} Symposium on Networked Systems Design and Implementation ({NSDI} 18). 2018.
line_devday2019
minorun365
rilmayer
lemiorhan
sinsoku
yokawasa
yoshiitaka
s2terminal
akkie76
sat
masasuzu
nagix
masa5555
62gerente
jcasabona
sachag
thekraken
kevinliebholz
dotmariusz
deanohume
frogandcode
aki_iinuma
cassininazir
morganepeng
kneath