Our goal is to create a "Smart Portal" that seamlessly connects people, information, services, companies, and brands to whatever they need, both online and offline, using "LINE" as a gateway. reference to our factbook 3
LINE 5 VM ASBR ASBR DCI-BB DCI-BB internet Other Region VPC Gateway (3,b) NAT(3,a) L4LB(2,a) L7LB(3,a) DNS(3,a) • Platform ◦ (1) Commercial Network Box ▪ Routing Platform: Clos-Fabric, External Networking ▪ Security Platform: Firewall, IDS, etc.. for Fintech ◦ (2) Baremetal x86 Server (XDP, TC-BPF, DPDK, Linux-kernel) ◦ (3) Virtual x86 Server (XDP, TC-BPF, DPDK, Linux-kernel) • How we use it ◦ (a) Shared Dplane: many tenant use the same cluster at the same time ◦ (b) Dedicated Dplane: only one tenant use their cluster(s) • Recent our Interests Landscape in Operation ◦ Issue: Noisy neighbor in (3,a) ◦ Challenge: Development Scalability ◦ Out-of-Focus (currently): ▪ Smart-NIC, Programmable Dataplane ▪ Hypervisor Networking big change (1,a)
(Baremetal, Shared), we usually don’t face noisy-neighbor affects • When we operate L7LB, NAT (Virtual, Shared), we usually face noisy-neighbor affects • It is difficult to judge, which is the reason, “VM Network Interface”, “Network Function Kind” or “Both”. WE WANT TO PROTECT SOME TENANT’s AVAILABILITY FROM SOME ANOTHER TENANT’s RUSH 10 2 Big Topics • Performance of NF’s Implementation • L4LB -> XDP • L7LB -> HAProxy • NAT -> Netfilter + ip-rule + lwt-bpf • Performance of VM networking • We don’t start to use multi-queue virtio-net in hypervisor. • We faced kernel panic when RPS is enabled with LWT-bpf
development using eBPF, DPDK, etc… • Approach-2: Use existing software like linux-kernel, ovs, etc… • In both case, scale out is necessary while operation • It is really difficult to achieve high performance for some cases (nat, tls-termination, etc…) • But we just want to scale-out the networking, instead of scale-up • But we always feel that single box performance is important for noisy neighbor • eBPF v.s. DPDK • XDP_PASS is amazing to focus for only FAST PATH • Our priority: development cost is higher than fine tuning 11
its kernel panic • About Distributed NAT routing architecture: linedevday/2020/2076 , gihyo/line2021/0002 • Background ◦ Increasing users after 1st release ◦ There were 6 Linux servers as NAT dplane ▪ They are working as act/act, No session state sync ▪ 8vCPU/8GB-RAM x6 = 48vCPU ▪ RPS/RSS are disabled → Only 6vCPU are working 12 Internnet Internnet Immediately after release Increased users NAT Dplane Client core Internnet
its kernel panic • We enable RPS to use all cores • Few days later… weird kernel panics are occured in some servers • Few weeks later… All dplane servers are downed one by one, due to the same issue… ◦ There are some 秘孔 to make the server downed... 13 Internnet RPS enabled core Internnet Internnet Increased users
its kernel panic • We enable RPS to use all cores • Few days later… weird kernel panics are occured in some servers • Few weeks later… All dplane servers are downed one by one, due to the same issue… ◦ There are some 秘孔 to make the server downed... 14 Internnet RPS enabled core Internnet Internnet Increased users Kernel Panic! It was HELL...
its kernel panic • Then, we disabled RPS again • And we scaled out dplane nodes x3 (6 servers → 18 servers) now it’s 68 servers • Lesson learned ◦ (1) If your environment isn’t Majority case, be careful for tuning (LWT-BPF, etc..) ◦ (2) Scale out is right ◦ (3) Almost user work-loads were HTTPs/HTTP, It was easy to maintain ◦ (4) Operation Rehearsal ◦ (5) Performance lab 15 Scaled out Internnet Increased users Internnet core Internnet When it’s Baremetal, it takes time, but it’s Virtual, we achieved scaling operation in 3 days
line_developers
kazeburo
yuj1osm
taishin
opelab
jousysmiler
kouzoukaikaku
shonansurvivors
jpishikawa
gakumura
epicsdao
ma2shita
loftkun
gr2m
paulrobertlloyd
shpigford
sachag
bkeepers
bryan
mongodb
garrettdimon
akmur
swwweet
aarron
rasmusluckow
