“Do you have a virtual router?” Discuss how to use virtual routers LINE Corporation Hiroki Shirokura https://www.janog.gr.jp/meeting/janog51/VR-en/ 1

• Senior Software Engineer @ Private Cloud • Responsibility: SDN, Cloud Networking • Design / Implementation / Reliability • SRv6, BGP OSS Upstream Developer • FRRouting, ExaBGP, etc.. • https://github.com/slankdev/ • HN: slankdev I’m Hiroki Shirokura from LINE 2 I both Control-plane, Data-plane

LINE’s Growth Strategy :: Smart Portal == Realising Super App Our goal is to create a "Smart Portal" that seamlessly connects people, information, services, companies, and brands to whatever they need, both online and offline, using "LINE" as a gateway. reference to our factbook 3

3 SWEs for stable-services ● system operator ○ customer support ○ maintenance ● software developer ● project manager 1 SWEs for newly-provided-services ● system architect ● software architect/developer ● project manager

(3, a) VPC Networking What is constructing Cloud Networking @ LINE 5 VM ASBR ASBR DCI-BB DCI-BB internet Other Region VPC Gateway (3,b) NAT(3,a) L4LB(2,a) L7LB(3,a) DNS(3,a) ● Platform ○ (1) Commercial Network Box ■ Routing Platform: Clos-Fabric, External Networking ■ Security Platform: Firewall, IDS, etc.. for Fintech ○ (2) Baremetal x86 Server (XDP, TC-BPF, DPDK, Linux-kernel) ○ (3) Virtual x86 Server (XDP, TC-BPF, DPDK, Linux-kernel) ● How we use it ○ (a) Shared Dplane: many tenant use the same cluster at the same time ○ (b) Dedicated Dplane: only one tenant use their cluster(s) ● Recent our Interests Landscape in Operation ○ Issue: Noisy neighbor in (3,a) ○ Challenge: Development Scalability ○ Out-of-Focus (currently): ■ Smart-NIC, Programmable Dataplane ■ Hypervisor Networking big change (1,a)

Many Network/Software Challenges are already Publicated 6 linedevday/2020/sessions/2076 linedevday/2019/sessions/F1-7 linedevday/2019/sessions/E1-2 janog48/linenfv janog48/linedns janog45/srv6xdp line.connpass/184927 line.connpass/184927 nvidia/gtc janog43/line wide meeting 2019 janog50/multiaz janog49/dcnsrv6

Region-B Region-C Region-A Internet CLOS Aggregate NW Verda Verda Verda Dedicated Infra Dedicated Infra Dedicated Infra verda is our private cloud Verda VMs 100,000+ Verda HVs 10,000+ Dedicated HVs ~300 (2023.01.01)

8 When we met “new year greeting rush” by you guys

9 0.1-% Application consumes 50+% Bandwidth 70 Gbps Amazingly, 50+% of bandwidth is for single Platform. 0.1-% consumes 50+% bandwidth @LINE’s DCN #VIP = 6000+ #VIP for single platform = 4

Shared Dataplane’s Noisy Neighbor Basics • When we operate L4LB (Baremetal, Shared), we usually don’t face noisy-neighbor affects • When we operate L7LB, NAT (Virtual, Shared), we usually face noisy-neighbor affects • It is difficult to judge, which is the reason, “VM Network Interface”, “Network Function Kind” or “Both”. WE WANT TO PROTECT SOME TENANT’s AVAILABILITY FROM SOME ANOTHER TENANT’s RUSH 10 2 Big Topics • Performance of NF’s Implementation • L4LB -> XDP • L7LB -> HAProxy • NAT -> Netfilter + ip-rule + lwt-bpf • Performance of VM networking • We don’t start to use multi-queue virtio-net in hypervisor. • We faced kernel panic when RPS is enabled with LWT-bpf

Preparing Software Based Network Function • Approaches: • Approach-1: Dplane development using eBPF, DPDK, etc… • Approach-2: Use existing software like linux-kernel, ovs, etc… • In both case, scale out is necessary while operation • It is really difficult to achieve high performance for some cases (nat, tls-termination, etc…) • But we just want to scale-out the networking, instead of scale-up • But we always feel that single box performance is important for noisy neighbor • eBPF v.s. DPDK • XDP_PASS is amazing to focus for only FAST PATH • Our priority: development cost is higher than fine tuning 11

SDN System Architecture Design Knowledge(2) NAT dplane performance issue and its kernel panic ● About Distributed NAT routing architecture: linedevday/2020/2076 , gihyo/line2021/0002 ● Background ○ Increasing users after 1st release ○ There were 6 Linux servers as NAT dplane ■ They are working as act/act, No session state sync ■ 8vCPU/8GB-RAM x6 = 48vCPU ■ RPS/RSS are disabled → Only 6vCPU are working 12 Internnet Internnet Immediately after release Increased users NAT Dplane Client core Internnet

SDN System Architecture Design Knowledge(2) NAT dplane performance issue and its kernel panic ● We enable RPS to use all cores ● Few days later… weird kernel panics are occured in some servers ● Few weeks later… All dplane servers are downed one by one, due to the same issue… ○ There are some 秘孔 to make the server downed... 13 Internnet RPS enabled core Internnet Internnet Increased users

SDN System Architecture Design Knowledge(2) NAT dplane performance issue and its kernel panic ● We enable RPS to use all cores ● Few days later… weird kernel panics are occured in some servers ● Few weeks later… All dplane servers are downed one by one, due to the same issue… ○ There are some 秘孔 to make the server downed... 14 Internnet RPS enabled core Internnet Internnet Increased users Kernel Panic! It was HELL...

SDN System Architecture Design Knowledge(2) NAT dplane performance issue and its kernel panic ● Then, we disabled RPS again ● And we scaled out dplane nodes x3 (6 servers → 18 servers) now it’s 68 servers ● Lesson learned ○ (1) If your environment isn’t Majority case, be careful for tuning (LWT-BPF, etc..) ○ (2) Scale out is right ○ (3) Almost user work-loads were HTTPs/HTTP, It was easy to maintain ○ (4) Operation Rehearsal ○ (5) Performance lab 15 Scaled out Internnet Increased users Internnet core Internnet When it’s Baremetal, it takes time, but it’s Virtual, we achieved scaling operation in 3 days

RollingUpdate ● RollingUpdate Controller Capability ○ policy: what kind of action for each “Endpoint” ● Endpoint Capability ○ maintenance mode: routing advertisement stop, etc.. 16

RollingUpdate 17 kind: RollingUpdate metadata: name: update-20221015 spec: policy: machine-recreate preAction: maint-mode target: kind: RoutingGateway name: Gateway params: {...} (1) create (2) watch

RollingUpdate 18 kind: RollingUpdate metadata: name: update-20221015 spec: policy: machine-recreate preAction: maint-mode target: kind: RoutingGateway name: Gateway params: {...} status: childTargets: - kind: RoutingEndpoint name: Endpoint1 state: NOT_STARTED - kind: RoutingEndpoint name: Endpoint2 state: NOT_STARTED - kind: RoutingEndpoint name: Endpoint3 state: NOT_STARTED (3) status set

RollingUpdate 19 kind: RollingUpdate metadata: name: update-20221015 spec: policy: machine-recreate preAction: maint-mode target: kind: RoutingGateway name: Gateway params: {...} status: childTargets: - kind: RoutingEndpoint name: Endpoint1 state: NOT_STARTED - kind: RoutingEndpoint name: Endpoint2 state: NOT_STARTED - kind: RoutingEndpoint name: Endpoint3 state: NOT_STARTED (4) Update Endpoint1 as maint-mode Maintenance Mode

RollingUpdate 20 kind: RollingUpdate metadata: name: update-20221015 spec: policy: machine-recreate preAction: maint-mode target: kind: RoutingGateway name: Gateway params: {...} status: childTargets: - kind: RoutingEndpoint name: Endpoint1 state: NOT_STARTED - kind: RoutingEndpoint name: Endpoint2 state: NOT_STARTED - kind: RoutingEndpoint name: Endpoint3 state: NOT_STARTED (5) Delete child NfvMachine Maintenance Mode Delete

RollingUpdate 21 kind: RollingUpdate metadata: name: update-20221015 spec: policy: machine-recreate preAction: maint-mode target: kind: RoutingGateway name: Gateway params: {...} status: childTargets: - kind: RoutingEndpoint name: Endpoint1 state: WAIT - kind: RoutingEndpoint name: Endpoint2 state: NOT_STARTED - kind: RoutingEndpoint name: Endpoint3 state: NOT_STARTED (6) wait boot-up (7) Reconcile to create NfvMachine for Endpoint RE Ctrlr Maintenance Mode Creating

RollingUpdate 22 kind: RollingUpdate metadata: name: update-20221015 spec: policy: machine-recreate preAction: maint-mode target: kind: RoutingGateway name: Gateway params: {...} status: childTargets: - kind: RoutingEndpoint name: Endpoint1 state: WAIT - kind: RoutingEndpoint name: Endpoint2 state: NOT_STARTED - kind: RoutingEndpoint name: Endpoint3 state: NOT_STARTED RE Ctrlr Maintenance Mode Created

RollingUpdate 23 kind: RollingUpdate metadata: name: update-20221015 spec: policy: machine-recreate preAction: maint-mode target: kind: RoutingGateway name: Gateway params: {...} status: childTargets: - kind: RoutingEndpoint name: Endpoint1 state: FINISHED - kind: RoutingEndpoint name: Endpoint2 state: NOT_STARTED - kind: RoutingEndpoint name: Endpoint3 state: NOT_STARTED Maintenance Mode (8) disable maintenance mode

RollingUpdate 24 kind: RollingUpdate metadata: name: update-20221015 spec: policy: machine-recreate preAction: maint-mode target: kind: RoutingGateway name: Gateway params: {...} status: childTargets: - kind: RoutingEndpoint name: Endpoint1 state: WAIT - kind: RoutingEndpoint name: Endpoint2 state: NOT_STARTED - kind: RoutingEndpoint name: Endpoint3 state: NOT_STARTED same procedures Done

Growing Journey of RollingUpdate ● 2021.09 initial release ○ machine-recreate ● 2022.05 policy abstraction ○ container-refresh ○ loader-container-refresh ● 2022.08 no-maint-mode 25

Conclusion ● In house software, eBPF is just a approach for us ○ System architecture which allows us to scale the whole performance is necessary ● ここで話しにくい内容 ぜひ懇親会やロビーでお話しましょう! ○ LINE ブースにお越しください! 26