Share
Embed
Hyperscale distributed NAT system and software engineering
Hiroki Shirokura
LINE ITSC Verda Network Development Team Senior Software Engineer
https://linedevday.linecorp.com/2020/ja/sessions/2076
https://linedevday.linecorp.com/2020/en/sessions/2076
LINE DevDay 2020
November 25, 2020
Transcript
80 slides
-
Speaker : Hiroki Shirokura Senior Software Engineer at Verda In
charge of Design/Develop-ing Network Service on Private Cloud Love SDN, Routing, Distributed System OSS contributor around Networking -
Verda is a Private Cloud for LINE is based on
OpenStack. since 2016~ FaaS PaaS IaaS NAT LB Bare metal Hypervisor 2,000+ Virtual Machine 55,000+ Physical Machine 20,000+ Verda Only +10,000/Half -
Simple Networking @ Verda eBGP eBGP APPROACH Full L3 (No-L2,
BGP Fabric) Protocol Reduction (No-Overlay) No Overlay Network Service We Get Simple & High Capacity -
Simple Networking @ Verda eBGP eBGP APPROACH Full L3 (No-L2,
BGP Fabric) Protocol Reduction (No-Overlay) No Overlay Network Service We Get Simple & High Capacity Issues No Managed Network policy Public-IP High Consumption We need gateway solution ! -
Let’s Provide Internet Gateway Service Internet Gateway Service NAT 142.8.0.1
NAT 142.8.0.2 Fair Balanced Flexible Config Extensibility Hyperscale 4 Requirements Service Project A Uses 142.8.0.1 Service Project B Uses 142.8.0.2 %JCNNGPIG Allow-list ACL Allow-list ACL Allow-list Dest Proto api.external.linecorp.com https, http ftp.jaist.ac.jp ftp, https github.com ssh, https -
Hyperscale Challenge…? Back to the Traditional Way Hardware Appliance Standby
Hardware Appliance ! Active -
Hyperscale Challenge…? Back to the Traditional Way Hardware Appliance Hardware
Appliance Dead New Active Graceful -
Hyperscale Challenge…? Back to the Traditional Way Hardware Appliance Hardware
Appliance Dead New Active Big Blast Radius… NG#1 100% Blast Radius -
Hyperscale Challenge…? Back to the Traditional Way Standby Active 50%
Blast Radius 50% Blast Radius Standby Active 2NxM Cluster Operation Cost NG#2 -
Hyperscale Challenge…? Back to the Traditional Way Standby Active Standby
Active L2 Network L2 Network Miss-match with Full L3 Network NG#3 -
Hyperscale Challenge…? Back to the Traditional Way Standby Active Standby
Active L2 Network L2 Network Miss-match with Full L3 Network NG#3 Mechanism needs (1) N+1 Active/Active Single Cluster (2) Full L3 Simple Network Aware Let’s Construct the New One -
FabricNAT: Hyperscale Distributed NAT System Match Src Action 10.0.0.1 NAT
142.8.0.1:1000-1031 10.0.0.2 Redirect dplane002 10.0.0.3 Redirect dplane003 Match Src Action 10.0.0.1 Redirect dplane001 10.0.0.2 NAT 142.8.0.1:1032:1063 10.0.0.3 Redirect dplane003 Match Src Action 10.0.0.1 Redirect dplane001 10.0.0.2 Redirect dplane002 10.0.0.3 NAT 142.8.0.1:1064-1095 Match Src Action 10.0.0.1 Redirect dplane001 10.0.0.2 Redirect dplane002 10.0.0.3 Redirect dplane003 NAT 142.8.0.1 dplane001 dplane002 dplane003 dplane004 -
FabricNAT: Hyperscale Distributed NAT System NAT 142.8.0.1 FabricNAT feature is
N+1 Active/Active in Single Cluster Full L3 Network awareness Match Src Action 10.0.0.1 NAT 142.8.0.1:1000-1031 10.0.0.2 Redirect dplane002 10.0.0.3 Redirect dplane003 Match Src Action 10.0.0.1 Redirect dplane001 10.0.0.2 NAT 142.8.0.1:1032:1063 10.0.0.3 Redirect dplane003 Match Src Action 10.0.0.1 Redirect dplane001 10.0.0.2 Redirect dplane002 10.0.0.3 NAT 142.8.0.1:1064-1095 Match Src Action 10.0.0.1 Redirect dplane001 10.0.0.2 Redirect dplane002 10.0.0.3 Redirect dplane003 dplane001 dplane002 dplane003 dplane004 FabricNAT is structured with… BGP Anycast Policy Based Routing IPIP tunneling -
FabricNAT: Hyperscale Distributed NAT System Match Src Action 10.0.0.1 NAT
142.8.0.1:1000-1031 10.0.0.2 Redirect dplane002 10.0.0.3 Redirect dplane003 Match Src Action 10.0.0.1 Redirect dplane001 10.0.0.2 NAT 142.8.0.1:1032:1063 10.0.0.3 Redirect dplane003 Match Src Action 10.0.0.1 Redirect dplane001 10.0.0.2 Redirect dplane002 10.0.0.3 NAT 142.8.0.1:1064-1095 Match Src Action 10.0.0.1 Redirect dplane001 10.0.0.2 Redirect dplane002 10.0.0.3 Redirect dplane003 dplane001 dplane002 dplane003 dplane004 eth0: 10.0.0.1 External service 119.0.0.1:80 -
FabricNAT: Hyperscale Distributed NAT System eth0: 10.0.0.1 Match Src Action
10.0.0.1 NAT 142.8.0.1:1000-1031 10.0.0.2 Redirect dplane002 10.0.0.3 Redirect dplane003 Match Src Action 10.0.0.1 Redirect dplane001 10.0.0.2 NAT 142.8.0.1:1032:1063 10.0.0.3 Redirect dplane003 Match Src Action 10.0.0.1 Redirect dplane001 10.0.0.2 Redirect dplane002 10.0.0.3 NAT 142.8.0.1:1064-1095 Match Src Action 10.0.0.1 Redirect dplane001 10.0.0.2 Redirect dplane002 10.0.0.3 Redirect dplane003 dplane001 dplane002 dplane003 dplane004 eth0: 10.0.0.1 FabricNAT Global NAT Configuration Table ID Client Address NAT Public-IP:Port-Range Worker 1 10.0.0.1 142.8.0.1:1000-1031 dplane001 2 10.0.0.2 142.8.0.1:1032-1063 dplane002 3 10.0.0.3 142.8.0.1:1064-1095 dplane003 External service 119.0.0.1:80 dplane003 dplane004 dplane001 dplane002 Match Src Action 10.0.0.1 Redirect dplane001 10.0.0.2 Redirect dplane002 10.0.0.3 NAT 142.8.0.1:1064-1095 Match Src Action 10.0.0.1 Redirect dplane001 10.0.0.2 Redirect dplane002 10.0.0.3 Redirect dplane003 Match Src Action 10.0.0.1 NAT 142.8.0.1:1000-1031 10.0.0.2 Redirect dplane002 10.0.0.3 Redirect dplane003 Match Src Action 10.0.0.1 Redirect dplane001 10.0.0.2 NAT 142.8.0.1:1032:1063 10.0.0.3 Redirect dplane003 -
FabricNAT: Hyperscale Distributed NAT System eth0: 10.0.0.1 Match Src Action
10.0.0.1 NAT 142.8.0.1:1000-1031 10.0.0.2 Redirect dplane002 10.0.0.3 Redirect dplane003 Match Src Action 10.0.0.1 Redirect dplane001 10.0.0.2 NAT 142.8.0.1:1032:1063 10.0.0.3 Redirect dplane003 Match Src Action 10.0.0.1 Redirect dplane001 10.0.0.2 Redirect dplane002 10.0.0.3 NAT 142.8.0.1:1064-1095 Match Src Action 10.0.0.1 Redirect dplane001 10.0.0.2 Redirect dplane002 10.0.0.3 Redirect dplane003 dplane001 dplane002 dplane003 dplane004 Match Src Action 10.0.0.1 NAT 142.8.0.1:1000-1031 10.0.0.2 Redirect dplane002 10.0.0.3 Redirect dplane003 Match Src Action 10.0.0.1 Redirect dplane001 10.0.0.2 NAT 142.8.0.1:1032:1063 10.0.0.3 Redirect dplane003 Match Src Action 10.0.0.1 Redirect dplane001 10.0.0.2 Redirect dplane002 10.0.0.3 NAT 142.8.0.1:1064-1095 Match Src Action 10.0.0.1 Redirect dplane001 10.0.0.2 Redirect dplane002 10.0.0.3 Redirect dplane003 dplane001 lo: 10.255.0.1 tun: 10.99.0.1 dplane002 lo: 10.255.0.2 tun: 10.99.0.1 dplane003 lo: 10.255.0.3 tun: 10.99.0.1 dplane004 lo: 10.255.0.4 tun: 10.99.0.1 Match Src Action 10.0.0.1 Redirect dplane001 10.0.0.2 Redirect dplane002 10.0.0.3 NAT 142.8.0.1:1064-1095 Match Src Action 10.0.0.1 Redirect dplane001 10.0.0.2 Redirect dplane002 10.0.0.3 Redirect dplane003 Match Src Action 10.0.0.1 NAT 142.8.0.1:1000-1031 10.0.0.2 Redirect dplane002 10.0.0.3 Redirect dplane003 Match Src Action 10.0.0.1 Redirect dplane001 10.0.0.2 NAT 142.8.0.1:1032:1063 10.0.0.3 Redirect dplane003 dplane003 dplane004 dplane001 dplane002 eth0: 10.0.0.1 FabricNAT Global NAT Configuration Table ID Client Address NAT Public-IP:Port-Range Worker 1 10.0.0.1 142.8.0.1:1000-1031 dplane001 2 10.0.0.2 142.8.0.1:1032-1063 dplane002 3 10.0.0.3 142.8.0.1:1064-1095 dplane003 External service 119.0.0.1:80 -
FabricNAT: Hyperscale Distributed NAT System eth0: 10.0.0.1 Match Src Action
10.0.0.1 NAT 142.8.0.1:1000-1031 10.0.0.2 Redirect dplane002 10.0.0.3 Redirect dplane003 Match Src Action 10.0.0.1 Redirect dplane001 10.0.0.2 NAT 142.8.0.1:1032:1063 10.0.0.3 Redirect dplane003 Match Src Action 10.0.0.1 Redirect dplane001 10.0.0.2 Redirect dplane002 10.0.0.3 NAT 142.8.0.1:1064-1095 Match Src Action 10.0.0.1 Redirect dplane001 10.0.0.2 Redirect dplane002 10.0.0.3 Redirect dplane003 dplane001 dplane002 dplane003 dplane004 eth0: 10.0.0.1 FabricNAT Global NAT Configuration Table ID Client Address NAT Public-IP:Port-Range Worker 1 10.0.0.1 142.8.0.1:1000-1031 dplane001 2 10.0.0.2 142.8.0.1:1032-1063 dplane002 3 10.0.0.3 142.8.0.1:1064-1095 dplane003 External service 119.0.0.1:80 -
FabricNAT: Hyperscale Distributed NAT System eth0: 10.0.0.1 External service 119.0.0.1:80
FabricNAT Global NAT Configuration Table ID Client Address NAT Public-IP:Port-Range Worker 1 10.0.0.1 142.8.0.1:1000-1031 dplane001 2 10.0.0.2 142.8.0.1:1032-1063 dplane002 3 10.0.0.3 142.8.0.1:1064-1095 dplane003 src: 10.0.0.1:2859 dst: 119.0.0.1:80 proto: tcp Match Src Action 10.0.0.1 NAT 142.8.0.1:1000-1031 10.0.0.2 Redirect dplane002 10.0.0.3 Redirect dplane003 Match Src Action 10.0.0.1 Redirect dplane001 10.0.0.2 NAT 142.8.0.1:1032:1063 10.0.0.3 Redirect dplane003 Match Src Action 10.0.0.1 Redirect dplane001 10.0.0.2 Redirect dplane002 10.0.0.3 NAT 142.8.0.1:1064-1095 Match Src Action 10.0.0.1 Redirect dplane001 10.0.0.2 Redirect dplane002 10.0.0.3 Redirect dplane003 dplane001 lo: 10.255.0.1 tun: 10.99.0.1 dplane002 lo: 10.255.0.2 tun: 10.99.0.1 dplane003 lo: 10.255.0.3 tun: 10.99.0.1 dplane004 lo: 10.255.0.4 tun: 10.99.0.1 dplane003 dplane004 dplane001 dplane002 -
FabricNAT: Hyperscale Distributed NAT System eth0: 10.0.0.1 External service 119.0.0.1:80
FabricNAT Global NAT Configuration Table ID Client Address NAT Public-IP:Port-Range Worker 1 10.0.0.1 142.8.0.1:1000-1031 dplane001 2 10.0.0.2 142.8.0.1:1032-1063 dplane002 3 10.0.0.3 142.8.0.1:1064-1095 dplane003 dst: 10.99.0.1 proto: ipip src: 10.0.0.1:2859 dst: 119.0.0.1:80 proto: tcp Match Src Action 10.0.0.1 NAT 142.8.0.1:1000-1031 10.0.0.2 Redirect dplane002 10.0.0.3 Redirect dplane003 Match Src Action 10.0.0.1 Redirect dplane001 10.0.0.2 NAT 142.8.0.1:1032:1063 10.0.0.3 Redirect dplane003 Match Src Action 10.0.0.1 Redirect dplane001 10.0.0.2 Redirect dplane002 10.0.0.3 NAT 142.8.0.1:1064-1095 Match Src Action 10.0.0.1 Redirect dplane001 10.0.0.2 Redirect dplane002 10.0.0.3 Redirect dplane003 dplane001 lo: 10.255.0.1 tun: 10.99.0.1 dplane002 lo: 10.255.0.2 tun: 10.99.0.1 dplane003 lo: 10.255.0.3 tun: 10.99.0.1 dplane004 lo: 10.255.0.4 tun: 10.99.0.1 dplane003 dplane004 dplane001 dplane002 -
Match Src Action 10.0.0.1 Redirect dplane001 10.0.0.2 Redirect dplane002 10.0.0.3
NAT 142.8.0.1:1064-1095 Match Src Action 10.0.0.1 Redirect dplane001 10.0.0.2 Redirect dplane002 10.0.0.3 Redirect dplane003 dplane003 lo: 10.255.0.3 tun: 10.99.0.1 dplane004 lo: 10.255.0.4 tun: 10.99.0.1 FabricNAT: Hyperscale Distributed NAT System eth0: 10.0.0.1 External service 119.0.0.1:80 FabricNAT Global NAT Configuration Table ID Client Address NAT Public-IP:Port-Range Worker 1 10.0.0.1 142.8.0.1:1000-1031 dplane001 2 10.0.0.2 142.8.0.1:1032-1063 dplane002 3 10.0.0.3 142.8.0.1:1064-1095 dplane003 BGP Advertisement 10.99.0.1/32 BGP Advertisement 10.99.0.1/32 BGP Advertisement 10.99.0.1/32 BGP Advertisement 10.99.0.1/32 Match Src Action 10.0.0.1 NAT 142.8.0.1:1000-1031 10.0.0.2 Redirect dplane002 10.0.0.3 Redirect dplane003 Match Src Action 10.0.0.1 Redirect dplane001 10.0.0.2 NAT 142.8.0.1:1032:1063 10.0.0.3 Redirect dplane003 dplane001 lo: 10.255.0.1 tun: 10.99.0.1 dplane002 lo: 10.255.0.2 tun: 10.99.0.1 dplane003 dplane004 dplane001 dplane002 dst: 10.99.0.1 proto: ipip src: 10.0.0.1:2859 dst: 119.0.0.1:80 proto: tcp -
Match Src Action 10.0.0.1 Redirect dplane001 10.0.0.2 Redirect dplane002 10.0.0.3
NAT 142.8.0.1:1064-1095 Match Src Action 10.0.0.1 Redirect dplane001 10.0.0.2 Redirect dplane002 10.0.0.3 Redirect dplane003 dplane003 lo: 10.255.0.3 tun: 10.99.0.1 dplane004 lo: 10.255.0.4 tun: 10.99.0.1 FabricNAT: Hyperscale Distributed NAT System eth0: 10.0.0.1 External service 119.0.0.1:80 FabricNAT Global NAT Configuration Table ID Client Address NAT Public-IP:Port-Range Worker 1 10.0.0.1 142.8.0.1:1000-1031 dplane001 2 10.0.0.2 142.8.0.1:1032-1063 dplane002 3 10.0.0.3 142.8.0.1:1064-1095 dplane003 Match Src Action 10.0.0.1 NAT 142.8.0.1:1000-1031 10.0.0.2 Redirect dplane002 10.0.0.3 Redirect dplane003 Match Src Action 10.0.0.1 Redirect dplane001 10.0.0.2 NAT 142.8.0.1:1032:1063 10.0.0.3 Redirect dplane003 dplane001 lo: 10.255.0.1 tun: 10.99.0.1 dplane002 lo: 10.255.0.2 tun: 10.99.0.1 dplane003 dplane004 dplane001 dplane002 dst: 10.99.0.1 proto: ipip src: 10.0.0.1:2859 dst: 119.0.0.1:80 proto: tcp -
Match Src Action 10.0.0.1 NAT 142.8.0.1:1000-1031 10.0.0.2 Redirect dplane002 10.0.0.3
Redirect dplane003 Match Src Action 10.0.0.1 Redirect dplane001 10.0.0.2 NAT 142.8.0.1:1032:1063 10.0.0.3 Redirect dplane003 Match Src Action 10.0.0.1 Redirect dplane001 10.0.0.2 Redirect dplane002 10.0.0.3 NAT 142.8.0.1:1064-1095 Match Src Action 10.0.0.1 Redirect dplane001 10.0.0.2 Redirect dplane002 10.0.0.3 Redirect dplane003 dplane001 lo: 10.255.0.1 tun: 10.99.0.1 dplane002 lo: 10.255.0.2 tun: 10.99.0.1 dplane003 lo: 10.255.0.3 tun: 10.99.0.1 dplane004 lo: 10.255.0.4 tun: 10.99.0.1 FabricNAT: Hyperscale Distributed NAT System eth0: 10.0.0.1 External service 119.0.0.1:80 FabricNAT Global NAT Configuration Table ID Client Address NAT Public-IP:Port-Range Worker 1 10.0.0.1 142.8.0.1:1000-1031 dplane001 2 10.0.0.2 142.8.0.1:1032-1063 dplane002 3 10.0.0.3 142.8.0.1:1064-1095 dplane003 dplane003 dplane004 dplane001 dplane002 src: 10.0.0.1:2859 dst: 119.0.0.1:80 proto: tcp -
Match Src Action 10.0.0.1 Redirect dplane001 10.0.0.2 Redirect dplane002 10.0.0.3
NAT 142.8.0.1:1064-1095 Match Src Action 10.0.0.1 Redirect dplane001 10.0.0.2 Redirect dplane002 10.0.0.3 Redirect dplane003 dplane003 lo: 10.255.0.3 tun: 10.99.0.1 dplane004 lo: 10.255.0.4 tun: 10.99.0.1 FabricNAT: Hyperscale Distributed NAT System eth0: 10.0.0.1 External service 119.0.0.1:80 FabricNAT Global NAT Configuration Table ID Client Address NAT Public-IP:Port-Range Worker 1 10.0.0.1 142.8.0.1:1000-1031 dplane001 2 10.0.0.2 142.8.0.1:1032-1063 dplane002 3 10.0.0.3 142.8.0.1:1064-1095 dplane003 Match Src Action 10.0.0.1 NAT 142.8.0.1:1000-1031 10.0.0.2 Redirect dplane002 10.0.0.3 Redirect dplane003 Match Src Action 10.0.0.1 Redirect dplane001 10.0.0.2 NAT 142.8.0.1:1032:1063 10.0.0.3 Redirect dplane003 dplane001 lo: 10.255.0.1 tun: 10.99.0.1 dplane002 lo: 10.255.0.2 tun: 10.99.0.1 dplane003 dplane004 dplane001 dplane002 src: 10.0.0.1:2859 dst: 119.0.0.1:80 proto: tcp -
Match Src Action 10.0.0.1 NAT 142.8.0.1:1000-1031 10.0.0.2 Redirect dplane002 10.0.0.3
Redirect dplane003 Match Src Action 10.0.0.1 Redirect dplane001 10.0.0.2 NAT 142.8.0.1:1032:1063 10.0.0.3 Redirect dplane003 dplane001 lo: 10.255.0.1 tun: 10.99.0.1 dplane002 lo: 10.255.0.2 tun: 10.99.0.1 Match Src Action 10.0.0.1 Redirect dplane001 10.0.0.2 Redirect dplane002 10.0.0.3 NAT 142.8.0.1:1064-1095 Match Src Action 10.0.0.1 Redirect dplane001 10.0.0.2 Redirect dplane002 10.0.0.3 Redirect dplane003 dplane003 lo: 10.255.0.3 tun: 10.99.0.1 dplane004 lo: 10.255.0.4 tun: 10.99.0.1 FabricNAT: Hyperscale Distributed NAT System eth0: 10.0.0.1 External service 119.0.0.1:80 FabricNAT Global NAT Configuration Table ID Client Address NAT Public-IP:Port-Range Worker 1 10.0.0.1 142.8.0.1:1000-1031 dplane001 2 10.0.0.2 142.8.0.1:1032-1063 dplane002 3 10.0.0.3 142.8.0.1:1064-1095 dplane003 dplane003 dplane004 dplane001 dplane002 dst:10.255.0.1 proto: ipip src: 10.0.0.1:2859 dst: 119.0.0.1:80 proto: tcp -
Match Src Action 10.0.0.1 NAT 142.8.0.1:1000-1031 10.0.0.2 Redirect dplane002 10.0.0.3
Redirect dplane003 Match Src Action 10.0.0.1 Redirect dplane001 10.0.0.2 NAT 142.8.0.1:1032:1063 10.0.0.3 Redirect dplane003 dplane001 lo: 10.255.0.1 tun: 10.99.0.1 dplane002 lo: 10.255.0.2 tun: 10.99.0.1 FabricNAT: Hyperscale Distributed NAT System eth0: 10.0.0.1 External service 119.0.0.1:80 FabricNAT Global NAT Configuration Table ID Client Address NAT Public-IP:Port-Range Worker 1 10.0.0.1 142.8.0.1:1000-1031 dplane001 2 10.0.0.2 142.8.0.1:1032-1063 dplane002 3 10.0.0.3 142.8.0.1:1064-1095 dplane003 dst:10.255.0.1 proto: ipip src: 10.0.0.1:2859 dst: 119.0.0.1:80 proto: tcp Match Src Action 10.0.0.1 Redirect dplane001 10.0.0.2 Redirect dplane002 10.0.0.3 NAT 142.8.0.1:1064-1095 Match Src Action 10.0.0.1 Redirect dplane001 10.0.0.2 Redirect dplane002 10.0.0.3 Redirect dplane003 dplane003 lo: 10.255.0.3 tun: 10.99.0.1 dplane004 lo: 10.255.0.4 tun: 10.99.0.1 dplane003 dplane004 dplane001 dplane002 -
FabricNAT: Hyperscale Distributed NAT System eth0: 10.0.0.1 External service 119.0.0.1:80
FabricNAT Global NAT Configuration Table ID Client Address NAT Public-IP:Port-Range Worker 1 10.0.0.1 142.8.0.1:1000-1031 dplane001 2 10.0.0.2 142.8.0.1:1032-1063 dplane002 3 10.0.0.3 142.8.0.1:1064-1095 dplane003 Match Src Action 10.0.0.1 NAT 142.8.0.1:1000-1031 10.0.0.2 Redirect dplane002 10.0.0.3 Redirect dplane003 Match Src Action 10.0.0.1 Redirect dplane001 10.0.0.2 NAT 142.8.0.1:1032:1063 10.0.0.3 Redirect dplane003 dplane001 lo: 10.255.0.1 tun: 10.99.0.1 dplane002 lo: 10.255.0.2 tun: 10.99.0.1 src: 10.0.0.1:2859 dst: 119.0.0.1:80 proto: tcp Match Src Action 10.0.0.1 Redirect dplane001 10.0.0.2 Redirect dplane002 10.0.0.3 NAT 142.8.0.1:1064-1095 Match Src Action 10.0.0.1 Redirect dplane001 10.0.0.2 Redirect dplane002 10.0.0.3 Redirect dplane003 dplane003 lo: 10.255.0.3 tun: 10.99.0.1 dplane004 lo: 10.255.0.4 tun: 10.99.0.1 dplane003 dplane004 dplane001 dplane002 -
FabricNAT: Hyperscale Distributed NAT System eth0: 10.0.0.1 External service 119.0.0.1:80
FabricNAT Global NAT Configuration Table ID Client Address NAT Public-IP:Port-Range Worker 1 10.0.0.1 142.8.0.1:1000-1031 dplane001 2 10.0.0.2 142.8.0.1:1032-1063 dplane002 3 10.0.0.3 142.8.0.1:1064-1095 dplane003 src: 142.8.0.1:1010 dst: 119.0.0.1:80 proto: tcp Match Src Action 10.0.0.1 NAT 142.8.0.1:1000-1031 10.0.0.2 Redirect dplane002 10.0.0.3 Redirect dplane003 Match Src Action 10.0.0.1 Redirect dplane001 10.0.0.2 NAT 142.8.0.1:1032:1063 10.0.0.3 Redirect dplane003 dplane001 lo: 10.255.0.1 tun: 10.99.0.1 dplane002 lo: 10.255.0.2 tun: 10.99.0.1 Match Src Action 10.0.0.1 Redirect dplane001 10.0.0.2 Redirect dplane002 10.0.0.3 NAT 142.8.0.1:1064-1095 Match Src Action 10.0.0.1 Redirect dplane001 10.0.0.2 Redirect dplane002 10.0.0.3 Redirect dplane003 dplane003 lo: 10.255.0.3 tun: 10.99.0.1 dplane004 lo: 10.255.0.4 tun: 10.99.0.1 dplane003 dplane004 dplane001 dplane002 -
FabricNAT: Hyperscale Distributed NAT System eth0: 10.0.0.1 External service 119.0.0.1:80
FabricNAT Global NAT Configuration Table ID Client Address NAT Public-IP:Port-Range Worker 1 10.0.0.1 142.8.0.1:1000-1031 dplane001 2 10.0.0.2 142.8.0.1:1032-1063 dplane002 3 10.0.0.3 142.8.0.1:1064-1095 dplane003 src: 142.8.0.1:1010 dst: 119.0.0.1:80 proto: tcp Match Src Action 10.0.0.1 NAT 142.8.0.1:1000-1031 10.0.0.2 Redirect dplane002 10.0.0.3 Redirect dplane003 Match Src Action 10.0.0.1 Redirect dplane001 10.0.0.2 NAT 142.8.0.1:1032:1063 10.0.0.3 Redirect dplane003 dplane001 lo: 10.255.0.1 tun: 10.99.0.1 dplane002 lo: 10.255.0.2 tun: 10.99.0.1 Match Src Action 10.0.0.1 Redirect dplane001 10.0.0.2 Redirect dplane002 10.0.0.3 NAT 142.8.0.1:1064-1095 Match Src Action 10.0.0.1 Redirect dplane001 10.0.0.2 Redirect dplane002 10.0.0.3 Redirect dplane003 dplane003 lo: 10.255.0.3 tun: 10.99.0.1 dplane004 lo: 10.255.0.4 tun: 10.99.0.1 dplane003 dplane004 dplane001 dplane002 -
Match Src Action 10.0.0.1 NAT 142.8.0.1:1000-1031 10.0.0.2 Redirect dplane002 10.0.0.3
Redirect dplane003 dplane001 FabricNAT: Hyperscale Distributed NAT System eth0: 10.0.0.1 External service 119.0.0.1:80 src: 142.8.0.1:1010 dst: 119.0.0.1:80 proto: tcp FabricNAT Global NAT Configuration Table ID Client Address NAT Public-IP:Port-Range Worker 1 10.0.0.1 142.8.0.1:1000-1031 dplane001 2 10.0.0.2 142.8.0.1:1032-1063 dplane002 3 10.0.0.3 142.8.0.1:1064-1095 dplane003 Match Src Action 10.0.0.1 Redirect dplane001 10.0.0.2 NAT 142.8.0.1:1032:1063 10.0.0.3 Redirect dplane003 dplane002 lo: 10.255.0.2 tun: 10.99.0.1 Match Src Action 10.0.0.1 Redirect dplane001 10.0.0.2 Redirect dplane002 10.0.0.3 NAT 142.8.0.1:1064-1095 Match Src Action 10.0.0.1 Redirect dplane001 10.0.0.2 Redirect dplane002 10.0.0.3 Redirect dplane003 dplane003 lo: 10.255.0.3 tun: 10.99.0.1 dplane004 lo: 10.255.0.4 tun: 10.99.0.1 dplane003 dplane004 dplane002 -
Match Src Action 10.0.0.1 NAT 142.8.0.1:1000-1031 10.0.0.2 Redirect dplane002 10.0.0.3
Redirect dplane003 dplane001 FabricNAT: Hyperscale Distributed NAT System eth0: 10.0.0.1 External service 119.0.0.1:80 src: 142.8.0.1:1010 dst: 119.0.0.1:80 proto: tcp FabricNAT Global NAT Configuration Table ID Client Address NAT Public-IP:Port-Range Worker 1 10.0.0.1 142.8.0.1:1000-1031 dplane001 2 10.0.0.2 142.8.0.1:1032-1063 dplane002 3 10.0.0.3 142.8.0.1:1064-1095 dplane003 Match Dst Action 142.8.0.1:1000-1031 Red dplane001 10.0.0.2 NAT 142.8.0.1:1032: 10.0.0.3 Redirect dplane003 dplane002 lo: 10.255.0.2 tun: 10.99.0.1 Match Dst Action 142.8.0.1:1000-1031 Red dplane001 10.0.0.2 Redirect dplane002 10.0.0.3 NAT 142.8.0.1:1064- Match Dst Action 142.8.0.1:1000-1031 Red dplane001 10.0.0.2 Redirect dplane002 10.0.0.3 Redirect dplane003 dplane003 lo: 10.255.0.3 tun: 10.99.0.1 dplane004 lo: 10.255.0.4 tun: 10.99.0.1 dplane003 dplane004 dplane002 -
Recap FabricNAT • Pre allocate Port Range for Session Quota
• Worker Election per each Clients and No State Sync for Scalability • 2-Step Routing to Construct Symmetric Routing -
FabricNAT: Hyperscale Distributed NAT System eth0: 10.0.0.1 External service 119.0.0.1:80
FabricNAT Global NAT Configuration Table ID Client Address NAT Public-IP:Port-Range Worker 1 10.0.0.1 142.8.0.1:1000-1031 dplane001 2 10.0.0.2 142.8.0.1:1032-1063 dplane002 3 10.0.0.3 142.8.0.1:1064-1095 dplane003 src: 142.8.0.1:1010 dst: 119.0.0.1:80 proto: tcp Match Src Action 10.0.0.1 NAT 142.8.0.1:1000-1031 10.0.0.2 Redirect dplane002 10.0.0.3 Redirect dplane003 Match Src Action 10.0.0.1 Redirect dplane001 10.0.0.2 NAT 142.8.0.1:1032:1063 10.0.0.3 Redirect dplane003 dplane001 dplane002 Match Src Action 10.0.0.1 Redirect dplane001 10.0.0.2 Redirect dplane002 10.0.0.3 NAT 142.8.0.1:1064-1095 Match Src Action 10.0.0.1 Redirect dplane001 10.0.0.2 Redirect dplane002 10.0.0.3 Redirect dplane003 dplane003 dplane004 Working as N+1 redundancy…? -
N+1 Active/Active…? 1/N Blast Radius …? eth0: 10.0.0.1 External service
119.0.0.1:80 Match Src Action 10.0.0.1 NAT 142.8.0.1:1000-1031 10.0.0.2 Redirect dplane002 10.0.0.3 Redirect dplane003 Match Src Action 10.0.0.1 Redirect dplane001 10.0.0.2 NAT 142.8.0.1:1032:1063 10.0.0.3 Redirect dplane003 dplane001 lo: 10.255.0.1 tun: 10.99.0.1 dplane002 lo: 10.255.0.2 tun: 10.99.0.1 Match Src Action 10.0.0.1 Redirect dplane001 10.0.0.2 Redirect dplane002 10.0.0.3 NAT 142.8.0.1:1064-1095 Match Src Action 10.0.0.1 Redirect dplane001 10.0.0.2 Redirect dplane002 10.0.0.3 Redirect dplane003 dplane003 lo: 10.255.0.3 tun: 10.99.0.1 dplane004 lo: 10.255.0.4 tun: 10.99.0.1 eth0: 10.0.0.2 eth0: 10.0.0.3 FabricNAT Global NAT Configuration Table ID Client Address NAT Public-IP:Port-Range Worker 1 10.0.0.1 142.8.0.1:1000-1031 dplane001 2 10.0.0.2 142.8.0.1:1032-1063 dplane002 3 10.0.0.3 142.8.0.1:1064-1095 dplane003 eth0: 10.0.0.1 eth0: 10.0.0.2 eth0: 10.0.0.3 dplane003 dplane004 dplane001 dplane002 -
N+1 Active/Active…? 1/N Blast Radius …? eth0: 10.0.0.1 External service
119.0.0.1:80 Match Src Action 10.0.0.1 NAT 142.8.0.1:1000-1031 10.0.0.2 Redirect dplane002 10.0.0.3 Redirect dplane003 Match Src Action 10.0.0.1 Redirect dplane001 10.0.0.2 NAT 142.8.0.1:1032:1063 10.0.0.3 Redirect dplane003 dplane001 dplane002 Match Src Action 10.0.0.1 Redirect dplane001 10.0.0.2 Redirect dplane002 10.0.0.3 NAT 142.8.0.1:1064-1095 Match Src Action 10.0.0.1 Redirect dplane001 10.0.0.2 Redirect dplane002 10.0.0.3 Redirect dplane003 dplane003 dplane004 eth0: 10.0.0.2 eth0: 10.0.0.3 FabricNAT Global NAT Configuration Table ID Client Address NAT Public-IP:Port-Range Worker 1 10.0.0.1 142.8.0.1:1000-1031 dplane001 2 10.0.0.2 142.8.0.1:1032-1063 dplane002 3 10.0.0.3 142.8.0.1:1064-1095 dplane003 eth0: 10.0.0.1 eth0: 10.0.0.2 eth0: 10.0.0.3 -
N+1 Active/Active…? 1/N Blast Radius …? eth0: 10.0.0.1 External service
119.0.0.1:80 Match Src Action 10.0.0.1 NAT 142.8.0.1:1000-1031 10.0.0.2 Redirect dplane002 10.0.0.3 Redirect dplane003 Match Src Action 10.0.0.1 Redirect dplane001 10.0.0.2 NAT 142.8.0.1:1032:1063 10.0.0.3 Redirect dplane003 dplane001 dplane002 Match Src Action 10.0.0.1 Redirect dplane001 10.0.0.2 Redirect dplane002 10.0.0.3 NAT 142.8.0.1:1064-1095 Match Src Action 10.0.0.1 Redirect dplane001 10.0.0.2 Redirect dplane002 10.0.0.3 Redirect dplane003 dplane003 dplane004 eth0: 10.0.0.2 eth0: 10.0.0.3 FabricNAT Global NAT Configuration Table ID Client Address NAT Public-IP:Port-Range Worker 1 10.0.0.1 142.8.0.1:1000-1031 dplane001 2 10.0.0.2 142.8.0.1:1032-1063 dplane002 3 10.0.0.3 142.8.0.1:1064-1095 dplane003 eth0: 10.0.0.1 eth0: 10.0.0.2 eth0: 10.0.0.3 External service 119.0.0.1:80 eth0: 10.0.0.1 -
N+1 Active/Active…? 1/N Blast Radius …? eth0: 10.0.0.1 External service
119.0.0.1:80 Match Src Action 10.0.0.1 NAT 142.8.0.1:1000-1031 10.0.0.2 Redirect dplane002 10.0.0.3 Redirect dplane003 Match Src Action 10.0.0.1 Redirect dplane001 10.0.0.2 NAT 142.8.0.1:1032:1063 10.0.0.3 Redirect dplane003 dplane001 lo: 10.255.0.1 tun: 10.99.0.1 dplane002 lo: 10.255.0.2 tun: 10.99.0.1 Match Src Action 10.0.0.1 Redirect dplane001 10.0.0.2 Redirect dplane002 10.0.0.3 NAT 142.8.0.1:1064-1095 Match Src Action 10.0.0.1 Redirect dplane001 10.0.0.2 Redirect dplane002 10.0.0.3 Redirect dplane003 dplane003 lo: 10.255.0.3 tun: 10.99.0.1 dplane004 lo: 10.255.0.4 tun: 10.99.0.1 eth0: 10.0.0.2 eth0: 10.0.0.3 FabricNAT Global NAT Configuration Table ID Client Address NAT Public-IP:Port-Range Micro-active 1 10.0.0.1 142.8.0.1:1000-1031 dplane001 2 10.0.0.2 142.8.0.1:1032-1063 dplane002 3 10.0.0.3 142.8.0.1:1064-1095 dplane003 eth0: 10.0.0.1 eth0: 10.0.0.2 eth0: 10.0.0.3 BGP Advertisement 10.99.0.1/32 BGP Advertisement 10.99.0.1/32 BGP Advertisement 10.99.0.1/32 BGP Advertisement 10.99.0.1/32 dplane001 lo: 10.255.0.1 tun: 10.99.0.1 dplane002 lo: 10.255.0.2 tun: 10.99.0.1 dplane003 lo: 10.255.0.3 tun: 10.99.0.1 dplane004 lo: 10.255.0.4 tun: 10.99.0.1 Match Src Action 10.0.0.1 NAT 142.8.0.1:1000-1031 10.0.0.2 Redirect dplane002 10.0.0.3 Redirect dplane003 Match Src Action 10.0.0.1 Redirect dplane001 10.0.0.2 NAT 142.8.0.1:1032:1063 10.0.0.3 Redirect dplane003 Match Src Action 10.0.0.1 Redirect dplane001 10.0.0.2 Redirect dplane002 10.0.0.3 NAT 142.8.0.1:1064-1095 Match Src Action 10.0.0.1 Redirect dplane001 10.0.0.2 Redirect dplane002 10.0.0.3 Redirect dplane003 eth0: 10.0.0.1 dplane003 dplane004 dplane001 dplane002 -
N+1 Active/Active…? 1/N Blast Radius …? eth0: 10.0.0.1 External service
119.0.0.1:80 Match Src Action 10.0.0.1 NAT 142.8.0.1:1000-1031 10.0.0.2 Redirect dplane002 10.0.0.3 Redirect dplane003 Match Src Action 10.0.0.1 Redirect dplane001 10.0.0.2 NAT 142.8.0.1:1032:1063 10.0.0.3 Redirect dplane003 dplane001 lo: 10.255.0.1 tun: 10.99.0.1 dplane002 lo: 10.255.0.2 tun: 10.99.0.1 Match Src Action 10.0.0.1 Redirect dplane001 10.0.0.2 Redirect dplane002 10.0.0.3 NAT 142.8.0.1:1064-1095 Match Src Action 10.0.0.1 Redirect dplane001 10.0.0.2 Redirect dplane002 10.0.0.3 Redirect dplane003 dplane003 lo: 10.255.0.3 tun: 10.99.0.1 dplane004 lo: 10.255.0.4 tun: 10.99.0.1 eth0: 10.0.0.2 eth0: 10.0.0.3 FabricNAT Global NAT Configuration Table ID Client Address NAT Public-IP:Port-Range Micro-active 1 10.0.0.1 142.8.0.1:1000-1031 dplane001 2 10.0.0.2 142.8.0.1:1032-1063 dplane002 3 10.0.0.3 142.8.0.1:1064-1095 dplane003 eth0: 10.0.0.1 eth0: 10.0.0.2 eth0: 10.0.0.3 dplane001 lo: 10.255.0.1 tun: 10.99.0.1 dplane002 lo: 10.255.0.2 tun: 10.99.0.1 dplane003 lo: 10.255.0.3 tun: 10.99.0.1 dplane004 lo: 10.255.0.4 tun: 10.99.0.1 External service 119.0.0.1:80 eth0: 10.0.0.1 eth0: 10.0.0.2 eth0: 10.0.0.3 dplane003 dplane004 dplane001 dplane002 -
N+1 Active/Active…? 1/N Blast Radius …? eth0: 10.0.0.1 External service
119.0.0.1:80 Match Src Action 10.0.0.1 NAT 142.8.0.1:1000-1031 10.0.0.2 Redirect dplane002 10.0.0.3 Redirect dplane003 Match Src Action 10.0.0.1 Redirect dplane001 10.0.0.2 NAT 142.8.0.1:1032:1063 10.0.0.3 Redirect dplane003 dplane001 lo: 10.255.0.1 tun: 10.99.0.1 dplane002 lo: 10.255.0.2 tun: 10.99.0.1 Match Src Action 10.0.0.1 Redirect dplane001 10.0.0.2 Redirect dplane002 10.0.0.3 NAT 142.8.0.1:1064-1095 Match Src Action 10.0.0.1 Redirect dplane001 10.0.0.2 Redirect dplane002 10.0.0.3 Redirect dplane003 dplane003 lo: 10.255.0.3 tun: 10.99.0.1 dplane004 lo: 10.255.0.4 tun: 10.99.0.1 eth0: 10.0.0.2 eth0: 10.0.0.3 FabricNAT Global NAT Configuration Table ID Client Address NAT Public-IP:Port-Range Micro-active 1 10.0.0.1 142.8.0.1:1000-1031 dplane001 2 10.0.0.2 142.8.0.1:1032-1063 dplane002 3 10.0.0.3 142.8.0.1:1064-1095 dplane003 eth0: 10.0.0.1 eth0: 10.0.0.2 eth0: 10.0.0.3 dplane001 dplane002 lo: 10.255.0.2 tun: 10.99.0.1 dplane003 lo: 10.255.0.3 tun: 10.99.0.1 dplane004 lo: 10.255.0.4 tun: 10.99.0.1 External service 119.0.0.1:80 eth0: 10.0.0.1 eth0: 10.0.0.2 eth0: 10.0.0.3 1/N Blast Radius dplane003 dplane004 dplane002 -
N+1 Active/Active…? 1/N Blast Radius …? eth0: 10.0.0.1 External service
119.0.0.1:80 eth0: 10.0.0.2 eth0: 10.0.0.3 eth0: 10.0.0.1 eth0: 10.0.0.2 eth0: 10.0.0.3 dplane001 dplane002 dplane003 dplane004 External service 119.0.0.1:80 dplane005 dplane006 dplane007 dplane008 -
N+1 Active/Active…? 1/N Blast Radius …? eth0: 10.0.0.1 External service
119.0.0.1:80 eth0: 10.0.0.2 eth0: 10.0.0.3 eth0: 10.0.0.1 eth0: 10.0.0.2 eth0: 10.0.0.3 dplane001 dplane002 dplane003 External service 119.0.0.1:80 dplane005 dplane006 dplane007 dplane008 1/8 Blast Radius == 1/N (N=8) dplane004 -
Re-Recap FabricNAT • 2 Step Routing • 1st step: Robust
with Many Stateless Active path • 2nd step: Not Robust but Blast Radius is 1/N • Gracefull v.s. Small Blast Radius • “Small Blast Radius and Hyperscale” >>> "Big Blast Radius and Graceful” • Getting Gracefully is Additional-Step -
How we configure …? eth0: 10.0.0.1 External service 119.0.0.1:80 Match
Src Action 10.0.0.1 NAT 142.8.0.1:1000-1031 10.0.0.2 Redirect dplane002 10.0.0.3 Redirect dplane003 Match Src Action 10.0.0.1 Redirect dplane001 10.0.0.2 NAT 142.8.0.1:1032:1063 10.0.0.3 Redirect dplane003 dplane001 lo: 10.255.0.1 tun: 10.99.0.1 dplane002 lo: 10.255.0.2 tun: 10.99.0.1 Match Src Action 10.0.0.1 Redirect dplane001 10.0.0.2 Redirect dplane002 10.0.0.3 NAT 142.8.0.1:1064-1095 Match Src Action 10.0.0.1 Redirect dplane001 10.0.0.2 Redirect dplane002 10.0.0.3 Redirect dplane003 dplane003 lo: 10.255.0.3 tun: 10.99.0.1 dplane004 lo: 10.255.0.4 tun: 10.99.0.1 eth0: 10.0.0.2 eth0: 10.0.0.3 FabricNAT Global NAT Configuration Table ID Client Address NAT Public-IP:Port-Range Worker 1 10.0.0.1 142.8.0.1:1000-1031 dplane001 2 10.0.0.2 142.8.0.1:1032-1063 dplane002 3 10.0.0.3 142.8.0.1:1064-1095 dplane003 eth0: 10.0.0.1 dplane003 dplane004 dplane001 dplane002 -
How we configure …? eth0: 10.0.0.1 External service 119.0.0.1:80 eth0:
10.0.0.2 eth0: 10.0.0.3 FabricNAT Global NAT Configuration Table ID Client Address NAT Public-IP:Port-Range Micro-active 1 10.0.0.1 142.8.0.1:1000-1031 dplane001 2 10.0.0.2 142.8.0.1:1032-1063 dplane002 3 10.0.0.3 142.8.0.1:1064-1095 dplane003 eth0: 10.0.0.1 Match Src Action 10.0.0.1 NAT 142.8.0.1:1000-1031 10.0.0.2 Redirect dplane002 10.0.0.3 Redirect dplane003 Match Src Action 10.0.0.1 Redirect dplane001 10.0.0.2 NAT 142.8.0.1:1032:1063 10.0.0.3 Redirect dplane003 dplane001 lo: 10.255.0.1 tun: 10.99.0.1 dplane002 lo: 10.255.0.2 tun: 10.99.0.1 Match Src Action 10.0.0.1 Redirect dplane001 10.0.0.2 Redirect dplane002 10.0.0.3 NAT 142.8.0.1:1064-1095 Match Src Action 10.0.0.1 Redirect dplane001 10.0.0.2 Redirect dplane002 10.0.0.3 Redirect dplane003 dplane003 lo: 10.255.0.3 tun: 10.99.0.1 dplane004 lo: 10.255.0.4 tun: 10.99.0.1 Dataplane Clustering with Consul dplane003 dplane004 dplane001 dplane002 -
How we configure …? eth0: 10.0.0.1 External service 119.0.0.1:80 eth0:
10.0.0.2 eth0: 10.0.0.3 Match Src Action 10.0.0.1 NAT 142.8.0.1:1000-1031 10.0.0.2 Redirect dplane002 10.0.0.3 Redirect dplane003 Match Src Action 10.0.0.1 Redirect dplane001 10.0.0.2 NAT 142.8.0.1:1032:1063 10.0.0.3 Redirect dplane003 dplane001 lo: 10.255.0.1 tun: 10.99.0.1 dplane002 lo: 10.255.0.2 tun: 10.99.0.1 Match Src Action 10.0.0.1 Redirect dplane001 10.0.0.2 Redirect dplane002 10.0.0.3 NAT 142.8.0.1:1064-1095 Match Src Action 10.0.0.1 Redirect dplane001 10.0.0.2 Redirect dplane002 10.0.0.3 Redirect dplane003 dplane003 lo: 10.255.0.3 tun: 10.99.0.1 dplane004 lo: 10.255.0.4 tun: 10.99.0.1 Distributed Routing Configuration with Etcd FabricNAT Global NAT Configuration Table ID Client Address NAT Public-IP:Port-Range Worker 1 10.0.0.1 142.8.0.1:1000-1031 dplane001 2 10.0.0.2 142.8.0.1:1032-1063 dplane002 3 10.0.0.3 142.8.0.1:1064-1095 dplane003 dplane003 dplane004 dplane001 dplane002 -
eth0: 10.0.0.1 External service 119.0.0.1:80 eth0: 10.0.0.2 eth0: 10.0.0.3 How
we configure …? dplane002 lo: 10.255.0.2 tun: 10.99.0.1 Match Src Action 10.0.0.1 Redirect dplane001 10.0.0.2 NAT 142.8.0.1:1032:1063 10.0.0.3 Redirect dplane003 Match Src Action 10.0.0.1 Redirect dplane001 10.0.0.2 Redirect dplane002 10.0.0.3 Redirect dplane003 dplane004 lo: 10.255.0.4 tun: 10.99.0.1 dplane001 lo: 10.255.0.1 tun: 10.99.0.1 dplane003 lo: 10.255.0.3 tun: 10.99.0.1 Match Src Action 10.0.0.1 NAT 142.8.0.1:1000-1031 10.0.0.2 Redirect dplane002 10.0.0.3 Redirect dplane003 Match Src Action 10.0.0.1 Redirect dplane001 10.0.0.2 Redirect dplane002 10.0.0.3 NAT 142.8.0.1:1064-1095 scheduler apiserver Natlet Natlet Natlet Natlet FabricNAT Global NAT Configuration Table ID Client Address NAT Public-IP:Port-Range Worker 1 10.0.0.1 142.8.0.1:1000-1031 dplane001 2 10.0.0.2 142.8.0.1:1032-1063 dplane002 3 10.0.0.3 142.8.0.1:1064-1095 dplane003 dplane002 dplane004 dplane001 dplane003 -
eth0: 10.0.0.1 External service 119.0.0.1:80 eth0: 10.0.0.2 eth0: 10.0.0.3 How
we configure …? dplane002 lo: 10.255.0.2 tun: 10.99.0.1 Match Src Action 10.0.0.1 Redirect dplane001 10.0.0.2 NAT 142.8.0.1:1032:1063 10.0.0.3 Redirect dplane003 Match Src Action 10.0.0.1 Redirect dplane001 10.0.0.2 Redirect dplane002 10.0.0.3 Redirect dplane003 dplane004 lo: 10.255.0.4 tun: 10.99.0.1 dplane001 lo: 10.255.0.1 tun: 10.99.0.1 dplane003 lo: 10.255.0.3 tun: 10.99.0.1 Match Src Action 10.0.0.1 NAT 142.8.0.1:1000-1031 10.0.0.2 Redirect dplane002 10.0.0.3 Redirect dplane003 Match Src Action 10.0.0.1 Redirect dplane001 10.0.0.2 Redirect dplane002 10.0.0.3 NAT 142.8.0.1:1064-1095 scheduler apiserver Natlet Natlet Natlet Natlet FabricNAT Global NAT Configuration Table ID Client Address NAT Public-IP:Port-Range Micro-active 1 10.0.0.1 142.8.0.1:1000-1031 dplane001 2 10.0.0.2 142.8.0.1:1032-1063 dplane002 3 10.0.0.3 142.8.0.1:1064-1095 dplane003 dplane002 dplane004 dplane001 dplane003 dplane002 dplane004 dplane001 dplane003 Consul Client Consul Client Consul Client Consul Client Consul Server Natlet Natlet Natlet Natlet Watch Consul Watch Consul Watch Consul Watch Consul Gossip protocol scheduler Watch Consul -
eth0: 10.0.0.1 External service 119.0.0.1:80 eth0: 10.0.0.2 eth0: 10.0.0.3 How
we configure …? dplane002 Match Src Action 10.0.0.1 Redirect dplane001 10.0.0.2 NAT 142.8.0.1:1032:1063 10.0.0.3 Redirect dplane003 Match Src Action 10.0.0.1 Redirect dplane001 10.0.0.2 Redirect dplane002 10.0.0.3 Redirect dplane003 dplane004 dplane001 dplane003 Match Src Action 10.0.0.1 NAT 142.8.0.1:1000-1031 10.0.0.2 Redirect dplane002 10.0.0.3 Redirect dplane003 Match Src Action 10.0.0.1 Redirect dplane001 10.0.0.2 Redirect dplane002 10.0.0.3 NAT 142.8.0.1:1064-1095 scheduler apiserver Natlet Natlet Natlet Natlet FabricNAT Global NAT Configuration Table ID Client Address NAT Public-IP:Port-Range Micro-active 1 10.0.0.1 142.8.0.1:1000-1031 dplane001 2 10.0.0.2 142.8.0.1:1032-1063 dplane002 3 10.0.0.3 142.8.0.1:1064-1095 dplane003 Etcd-Watch Etcd-Watch Etcd-Watch Etcd-Watch Natlet Natlet Natlet Natlet apiserver Etcd-Put scheduler Etcd-Watch -
eth0: 10.0.0.1 External service 119.0.0.1:80 eth0: 10.0.0.2 eth0: 10.0.0.3 How
we configure …? dplane002 lo: 10.255.0.2 tun: 10.99.0.1 Match Src Action 10.0.0.1 Redirect dplane001 10.0.0.2 NAT 142.8.0.1:1032:1063 10.0.0.3 Redirect dplane003 Match Src Action 10.0.0.1 Redirect dplane001 10.0.0.2 Redirect dplane002 10.0.0.3 Redirect dplane003 dplane004 lo: 10.255.0.4 tun: 10.99.0.1 dplane001 lo: 10.255.0.1 tun: 10.99.0.1 dplane003 lo: 10.255.0.3 tun: 10.99.0.1 Match Src Action 10.0.0.1 NAT 142.8.0.1:1000-1031 10.0.0.2 Redirect dplane002 10.0.0.3 Redirect dplane003 Match Src Action 10.0.0.1 Redirect dplane001 10.0.0.2 Redirect dplane002 10.0.0.3 NAT 142.8.0.1:1064-1095 scheduler apiserver Natlet eth0: 10.0.0.4 Natlet Natlet Natlet 3. Etcd DATA PUT 4. Notice New Resource 2. Port-allocation 1096-1127 FabricNAT Global NAT Configuration Table ID Client Address NAT Public-IP:Port-Range Worker 1 10.0.0.1 142.8.0.1:1000-1031 dplane001 2 10.0.0.2 142.8.0.1:1032-1063 dplane002 3 10.0.0.3 142.8.0.1:1064-1095 dplane003 4 10.0.0.4 142.8.0.1:1096-1127 NULL 1. NAT Create “client“: "10.0.04” “vip”: “142.2.0.1” dplane002 dplane004 dplane001 dplane003 NAT Configuration Client 10.0.0.4 VIP 142.8.0.1 -
eth0: 10.0.0.1 External service 119.0.0.1:80 eth0: 10.0.0.2 eth0: 10.0.0.3 How
we configure …? dplane002 Match Src Action 10.0.0.1 Redirect dplane001 10.0.0.2 NAT 142.8.0.1:1032:1063 10.0.0.3 Redirect dplane003 Match Src Action 10.0.0.1 Redirect dplane001 10.0.0.2 Redirect dplane002 10.0.0.3 Redirect dplane003 dplane004 dplane001 dplane003 Match Src Action 10.0.0.1 NAT 142.8.0.1:1000-1031 10.0.0.2 Redirect dplane002 10.0.0.3 Redirect dplane003 Match Src Action 10.0.0.1 Redirect dplane001 10.0.0.2 Redirect dplane002 10.0.0.3 NAT 142.8.0.1:1064-1095 apiserver Natlet eth0: 10.0.0.4 Natlet Natlet Natlet FabricNAT Global NAT Configuration Table ID Client Address NAT Public-IP:Port-Range Micro-active 1 10.0.0.1 142.8.0.1:1000-1031 dplane001 2 10.0.0.2 142.8.0.1:1032-1063 dplane002 3 10.0.0.3 142.8.0.1:1064-1095 dplane003 4 10.0.0.4 142.8.0.1:1096-1127 NULL Get Healthy Dplane nodes scheduler -
eth0: 10.0.0.1 External service 119.0.0.1:80 eth0: 10.0.0.2 eth0: 10.0.0.3 How
we configure …? dplane002 lo: 10.255.0.2 tun: 10.99.0.1 Match Src Action 10.0.0.1 Redirect dplane001 10.0.0.2 NAT 142.8.0.1:1032:1063 10.0.0.3 Redirect dplane003 Match Src Action 10.0.0.1 Redirect dplane001 10.0.0.2 Redirect dplane002 10.0.0.3 Redirect dplane003 dplane004 lo: 10.255.0.4 tun: 10.99.0.1 dplane001 lo: 10.255.0.1 tun: 10.99.0.1 dplane003 lo: 10.255.0.3 tun: 10.99.0.1 Match Src Action 10.0.0.1 NAT 142.8.0.1:1000-1031 10.0.0.2 Redirect dplane002 10.0.0.3 Redirect dplane003 Match Src Action 10.0.0.1 Redirect dplane001 10.0.0.2 Redirect dplane002 10.0.0.3 NAT 142.8.0.1:1064-1095 scheduler apiserver Natlet eth0: 10.0.0.4 Natlet Natlet Natlet 1. Get Healthy Dplane nodes 2. Cluster Info 4. Re-write Scheduled Resource 3. Worker election dplane004 dplane002 dplane004 dplane001 dplane003 FabricNAT Global NAT Configuration Table ID Client Address NAT Public-IP:Port-Range Worker 1 10.0.0.1 142.8.0.1:1000-1031 dplane001 2 10.0.0.2 142.8.0.1:1032-1063 dplane002 3 10.0.0.3 142.8.0.1:1064-1095 dplane003 4 10.0.0.4 142.8.0.1:1096-1127 dplane004 -
eth0: 10.0.0.1 External service 119.0.0.1:80 eth0: 10.0.0.2 eth0: 10.0.0.3 How
we configure …? dplane002 Match Src Action 10.0.0.1 Redirect dplane001 10.0.0.2 NAT 142.8.0.1:1032:1063 10.0.0.3 Redirect dplane003 Match Src Action 10.0.0.1 Redirect dplane001 10.0.0.2 Redirect dplane002 10.0.0.3 Redirect dplane003 dplane004 dplane001 dplane003 Match Src Action 10.0.0.1 NAT 142.8.0.1:1000-1031 10.0.0.2 Redirect dplane002 10.0.0.3 Redirect dplane003 Match Src Action 10.0.0.1 Redirect dplane001 10.0.0.2 Redirect dplane002 10.0.0.3 NAT 142.8.0.1:1064-1095 scheduler apiserver eth0: 10.0.0.4 Natlet Natlet Natlet Natlet Client-IP=10.0.0.1 NAT=142.8.0.1:1096-1127 Worker=dplane4 FabricNAT Global NAT Configuration Table ID Client Address NAT Public-IP:Port-Range Worker 1 10.0.0.1 142.8.0.1:1000-1031 dplane001 2 10.0.0.2 142.8.0.1:1032-1063 dplane002 3 10.0.0.3 142.8.0.1:1064-1095 dplane003 4 10.0.0.4 142.8.0.1:1096-1127 dplane004 Etcd Watch Etcd Watch Etcd Watch Etcd Watch -
eth0: 10.0.0.1 External service 119.0.0.1:80 eth0: 10.0.0.2 eth0: 10.0.0.3 How
we configure …? dplane002 dplane004 dplane001 dplane003 scheduler apiserver eth0: 10.0.0.4 Natlet Natlet Natlet Natlet Match Src Action 10.0.0.1 Redirect dplane001 10.0.0.2 NAT 142.8.0.1:1032:1063 10.0.0.3 Redirect dplane003 10.0.0.4 Redirect dplane004 Match Src Action 10.0.0.1 NAT 142.8.0.1:1000-1031 10.0.0.2 Redirect dplane002 10.0.0.3 Redirect dplane003 10.0.0.4 Redirect dplane004 Match Src Action 10.0.0.1 Redirect dplane001 10.0.0.2 Redirect dplane002 10.0.0.3 NAT 142.8.0.1:1064-1095 10.0.0.4 Redirect dplane004 NON-WORKER FabricNAT Global NAT Configuration Table ID Client Address NAT Public-IP:Port-Range Worker 1 10.0.0.1 142.8.0.1:1000-1031 dplane001 2 10.0.0.2 142.8.0.1:1032-1063 dplane002 3 10.0.0.3 142.8.0.1:1064-1095 dplane003 4 10.0.0.4 142.8.0.1:1096-1127 dplane004 Client-IP=10.0.0.1 NAT=142.8.0.1:1096-1127 Worker=dplane4 WORKER Match Src Action 10.0.0.1 Redirect dplane001 10.0.0.2 Redirect dplane002 10.0.0.3 Redirect dplane003 10.0.0.4 NAT 142.8.0.1:1096-1127 -
Recap FabricNAT SDN • SDN Design Principle • Without RPC
• With • Declarative Configuration • Resource-Watch • Reconciliation • Use CloudNative Parts • Use Consul for dataplane clustering • Use Etcd as Watchable Robust KVS Respecting K8s -
eth0: 10.0.0.1 External service 119.0.0.1:80 eth0: 10.0.0.2 eth0: 10.0.0.3 How
we configure …? scheduler apiserver eth0: 10.0.0.4 FabricNAT Global NAT Configuration Table ID Client Address NAT Public-IP:Port-Range Worker 1 10.0.0.1 142.8.0.1:1000-1031 dplane001 2 10.0.0.2 142.8.0.1:1032-1063 dplane002 3 10.0.0.3 142.8.0.1:1064-1095 dplane003 dplane002 lo: 10.255.0.2 tun: 10.99.0.1 Match Src Action 10.0.0.1 Redirect dplane001 10.0.0.2 NAT 142.8.0.1:1032:1063 10.0.0.3 Redirect dplane003 Match Src Action 10.0.0.1 Redirect dplane001 10.0.0.2 Redirect dplane002 10.0.0.3 Redirect dplane003 dplane004 lo: 10.255.0.4 tun: 10.99.0.1 dplane001 lo: 10.255.0.1 tun: 10.99.0.1 dplane003 lo: 10.255.0.3 tun: 10.99.0.1 Match Src Action 10.0.0.1 NAT 142.8.0.1:1000-1031 10.0.0.2 Redirect dplane002 10.0.0.3 Redirect dplane003 Match Src Action 10.0.0.1 Redirect dplane001 10.0.0.2 Redirect dplane002 10.0.0.3 NAT 142.8.0.1:1064-1095 Natlet Natlet Natlet Natlet dplane002 dplane004 dplane001 dplane003 -
eth0: 10.0.0.1 External service 119.0.0.1:80 eth0: 10.0.0.2 eth0: 10.0.0.3 How
we configure …? scheduler apiserver eth0: 10.0.0.4 FabricNAT Global NAT Configuration Table ID Client Address NAT Public-IP:Port-Range Worker 1 10.0.0.1 142.8.0.1:1000-1031 dplane001 2 10.0.0.2 142.8.0.1:1032-1063 dplane002 3 10.0.0.3 142.8.0.1:1064-1095 dplane003 dplane002 lo: 10.255.0.2 tun: 10.99.0.1 Match Src Action 10.0.0.1 Redirect dplane001 10.0.0.2 NAT 142.8.0.1:1032:1063 10.0.0.3 Redirect dplane003 Match Src Action 10.0.0.1 Redirect dplane001 10.0.0.2 Redirect dplane002 10.0.0.3 Redirect dplane003 dplane004 lo: 10.255.0.4 tun: 10.99.0.1 dplane001 lo: 10.255.0.1 tun: 10.99.0.1 dplane003 lo: 10.255.0.3 tun: 10.99.0.1 Match Src Action 10.0.0.1 NAT 142.8.0.1:1000-1031 10.0.0.2 Redirect dplane002 10.0.0.3 Redirect dplane003 Match Src Action 10.0.0.1 Redirect dplane001 10.0.0.2 Redirect dplane002 10.0.0.3 NAT 142.8.0.1:1064-1095 Natlet Natlet Natlet Natlet dplane002 dplane004 dplane001 dplane003 -
eth0: 10.0.0.1 External service 119.0.0.1:80 eth0: 10.0.0.2 eth0: 10.0.0.3 How
we configure …? scheduler apiserver eth0: 10.0.0.4 dplane002 lo: 10.255.0.2 tun: 10.99.0.1 Match Src Action 10.0.0.1 Redirect dplane001 10.0.0.2 NAT 142.8.0.1:1032:1063 10.0.0.3 Redirect dplane003 Match Src Action 10.0.0.1 Redirect dplane001 10.0.0.2 Redirect dplane002 10.0.0.3 Redirect dplane003 dplane004 lo: 10.255.0.4 tun: 10.99.0.1 dplane001 lo: 10.255.0.1 tun: 10.99.0.1 dplane003 lo: 10.255.0.3 tun: 10.99.0.1 Match Src Action 10.0.0.1 NAT 142.8.0.1:1000-1031 10.0.0.2 Redirect dplane002 10.0.0.3 Redirect dplane003 Match Src Action 10.0.0.1 Redirect dplane001 10.0.0.2 Redirect dplane002 10.0.0.3 NAT 142.8.0.1:1064-1095 Natlet Natlet Natlet Natlet ! ! ! ! 1. dplane001 was down 1. dplane001 was down 1. dplane001 was down 1. dplane001 was down dplane002 dplane004 dplane001 dplane003 scheduler ! 2. dplane001 was down FabricNAT Global NAT Configuration Table ID Client Address NAT Public-IP:Port-Range Micro-active 1 10.0.0.1 142.8.0.1:1000-1031 dplane001 2 10.0.0.2 142.8.0.1:1032-1063 dplane002 3 10.0.0.3 142.8.0.1:1064-1095 dplane003 -
eth0: 10.0.0.1 External service 119.0.0.1:80 eth0: 10.0.0.2 eth0: 10.0.0.3 How
we configure …? scheduler apiserver eth0: 10.0.0.4 FabricNAT Global NAT Configuration Table ID Client Address NAT Public-IP:Port-Range Micro-active 1 10.0.0.1 142.8.0.1:1000-1031 dplane001 2 10.0.0.2 142.8.0.1:1032-1063 dplane002 3 10.0.0.3 142.8.0.1:1064-1095 dplane003 4 10.0.0.4 142.8.0.1:1096-1127 NULL dplane002 lo: 10.255.0.2 tun: 10.99.0.1 dplane004 lo: 10.255.0.4 tun: 10.99.0.1 dplane003 lo: 10.255.0.3 tun: 10.99.0.1 Match Src Action 10.0.0.1 NAT 142.8.0.1:1000-1031 10.0.0.2 Redirect dplane002 10.0.0.3 Redirect dplane003 Natlet Natlet Natlet Natlet scheduler FabricNAT Global NAT Configuration Table-1 ID Client Address NAT Public-IP:Port-Range Worker 1 10.0.0.1 142.8.0.1:1000-1031 dplane004 2 10.0.0.2 142.8.0.1:1032-1063 dplane002 3 10.0.0.3 142.8.0.1:1064-1095 dplane003 dplane002 dplane004 dplane001 dplane003 ! ! ! Match Src Action 10.0.0.1 Redirect dplane004 10.0.0.2 NAT 142.8.0.1:1032:1063 10.0.0.3 Redirect dplane003 Match Src Action 10.0.0.1 NAT 142.8.0.1:1000-1031 10.0.0.2 Redirect dplane002 10.0.0.3 Redirect dplane003 Match Src Action 10.0.0.1 Redirect dplane004 10.0.0.2 Redirect dplane002 10.0.0.3 NAT 142.8.0.1:1064-1095 -
Recap FabricNAT SDN • SDN Design Principle • Without RPC
• With • Declarative Configuration • Resource-Watch • Reconciliation • Use CloudNative Parts • Use Consul for dataplane clustering • Use Etcd as Watchable Robust KVS Offload Complex Mechanism -
A Software’s Stability • “Proprietary Software” v.s. “Major OSS” •
Which looks be stable …? • Compare the Upgrade Period • Natlet: ?? • Etcd: ?? -
A Software’s Stability • “Proprietary Software” v.s. “Major OSS” •
Which looks be stable …? • Compare the Upgrade Period • Natlet: upgrade once/day • Etcd: upgrade once/month~ -
Software will be Broken • Development Difficulty • non broken
software → Really Really Difficult • reboot-able software → More Easy and Realistic • Basic Principle • Casual Maintenance • Frequency Upgrade -
eth0: 10.0.0.1 External service 119.0.0.1:80 eth0: 10.0.0.2 eth0: 10.0.0.3 How
we configure …? dplane002 Match Src Action 10.0.0.1 Redirect dplane001 10.0.0.2 NAT 142.8.0.1:1032:1063 10.0.0.3 Redirect dplane003 Match Src Action 10.0.0.1 Redirect dplane001 10.0.0.2 Redirect dplane002 10.0.0.3 Redirect dplane003 dplane004 dplane001 dplane003 Match Src Action 10.0.0.1 NAT 142.8.0.1:1000-1031 10.0.0.2 Redirect dplane002 10.0.0.3 Redirect dplane003 Match Src Action 10.0.0.1 Redirect dplane001 10.0.0.2 Redirect dplane002 10.0.0.3 NAT 142.8.0.1:1064-1095 scheduler apiserver eth0: 10.0.0.4 Natlet Natlet Natlet Natlet RPC FabricNAT Global NAT Configuration Table ID Client Address NAT Public-IP:Port-Range Worker 1 10.0.0.1 142.8.0.1:1000-1031 dplane001 2 10.0.0.2 142.8.0.1:1032-1063 dplane002 3 10.0.0.3 142.8.0.1:1064-1095 dplane003 4 10.0.0.4 142.8.0.1:1096-1127 dplane004 RPC RPC RPC Restarting -
eth0: 10.0.0.1 External service 119.0.0.1:80 eth0: 10.0.0.2 eth0: 10.0.0.3 How
we configure …? dplane002 Match Src Action 10.0.0.1 Redirect dplane001 10.0.0.2 NAT 142.8.0.1:1032:1063 10.0.0.3 Redirect dplane003 Match Src Action 10.0.0.1 Redirect dplane001 10.0.0.2 Redirect dplane002 10.0.0.3 Redirect dplane003 dplane004 dplane001 dplane003 Match Src Action 10.0.0.1 NAT 142.8.0.1:1000-1031 10.0.0.2 Redirect dplane002 10.0.0.3 Redirect dplane003 Match Src Action 10.0.0.1 Redirect dplane001 10.0.0.2 Redirect dplane002 10.0.0.3 NAT 142.8.0.1:1064-1095 scheduler apiserver eth0: 10.0.0.4 Natlet Natlet Natlet Natlet Watch FabricNAT Global NAT Configuration Table ID Client Address NAT Public-IP:Port-Range Worker 1 10.0.0.1 142.8.0.1:1000-1031 dplane001 2 10.0.0.2 142.8.0.1:1032-1063 dplane002 3 10.0.0.3 142.8.0.1:1064-1095 dplane003 4 10.0.0.4 142.8.0.1:1096-1127 dplane004 Watch Watch Watch Restarting Restart Watch & Reconcile -
Software will be Broken • Development Difficulty • non broken
software → Really Really Difficult • reboot-able software → More Easy and Realistic • Basic Principle • Casual Maintenance • Frequency Upgrade Enabled by Declarative Model and Reconciliation loop -
Deploy/Upgrade/Restart Components with Ansible Internet Gateway Service NAT 142.8.0.1 NAT
142.8.0.2 dplane001 dplane002 dplane003 dplane004 Natlet Natlet Natlet Natlet scheduler apiserver -
Monitoring :: Basic Metrics Internet Gateway Service NAT 142.8.0.1 NAT
142.8.0.2 Natlet dplane001 dplane002 dplane003 dplane004 Natlet Natlet Natlet scheduler apiserver k8s consul_sd_config kube_ state_ metrics -
Monitoring :: Dynamic Robot Internet Gateway Service apiserver 1. OpenStack
Token Issue 2. NAT Create 3. Replay Connection Information. Source IP Source Port etc.. 4. Check Connection Statement 5. NAT Delete NAT 142.8.0.1 NAT 142.8.0.2 -
Monitoring :: Static Robot Internet Gateway Service NAT 142.8.0.1 NAT
142.8.0.2 Use Blackbox Exporter consul_sd_config -
Next Step / Future Work • NAME based ACL •
Automated Robot Operation • Private Network Gateway -
NAME based Access Control Internet Gateway Service NAT 142.8.0.1 NAT
142.8.0.2 Allow-list ACL Allow-list ACL Allow-list Dest Proto api.external.linecorp.com https, http ftp.jaist.ac.jp ftp, https github.com ssh, https DNS -
Automated Robot Operation dplane002 Match Src Action 10.0.0.1 Redirect dplane001
10.0.0.2 NAT 142.8.0.1:1032:1063 10.0.0.3 Redirect dplane003 Match Src Action 10.0.0.1 Redirect dplane001 10.0.0.2 Redirect dplane002 10.0.0.3 Redirect dplane003 dplane004 dplane001 dplane003 Match Src Action 10.0.0.1 NAT 142.8.0.1:1000-1031 10.0.0.2 Redirect dplane002 10.0.0.3 Redirect dplane003 Match Src Action 10.0.0.1 Redirect dplane001 10.0.0.2 Redirect dplane002 10.0.0.3 NAT 142.8.0.1:1064-1095 Natlet Natlet Natlet Natlet scheduler apiserver Current Ops -
Automated Robot Operation dplane002 Match Src Action 10.0.0.1 Redirect dplane001
10.0.0.2 NAT 142.8.0.1:1032:1063 10.0.0.3 Redirect dplane003 Match Src Action 10.0.0.1 Redirect dplane001 10.0.0.2 Redirect dplane002 10.0.0.3 Redirect dplane003 dplane004 dplane001 dplane003 Match Src Action 10.0.0.1 NAT 142.8.0.1:1000-1031 10.0.0.2 Redirect dplane002 10.0.0.3 Redirect dplane003 Match Src Action 10.0.0.1 Redirect dplane001 10.0.0.2 Redirect dplane002 10.0.0.3 NAT 142.8.0.1:1064-1095 Natlet Natlet Natlet Natlet scheduler apiserver Automated Operation Manifest Monitor- ing -
Private Zone Private Zone A NW-A1 NW-A2 NW-GW Private Zone
B NW-B1 NW-B2 NW-GW -
Summary • Newly Provided “Internet Gateway” • We build the
new routing mechanism called “FabricNAT” for backend • Distributed NAT mech achieving “Small Blast Radius” and “Hyperscale” • SDN Design using CloudNative blocks • Consul for Easy/Massive clustering • Etcd for Declarative Configuration • Stateless software component can be restart casually • Many Network Challenge on our Private Cloud :)