Hyperscale distributed NAT system and software engineering

Hiroki Shirokura
LINE ITSC Verda Network Development Team Senior Software Engineer
https://linedevday.linecorp.com/2020/ja/sessions/2076
https://linedevday.linecorp.com/2020/en/sessions/2076

Transcript

2. Speaker : Hiroki Shirokura Senior Software Engineer at Verda In charge of Design/Develop-ing Network Service on Private Cloud Love SDN, Routing, Distributed System OSS contributor around Networking
3. Network Service Architecture 
 Optimized for LINE’s Private Cloud
4. Verda is a Private Cloud for LINE is based on OpenStack. since 2016~ FaaS PaaS IaaS NAT LB Bare
 metal Hypervisor 2,000+ Virtual Machine 55,000+ Physical Machine 20,000+ Verda Only +10,000/Half
5. Simple Networking @ Verda eBGP eBGP APPROACH
 Full L3 (No-L2, BGP Fabric) Protocol Reduction (No-Overlay) No Overlay Network Service We Get Simple & High Capacity
6. Simple Networking @ Verda eBGP eBGP APPROACH
 Full L3 (No-L2, BGP Fabric) Protocol Reduction (No-Overlay) No Overlay Network Service We Get Simple & High Capacity Issues No Managed Network policy Public-IP High Consumption We need gateway solution !
7. Let’s Provide Internet Gateway Service Internet Gateway Service NAT 142.8.0.1 NAT 142.8.0.2
8. Let’s Provide Internet Gateway Service Internet Gateway Service NAT 142.8.0.1 NAT 142.8.0.2 Fair Balanced Flexible Config Extensibility Hyperscale 4 Requirements Service Project A
 Uses 142.8.0.1 Service Project B Uses 142.8.0.2 %JCNNGPIG Allow-list ACL Allow-list ACL Allow-list Dest Proto api.external.linecorp.com https, http ftp.jaist.ac.jp ftp, https github.com ssh, https
9. Hyperscale Challenge…? Back to the Traditional Way Hardware
 Appliance Standby Hardware
 Appliance ! Active
10. Hyperscale Challenge…? Back to the Traditional Way Hardware
 Appliance Hardware
 Appliance Dead New Active Graceful
11. Hyperscale Challenge…? Back to the Traditional Way Hardware
 Appliance Hardware
 Appliance Dead New Active Big Blast Radius… NG#1 100% Blast Radius
12. Hyperscale Challenge…? Back to the Traditional Way Standby Active 50% Blast Radius 50% Blast Radius Standby Active 2NxM Cluster Operation Cost NG#2
13. Hyperscale Challenge…? Back to the Traditional Way Standby Active Standby Active L2 Network L2 Network Miss-match with Full L3 Network NG#3
14. Hyperscale Challenge…? Back to the Traditional Way Standby Active Standby Active L2 Network L2 Network Miss-match with Full L3 Network NG#3 Mechanism needs (1) N+1 Active/Active Single Cluster (2) Full L3 Simple Network Aware Let’s Construct the New One
15. FabricNAT: Hyperscale Distributed NAT System Match Src Action 10.0.0.1 NAT 142.8.0.1:1000-1031 10.0.0.2 Redirect dplane002 10.0.0.3 Redirect dplane003 Match Src Action 10.0.0.1 Redirect dplane001 10.0.0.2 NAT 142.8.0.1:1032:1063 10.0.0.3 Redirect dplane003 Match Src Action 10.0.0.1 Redirect dplane001 10.0.0.2 Redirect dplane002 10.0.0.3 NAT 142.8.0.1:1064-1095 Match Src Action 10.0.0.1 Redirect dplane001 10.0.0.2 Redirect dplane002 10.0.0.3 Redirect dplane003 NAT 142.8.0.1 dplane001 dplane002 dplane003 dplane004
16. FabricNAT: Hyperscale Distributed NAT System NAT 142.8.0.1 FabricNAT feature is N+1 Active/Active in Single Cluster Full L3 Network awareness Match Src Action 10.0.0.1 NAT 142.8.0.1:1000-1031 10.0.0.2 Redirect dplane002 10.0.0.3 Redirect dplane003 Match Src Action 10.0.0.1 Redirect dplane001 10.0.0.2 NAT 142.8.0.1:1032:1063 10.0.0.3 Redirect dplane003 Match Src Action 10.0.0.1 Redirect dplane001 10.0.0.2 Redirect dplane002 10.0.0.3 NAT 142.8.0.1:1064-1095 Match Src Action 10.0.0.1 Redirect dplane001 10.0.0.2 Redirect dplane002 10.0.0.3 Redirect dplane003 dplane001 dplane002 dplane003 dplane004 FabricNAT is structured with… BGP Anycast Policy Based Routing IPIP tunneling
17. FabricNAT: Hyperscale Distributed NAT System Match Src Action 10.0.0.1 NAT 142.8.0.1:1000-1031 10.0.0.2 Redirect dplane002 10.0.0.3 Redirect dplane003 Match Src Action 10.0.0.1 Redirect dplane001 10.0.0.2 NAT 142.8.0.1:1032:1063 10.0.0.3 Redirect dplane003 Match Src Action 10.0.0.1 Redirect dplane001 10.0.0.2 Redirect dplane002 10.0.0.3 NAT 142.8.0.1:1064-1095 Match Src Action 10.0.0.1 Redirect dplane001 10.0.0.2 Redirect dplane002 10.0.0.3 Redirect dplane003 dplane001 dplane002 dplane003 dplane004 eth0: 10.0.0.1 External service 119.0.0.1:80
18. FabricNAT: Hyperscale Distributed NAT System eth0: 10.0.0.1 Match Src Action 10.0.0.1 NAT 142.8.0.1:1000-1031 10.0.0.2 Redirect dplane002 10.0.0.3 Redirect dplane003 Match Src Action 10.0.0.1 Redirect dplane001 10.0.0.2 NAT 142.8.0.1:1032:1063 10.0.0.3 Redirect dplane003 Match Src Action 10.0.0.1 Redirect dplane001 10.0.0.2 Redirect dplane002 10.0.0.3 NAT 142.8.0.1:1064-1095 Match Src Action 10.0.0.1 Redirect dplane001 10.0.0.2 Redirect dplane002 10.0.0.3 Redirect dplane003 dplane001 dplane002 dplane003 dplane004 eth0: 10.0.0.1 FabricNAT Global NAT Configuration Table ID Client Address NAT Public-IP:Port-Range Worker 1 10.0.0.1 142.8.0.1:1000-1031 dplane001 2 10.0.0.2 142.8.0.1:1032-1063 dplane002 3 10.0.0.3 142.8.0.1:1064-1095 dplane003 External service 119.0.0.1:80 dplane003 dplane004 dplane001 dplane002 Match Src Action 10.0.0.1 Redirect dplane001 10.0.0.2 Redirect dplane002 10.0.0.3 NAT 142.8.0.1:1064-1095 Match Src Action 10.0.0.1 Redirect dplane001 10.0.0.2 Redirect dplane002 10.0.0.3 Redirect dplane003 Match Src Action 10.0.0.1 NAT 142.8.0.1:1000-1031 10.0.0.2 Redirect dplane002 10.0.0.3 Redirect dplane003 Match Src Action 10.0.0.1 Redirect dplane001 10.0.0.2 NAT 142.8.0.1:1032:1063 10.0.0.3 Redirect dplane003
19. FabricNAT: Hyperscale Distributed NAT System eth0: 10.0.0.1 Match Src Action 10.0.0.1 NAT 142.8.0.1:1000-1031 10.0.0.2 Redirect dplane002 10.0.0.3 Redirect dplane003 Match Src Action 10.0.0.1 Redirect dplane001 10.0.0.2 NAT 142.8.0.1:1032:1063 10.0.0.3 Redirect dplane003 Match Src Action 10.0.0.1 Redirect dplane001 10.0.0.2 Redirect dplane002 10.0.0.3 NAT 142.8.0.1:1064-1095 Match Src Action 10.0.0.1 Redirect dplane001 10.0.0.2 Redirect dplane002 10.0.0.3 Redirect dplane003 dplane001 dplane002 dplane003 dplane004 Match Src Action 10.0.0.1 NAT 142.8.0.1:1000-1031 10.0.0.2 Redirect dplane002 10.0.0.3 Redirect dplane003 Match Src Action 10.0.0.1 Redirect dplane001 10.0.0.2 NAT 142.8.0.1:1032:1063 10.0.0.3 Redirect dplane003 Match Src Action 10.0.0.1 Redirect dplane001 10.0.0.2 Redirect dplane002 10.0.0.3 NAT 142.8.0.1:1064-1095 Match Src Action 10.0.0.1 Redirect dplane001 10.0.0.2 Redirect dplane002 10.0.0.3 Redirect dplane003 dplane001 lo: 10.255.0.1 tun: 10.99.0.1 dplane002 lo: 10.255.0.2 tun: 10.99.0.1 dplane003 lo: 10.255.0.3 tun: 10.99.0.1 dplane004 lo: 10.255.0.4 tun: 10.99.0.1 Match Src Action 10.0.0.1 Redirect dplane001 10.0.0.2 Redirect dplane002 10.0.0.3 NAT 142.8.0.1:1064-1095 Match Src Action 10.0.0.1 Redirect dplane001 10.0.0.2 Redirect dplane002 10.0.0.3 Redirect dplane003 Match Src Action 10.0.0.1 NAT 142.8.0.1:1000-1031 10.0.0.2 Redirect dplane002 10.0.0.3 Redirect dplane003 Match Src Action 10.0.0.1 Redirect dplane001 10.0.0.2 NAT 142.8.0.1:1032:1063 10.0.0.3 Redirect dplane003 dplane003 dplane004 dplane001 dplane002 eth0: 10.0.0.1 FabricNAT Global NAT Configuration Table ID Client Address NAT Public-IP:Port-Range Worker 1 10.0.0.1 142.8.0.1:1000-1031 dplane001 2 10.0.0.2 142.8.0.1:1032-1063 dplane002 3 10.0.0.3 142.8.0.1:1064-1095 dplane003 External service 119.0.0.1:80
20. FabricNAT: Hyperscale Distributed NAT System eth0: 10.0.0.1 Match Src Action 10.0.0.1 NAT 142.8.0.1:1000-1031 10.0.0.2 Redirect dplane002 10.0.0.3 Redirect dplane003 Match Src Action 10.0.0.1 Redirect dplane001 10.0.0.2 NAT 142.8.0.1:1032:1063 10.0.0.3 Redirect dplane003 Match Src Action 10.0.0.1 Redirect dplane001 10.0.0.2 Redirect dplane002 10.0.0.3 NAT 142.8.0.1:1064-1095 Match Src Action 10.0.0.1 Redirect dplane001 10.0.0.2 Redirect dplane002 10.0.0.3 Redirect dplane003 dplane001 dplane002 dplane003 dplane004 eth0: 10.0.0.1 FabricNAT Global NAT Configuration Table ID Client Address NAT Public-IP:Port-Range Worker 1 10.0.0.1 142.8.0.1:1000-1031 dplane001 2 10.0.0.2 142.8.0.1:1032-1063 dplane002 3 10.0.0.3 142.8.0.1:1064-1095 dplane003 External service 119.0.0.1:80
21. FabricNAT: Hyperscale Distributed NAT System eth0: 10.0.0.1 External service 119.0.0.1:80 FabricNAT Global NAT Configuration Table ID Client Address NAT Public-IP:Port-Range Worker 1 10.0.0.1 142.8.0.1:1000-1031 dplane001 2 10.0.0.2 142.8.0.1:1032-1063 dplane002 3 10.0.0.3 142.8.0.1:1064-1095 dplane003 src: 10.0.0.1:2859 dst: 119.0.0.1:80 proto: tcp Match Src Action 10.0.0.1 NAT 142.8.0.1:1000-1031 10.0.0.2 Redirect dplane002 10.0.0.3 Redirect dplane003 Match Src Action 10.0.0.1 Redirect dplane001 10.0.0.2 NAT 142.8.0.1:1032:1063 10.0.0.3 Redirect dplane003 Match Src Action 10.0.0.1 Redirect dplane001 10.0.0.2 Redirect dplane002 10.0.0.3 NAT 142.8.0.1:1064-1095 Match Src Action 10.0.0.1 Redirect dplane001 10.0.0.2 Redirect dplane002 10.0.0.3 Redirect dplane003 dplane001 lo: 10.255.0.1 tun: 10.99.0.1 dplane002 lo: 10.255.0.2 tun: 10.99.0.1 dplane003 lo: 10.255.0.3 tun: 10.99.0.1 dplane004 lo: 10.255.0.4 tun: 10.99.0.1 dplane003 dplane004 dplane001 dplane002
22. FabricNAT: Hyperscale Distributed NAT System eth0: 10.0.0.1 External service 119.0.0.1:80 FabricNAT Global NAT Configuration Table ID Client Address NAT Public-IP:Port-Range Worker 1 10.0.0.1 142.8.0.1:1000-1031 dplane001 2 10.0.0.2 142.8.0.1:1032-1063 dplane002 3 10.0.0.3 142.8.0.1:1064-1095 dplane003 dst: 10.99.0.1 proto: ipip src: 10.0.0.1:2859 dst: 119.0.0.1:80 proto: tcp Match Src Action 10.0.0.1 NAT 142.8.0.1:1000-1031 10.0.0.2 Redirect dplane002 10.0.0.3 Redirect dplane003 Match Src Action 10.0.0.1 Redirect dplane001 10.0.0.2 NAT 142.8.0.1:1032:1063 10.0.0.3 Redirect dplane003 Match Src Action 10.0.0.1 Redirect dplane001 10.0.0.2 Redirect dplane002 10.0.0.3 NAT 142.8.0.1:1064-1095 Match Src Action 10.0.0.1 Redirect dplane001 10.0.0.2 Redirect dplane002 10.0.0.3 Redirect dplane003 dplane001 lo: 10.255.0.1 tun: 10.99.0.1 dplane002 lo: 10.255.0.2 tun: 10.99.0.1 dplane003 lo: 10.255.0.3 tun: 10.99.0.1 dplane004 lo: 10.255.0.4 tun: 10.99.0.1 dplane003 dplane004 dplane001 dplane002
23. Match Src Action 10.0.0.1 Redirect dplane001 10.0.0.2 Redirect dplane002 10.0.0.3 NAT 142.8.0.1:1064-1095 Match Src Action 10.0.0.1 Redirect dplane001 10.0.0.2 Redirect dplane002 10.0.0.3 Redirect dplane003 dplane003 lo: 10.255.0.3 tun: 10.99.0.1 dplane004 lo: 10.255.0.4 tun: 10.99.0.1 FabricNAT: Hyperscale Distributed NAT System eth0: 10.0.0.1 External service 119.0.0.1:80 FabricNAT Global NAT Configuration Table ID Client Address NAT Public-IP:Port-Range Worker 1 10.0.0.1 142.8.0.1:1000-1031 dplane001 2 10.0.0.2 142.8.0.1:1032-1063 dplane002 3 10.0.0.3 142.8.0.1:1064-1095 dplane003 BGP Advertisement 10.99.0.1/32 BGP Advertisement 10.99.0.1/32 BGP Advertisement 10.99.0.1/32 BGP Advertisement 10.99.0.1/32 Match Src Action 10.0.0.1 NAT 142.8.0.1:1000-1031 10.0.0.2 Redirect dplane002 10.0.0.3 Redirect dplane003 Match Src Action 10.0.0.1 Redirect dplane001 10.0.0.2 NAT 142.8.0.1:1032:1063 10.0.0.3 Redirect dplane003 dplane001 lo: 10.255.0.1 tun: 10.99.0.1 dplane002 lo: 10.255.0.2 tun: 10.99.0.1 dplane003 dplane004 dplane001 dplane002 dst: 10.99.0.1 proto: ipip src: 10.0.0.1:2859 dst: 119.0.0.1:80 proto: tcp
24. Match Src Action 10.0.0.1 Redirect dplane001 10.0.0.2 Redirect dplane002 10.0.0.3 NAT 142.8.0.1:1064-1095 Match Src Action 10.0.0.1 Redirect dplane001 10.0.0.2 Redirect dplane002 10.0.0.3 Redirect dplane003 dplane003 lo: 10.255.0.3 tun: 10.99.0.1 dplane004 lo: 10.255.0.4 tun: 10.99.0.1 FabricNAT: Hyperscale Distributed NAT System eth0: 10.0.0.1 External service 119.0.0.1:80 FabricNAT Global NAT Configuration Table ID Client Address NAT Public-IP:Port-Range Worker 1 10.0.0.1 142.8.0.1:1000-1031 dplane001 2 10.0.0.2 142.8.0.1:1032-1063 dplane002 3 10.0.0.3 142.8.0.1:1064-1095 dplane003 Match Src Action 10.0.0.1 NAT 142.8.0.1:1000-1031 10.0.0.2 Redirect dplane002 10.0.0.3 Redirect dplane003 Match Src Action 10.0.0.1 Redirect dplane001 10.0.0.2 NAT 142.8.0.1:1032:1063 10.0.0.3 Redirect dplane003 dplane001 lo: 10.255.0.1 tun: 10.99.0.1 dplane002 lo: 10.255.0.2 tun: 10.99.0.1 dplane003 dplane004 dplane001 dplane002 dst: 10.99.0.1 proto: ipip src: 10.0.0.1:2859 dst: 119.0.0.1:80 proto: tcp
25. Match Src Action 10.0.0.1 NAT 142.8.0.1:1000-1031 10.0.0.2 Redirect dplane002 10.0.0.3 Redirect dplane003 Match Src Action 10.0.0.1 Redirect dplane001 10.0.0.2 NAT 142.8.0.1:1032:1063 10.0.0.3 Redirect dplane003 Match Src Action 10.0.0.1 Redirect dplane001 10.0.0.2 Redirect dplane002 10.0.0.3 NAT 142.8.0.1:1064-1095 Match Src Action 10.0.0.1 Redirect dplane001 10.0.0.2 Redirect dplane002 10.0.0.3 Redirect dplane003 dplane001 lo: 10.255.0.1 tun: 10.99.0.1 dplane002 lo: 10.255.0.2 tun: 10.99.0.1 dplane003 lo: 10.255.0.3 tun: 10.99.0.1 dplane004 lo: 10.255.0.4 tun: 10.99.0.1 FabricNAT: Hyperscale Distributed NAT System eth0: 10.0.0.1 External service 119.0.0.1:80 FabricNAT Global NAT Configuration Table ID Client Address NAT Public-IP:Port-Range Worker 1 10.0.0.1 142.8.0.1:1000-1031 dplane001 2 10.0.0.2 142.8.0.1:1032-1063 dplane002 3 10.0.0.3 142.8.0.1:1064-1095 dplane003 dplane003 dplane004 dplane001 dplane002 src: 10.0.0.1:2859 dst: 119.0.0.1:80 proto: tcp
26. Match Src Action 10.0.0.1 Redirect dplane001 10.0.0.2 Redirect dplane002 10.0.0.3 NAT 142.8.0.1:1064-1095 Match Src Action 10.0.0.1 Redirect dplane001 10.0.0.2 Redirect dplane002 10.0.0.3 Redirect dplane003 dplane003 lo: 10.255.0.3 tun: 10.99.0.1 dplane004 lo: 10.255.0.4 tun: 10.99.0.1 FabricNAT: Hyperscale Distributed NAT System eth0: 10.0.0.1 External service 119.0.0.1:80 FabricNAT Global NAT Configuration Table ID Client Address NAT Public-IP:Port-Range Worker 1 10.0.0.1 142.8.0.1:1000-1031 dplane001 2 10.0.0.2 142.8.0.1:1032-1063 dplane002 3 10.0.0.3 142.8.0.1:1064-1095 dplane003 Match Src Action 10.0.0.1 NAT 142.8.0.1:1000-1031 10.0.0.2 Redirect dplane002 10.0.0.3 Redirect dplane003 Match Src Action 10.0.0.1 Redirect dplane001 10.0.0.2 NAT 142.8.0.1:1032:1063 10.0.0.3 Redirect dplane003 dplane001 lo: 10.255.0.1 tun: 10.99.0.1 dplane002 lo: 10.255.0.2 tun: 10.99.0.1 dplane003 dplane004 dplane001 dplane002 src: 10.0.0.1:2859 dst: 119.0.0.1:80 proto: tcp
27. Match Src Action 10.0.0.1 NAT 142.8.0.1:1000-1031 10.0.0.2 Redirect dplane002 10.0.0.3 Redirect dplane003 Match Src Action 10.0.0.1 Redirect dplane001 10.0.0.2 NAT 142.8.0.1:1032:1063 10.0.0.3 Redirect dplane003 dplane001 lo: 10.255.0.1 tun: 10.99.0.1 dplane002 lo: 10.255.0.2 tun: 10.99.0.1 Match Src Action 10.0.0.1 Redirect dplane001 10.0.0.2 Redirect dplane002 10.0.0.3 NAT 142.8.0.1:1064-1095 Match Src Action 10.0.0.1 Redirect dplane001 10.0.0.2 Redirect dplane002 10.0.0.3 Redirect dplane003 dplane003 lo: 10.255.0.3 tun: 10.99.0.1 dplane004 lo: 10.255.0.4 tun: 10.99.0.1 FabricNAT: Hyperscale Distributed NAT System eth0: 10.0.0.1 External service 119.0.0.1:80 FabricNAT Global NAT Configuration Table ID Client Address NAT Public-IP:Port-Range Worker 1 10.0.0.1 142.8.0.1:1000-1031 dplane001 2 10.0.0.2 142.8.0.1:1032-1063 dplane002 3 10.0.0.3 142.8.0.1:1064-1095 dplane003 dplane003 dplane004 dplane001 dplane002 dst:10.255.0.1 proto: ipip src: 10.0.0.1:2859 dst: 119.0.0.1:80 proto: tcp
28. Match Src Action 10.0.0.1 NAT 142.8.0.1:1000-1031 10.0.0.2 Redirect dplane002 10.0.0.3 Redirect dplane003 Match Src Action 10.0.0.1 Redirect dplane001 10.0.0.2 NAT 142.8.0.1:1032:1063 10.0.0.3 Redirect dplane003 dplane001 lo: 10.255.0.1 tun: 10.99.0.1 dplane002 lo: 10.255.0.2 tun: 10.99.0.1 FabricNAT: Hyperscale Distributed NAT System eth0: 10.0.0.1 External service 119.0.0.1:80 FabricNAT Global NAT Configuration Table ID Client Address NAT Public-IP:Port-Range Worker 1 10.0.0.1 142.8.0.1:1000-1031 dplane001 2 10.0.0.2 142.8.0.1:1032-1063 dplane002 3 10.0.0.3 142.8.0.1:1064-1095 dplane003 dst:10.255.0.1 proto: ipip src: 10.0.0.1:2859 dst: 119.0.0.1:80 proto: tcp Match Src Action 10.0.0.1 Redirect dplane001 10.0.0.2 Redirect dplane002 10.0.0.3 NAT 142.8.0.1:1064-1095 Match Src Action 10.0.0.1 Redirect dplane001 10.0.0.2 Redirect dplane002 10.0.0.3 Redirect dplane003 dplane003 lo: 10.255.0.3 tun: 10.99.0.1 dplane004 lo: 10.255.0.4 tun: 10.99.0.1 dplane003 dplane004 dplane001 dplane002
29. FabricNAT: Hyperscale Distributed NAT System eth0: 10.0.0.1 External service 119.0.0.1:80 FabricNAT Global NAT Configuration Table ID Client Address NAT Public-IP:Port-Range Worker 1 10.0.0.1 142.8.0.1:1000-1031 dplane001 2 10.0.0.2 142.8.0.1:1032-1063 dplane002 3 10.0.0.3 142.8.0.1:1064-1095 dplane003 Match Src Action 10.0.0.1 NAT 142.8.0.1:1000-1031 10.0.0.2 Redirect dplane002 10.0.0.3 Redirect dplane003 Match Src Action 10.0.0.1 Redirect dplane001 10.0.0.2 NAT 142.8.0.1:1032:1063 10.0.0.3 Redirect dplane003 dplane001 lo: 10.255.0.1 tun: 10.99.0.1 dplane002 lo: 10.255.0.2 tun: 10.99.0.1 src: 10.0.0.1:2859 dst: 119.0.0.1:80 proto: tcp Match Src Action 10.0.0.1 Redirect dplane001 10.0.0.2 Redirect dplane002 10.0.0.3 NAT 142.8.0.1:1064-1095 Match Src Action 10.0.0.1 Redirect dplane001 10.0.0.2 Redirect dplane002 10.0.0.3 Redirect dplane003 dplane003 lo: 10.255.0.3 tun: 10.99.0.1 dplane004 lo: 10.255.0.4 tun: 10.99.0.1 dplane003 dplane004 dplane001 dplane002
30. FabricNAT: Hyperscale Distributed NAT System eth0: 10.0.0.1 External service 119.0.0.1:80 FabricNAT Global NAT Configuration Table ID Client Address NAT Public-IP:Port-Range Worker 1 10.0.0.1 142.8.0.1:1000-1031 dplane001 2 10.0.0.2 142.8.0.1:1032-1063 dplane002 3 10.0.0.3 142.8.0.1:1064-1095 dplane003 src: 142.8.0.1:1010 dst: 119.0.0.1:80 proto: tcp Match Src Action 10.0.0.1 NAT 142.8.0.1:1000-1031 10.0.0.2 Redirect dplane002 10.0.0.3 Redirect dplane003 Match Src Action 10.0.0.1 Redirect dplane001 10.0.0.2 NAT 142.8.0.1:1032:1063 10.0.0.3 Redirect dplane003 dplane001 lo: 10.255.0.1 tun: 10.99.0.1 dplane002 lo: 10.255.0.2 tun: 10.99.0.1 Match Src Action 10.0.0.1 Redirect dplane001 10.0.0.2 Redirect dplane002 10.0.0.3 NAT 142.8.0.1:1064-1095 Match Src Action 10.0.0.1 Redirect dplane001 10.0.0.2 Redirect dplane002 10.0.0.3 Redirect dplane003 dplane003 lo: 10.255.0.3 tun: 10.99.0.1 dplane004 lo: 10.255.0.4 tun: 10.99.0.1 dplane003 dplane004 dplane001 dplane002
31. FabricNAT: Hyperscale Distributed NAT System eth0: 10.0.0.1 External service 119.0.0.1:80 FabricNAT Global NAT Configuration Table ID Client Address NAT Public-IP:Port-Range Worker 1 10.0.0.1 142.8.0.1:1000-1031 dplane001 2 10.0.0.2 142.8.0.1:1032-1063 dplane002 3 10.0.0.3 142.8.0.1:1064-1095 dplane003 src: 142.8.0.1:1010 dst: 119.0.0.1:80 proto: tcp Match Src Action 10.0.0.1 NAT 142.8.0.1:1000-1031 10.0.0.2 Redirect dplane002 10.0.0.3 Redirect dplane003 Match Src Action 10.0.0.1 Redirect dplane001 10.0.0.2 NAT 142.8.0.1:1032:1063 10.0.0.3 Redirect dplane003 dplane001 lo: 10.255.0.1 tun: 10.99.0.1 dplane002 lo: 10.255.0.2 tun: 10.99.0.1 Match Src Action 10.0.0.1 Redirect dplane001 10.0.0.2 Redirect dplane002 10.0.0.3 NAT 142.8.0.1:1064-1095 Match Src Action 10.0.0.1 Redirect dplane001 10.0.0.2 Redirect dplane002 10.0.0.3 Redirect dplane003 dplane003 lo: 10.255.0.3 tun: 10.99.0.1 dplane004 lo: 10.255.0.4 tun: 10.99.0.1 dplane003 dplane004 dplane001 dplane002
32. Match Src Action 10.0.0.1 NAT 142.8.0.1:1000-1031 10.0.0.2 Redirect dplane002 10.0.0.3 Redirect dplane003 dplane001 FabricNAT: Hyperscale Distributed NAT System eth0: 10.0.0.1 External service 119.0.0.1:80 src: 142.8.0.1:1010 dst: 119.0.0.1:80 proto: tcp FabricNAT Global NAT Configuration Table ID Client Address NAT Public-IP:Port-Range Worker 1 10.0.0.1 142.8.0.1:1000-1031 dplane001 2 10.0.0.2 142.8.0.1:1032-1063 dplane002 3 10.0.0.3 142.8.0.1:1064-1095 dplane003 Match Src Action 10.0.0.1 Redirect dplane001 10.0.0.2 NAT 142.8.0.1:1032:1063 10.0.0.3 Redirect dplane003 dplane002 lo: 10.255.0.2 tun: 10.99.0.1 Match Src Action 10.0.0.1 Redirect dplane001 10.0.0.2 Redirect dplane002 10.0.0.3 NAT 142.8.0.1:1064-1095 Match Src Action 10.0.0.1 Redirect dplane001 10.0.0.2 Redirect dplane002 10.0.0.3 Redirect dplane003 dplane003 lo: 10.255.0.3 tun: 10.99.0.1 dplane004 lo: 10.255.0.4 tun: 10.99.0.1 dplane003 dplane004 dplane002
33. Match Src Action 10.0.0.1 NAT 142.8.0.1:1000-1031 10.0.0.2 Redirect dplane002 10.0.0.3 Redirect dplane003 dplane001 FabricNAT: Hyperscale Distributed NAT System eth0: 10.0.0.1 External service 119.0.0.1:80 src: 142.8.0.1:1010 dst: 119.0.0.1:80 proto: tcp FabricNAT Global NAT Configuration Table ID Client Address NAT Public-IP:Port-Range Worker 1 10.0.0.1 142.8.0.1:1000-1031 dplane001 2 10.0.0.2 142.8.0.1:1032-1063 dplane002 3 10.0.0.3 142.8.0.1:1064-1095 dplane003 Match Dst Action 142.8.0.1:1000-1031 Red dplane001 10.0.0.2 NAT 142.8.0.1:1032: 10.0.0.3 Redirect dplane003 dplane002 lo: 10.255.0.2 tun: 10.99.0.1 Match Dst Action 142.8.0.1:1000-1031 Red dplane001 10.0.0.2 Redirect dplane002 10.0.0.3 NAT 142.8.0.1:1064- Match Dst Action 142.8.0.1:1000-1031 Red dplane001 10.0.0.2 Redirect dplane002 10.0.0.3 Redirect dplane003 dplane003 lo: 10.255.0.3 tun: 10.99.0.1 dplane004 lo: 10.255.0.4 tun: 10.99.0.1 dplane003 dplane004 dplane002
34. Recap FabricNAT • Pre allocate Port Range for Session Quota
 • Worker Election per each Clients and No State Sync for Scalability
 • 2-Step Routing to Construct Symmetric Routing
35. FabricNAT: Hyperscale Distributed NAT System eth0: 10.0.0.1 External service 119.0.0.1:80 FabricNAT Global NAT Configuration Table ID Client Address NAT Public-IP:Port-Range Worker 1 10.0.0.1 142.8.0.1:1000-1031 dplane001 2 10.0.0.2 142.8.0.1:1032-1063 dplane002 3 10.0.0.3 142.8.0.1:1064-1095 dplane003 src: 142.8.0.1:1010 dst: 119.0.0.1:80 proto: tcp Match Src Action 10.0.0.1 NAT 142.8.0.1:1000-1031 10.0.0.2 Redirect dplane002 10.0.0.3 Redirect dplane003 Match Src Action 10.0.0.1 Redirect dplane001 10.0.0.2 NAT 142.8.0.1:1032:1063 10.0.0.3 Redirect dplane003 dplane001 dplane002 Match Src Action 10.0.0.1 Redirect dplane001 10.0.0.2 Redirect dplane002 10.0.0.3 NAT 142.8.0.1:1064-1095 Match Src Action 10.0.0.1 Redirect dplane001 10.0.0.2 Redirect dplane002 10.0.0.3 Redirect dplane003 dplane003 dplane004 Working as N+1 redundancy…?
36. N+1 Active/Active…? 1/N Blast Radius …? eth0: 10.0.0.1 External service 119.0.0.1:80 Match Src Action 10.0.0.1 NAT 142.8.0.1:1000-1031 10.0.0.2 Redirect dplane002 10.0.0.3 Redirect dplane003 Match Src Action 10.0.0.1 Redirect dplane001 10.0.0.2 NAT 142.8.0.1:1032:1063 10.0.0.3 Redirect dplane003 dplane001 lo: 10.255.0.1 tun: 10.99.0.1 dplane002 lo: 10.255.0.2 tun: 10.99.0.1 Match Src Action 10.0.0.1 Redirect dplane001 10.0.0.2 Redirect dplane002 10.0.0.3 NAT 142.8.0.1:1064-1095 Match Src Action 10.0.0.1 Redirect dplane001 10.0.0.2 Redirect dplane002 10.0.0.3 Redirect dplane003 dplane003 lo: 10.255.0.3 tun: 10.99.0.1 dplane004 lo: 10.255.0.4 tun: 10.99.0.1 eth0: 10.0.0.2 eth0: 10.0.0.3 FabricNAT Global NAT Configuration Table ID Client Address NAT Public-IP:Port-Range Worker 1 10.0.0.1 142.8.0.1:1000-1031 dplane001 2 10.0.0.2 142.8.0.1:1032-1063 dplane002 3 10.0.0.3 142.8.0.1:1064-1095 dplane003 eth0: 10.0.0.1 eth0: 10.0.0.2 eth0: 10.0.0.3 dplane003 dplane004 dplane001 dplane002
37. N+1 Active/Active…? 1/N Blast Radius …? eth0: 10.0.0.1 External service 119.0.0.1:80 Match Src Action 10.0.0.1 NAT 142.8.0.1:1000-1031 10.0.0.2 Redirect dplane002 10.0.0.3 Redirect dplane003 Match Src Action 10.0.0.1 Redirect dplane001 10.0.0.2 NAT 142.8.0.1:1032:1063 10.0.0.3 Redirect dplane003 dplane001 dplane002 Match Src Action 10.0.0.1 Redirect dplane001 10.0.0.2 Redirect dplane002 10.0.0.3 NAT 142.8.0.1:1064-1095 Match Src Action 10.0.0.1 Redirect dplane001 10.0.0.2 Redirect dplane002 10.0.0.3 Redirect dplane003 dplane003 dplane004 eth0: 10.0.0.2 eth0: 10.0.0.3 FabricNAT Global NAT Configuration Table ID Client Address NAT Public-IP:Port-Range Worker 1 10.0.0.1 142.8.0.1:1000-1031 dplane001 2 10.0.0.2 142.8.0.1:1032-1063 dplane002 3 10.0.0.3 142.8.0.1:1064-1095 dplane003 eth0: 10.0.0.1 eth0: 10.0.0.2 eth0: 10.0.0.3
38. N+1 Active/Active…? 1/N Blast Radius …? eth0: 10.0.0.1 External service 119.0.0.1:80 Match Src Action 10.0.0.1 NAT 142.8.0.1:1000-1031 10.0.0.2 Redirect dplane002 10.0.0.3 Redirect dplane003 Match Src Action 10.0.0.1 Redirect dplane001 10.0.0.2 NAT 142.8.0.1:1032:1063 10.0.0.3 Redirect dplane003 dplane001 dplane002 Match Src Action 10.0.0.1 Redirect dplane001 10.0.0.2 Redirect dplane002 10.0.0.3 NAT 142.8.0.1:1064-1095 Match Src Action 10.0.0.1 Redirect dplane001 10.0.0.2 Redirect dplane002 10.0.0.3 Redirect dplane003 dplane003 dplane004 eth0: 10.0.0.2 eth0: 10.0.0.3 FabricNAT Global NAT Configuration Table ID Client Address NAT Public-IP:Port-Range Worker 1 10.0.0.1 142.8.0.1:1000-1031 dplane001 2 10.0.0.2 142.8.0.1:1032-1063 dplane002 3 10.0.0.3 142.8.0.1:1064-1095 dplane003 eth0: 10.0.0.1 eth0: 10.0.0.2 eth0: 10.0.0.3 External service 119.0.0.1:80 eth0: 10.0.0.1
39. N+1 Active/Active…? 1/N Blast Radius …? eth0: 10.0.0.1 External service 119.0.0.1:80 Match Src Action 10.0.0.1 NAT 142.8.0.1:1000-1031 10.0.0.2 Redirect dplane002 10.0.0.3 Redirect dplane003 Match Src Action 10.0.0.1 Redirect dplane001 10.0.0.2 NAT 142.8.0.1:1032:1063 10.0.0.3 Redirect dplane003 dplane001 lo: 10.255.0.1 tun: 10.99.0.1 dplane002 lo: 10.255.0.2 tun: 10.99.0.1 Match Src Action 10.0.0.1 Redirect dplane001 10.0.0.2 Redirect dplane002 10.0.0.3 NAT 142.8.0.1:1064-1095 Match Src Action 10.0.0.1 Redirect dplane001 10.0.0.2 Redirect dplane002 10.0.0.3 Redirect dplane003 dplane003 lo: 10.255.0.3 tun: 10.99.0.1 dplane004 lo: 10.255.0.4 tun: 10.99.0.1 eth0: 10.0.0.2 eth0: 10.0.0.3 FabricNAT Global NAT Configuration Table ID Client Address NAT Public-IP:Port-Range Micro-active 1 10.0.0.1 142.8.0.1:1000-1031 dplane001 2 10.0.0.2 142.8.0.1:1032-1063 dplane002 3 10.0.0.3 142.8.0.1:1064-1095 dplane003 eth0: 10.0.0.1 eth0: 10.0.0.2 eth0: 10.0.0.3 BGP Advertisement 10.99.0.1/32 BGP Advertisement 10.99.0.1/32 BGP Advertisement 10.99.0.1/32 BGP Advertisement 10.99.0.1/32 dplane001 lo: 10.255.0.1 tun: 10.99.0.1 dplane002 lo: 10.255.0.2 tun: 10.99.0.1 dplane003 lo: 10.255.0.3 tun: 10.99.0.1 dplane004 lo: 10.255.0.4 tun: 10.99.0.1 Match Src Action 10.0.0.1 NAT 142.8.0.1:1000-1031 10.0.0.2 Redirect dplane002 10.0.0.3 Redirect dplane003 Match Src Action 10.0.0.1 Redirect dplane001 10.0.0.2 NAT 142.8.0.1:1032:1063 10.0.0.3 Redirect dplane003 Match Src Action 10.0.0.1 Redirect dplane001 10.0.0.2 Redirect dplane002 10.0.0.3 NAT 142.8.0.1:1064-1095 Match Src Action 10.0.0.1 Redirect dplane001 10.0.0.2 Redirect dplane002 10.0.0.3 Redirect dplane003 eth0: 10.0.0.1 dplane003 dplane004 dplane001 dplane002
40. N+1 Active/Active…? 1/N Blast Radius …? eth0: 10.0.0.1 External service 119.0.0.1:80 Match Src Action 10.0.0.1 NAT 142.8.0.1:1000-1031 10.0.0.2 Redirect dplane002 10.0.0.3 Redirect dplane003 Match Src Action 10.0.0.1 Redirect dplane001 10.0.0.2 NAT 142.8.0.1:1032:1063 10.0.0.3 Redirect dplane003 dplane001 lo: 10.255.0.1 tun: 10.99.0.1 dplane002 lo: 10.255.0.2 tun: 10.99.0.1 Match Src Action 10.0.0.1 Redirect dplane001 10.0.0.2 Redirect dplane002 10.0.0.3 NAT 142.8.0.1:1064-1095 Match Src Action 10.0.0.1 Redirect dplane001 10.0.0.2 Redirect dplane002 10.0.0.3 Redirect dplane003 dplane003 lo: 10.255.0.3 tun: 10.99.0.1 dplane004 lo: 10.255.0.4 tun: 10.99.0.1 eth0: 10.0.0.2 eth0: 10.0.0.3 FabricNAT Global NAT Configuration Table ID Client Address NAT Public-IP:Port-Range Micro-active 1 10.0.0.1 142.8.0.1:1000-1031 dplane001 2 10.0.0.2 142.8.0.1:1032-1063 dplane002 3 10.0.0.3 142.8.0.1:1064-1095 dplane003 eth0: 10.0.0.1 eth0: 10.0.0.2 eth0: 10.0.0.3 dplane001 lo: 10.255.0.1 tun: 10.99.0.1 dplane002 lo: 10.255.0.2 tun: 10.99.0.1 dplane003 lo: 10.255.0.3 tun: 10.99.0.1 dplane004 lo: 10.255.0.4 tun: 10.99.0.1 External service 119.0.0.1:80 eth0: 10.0.0.1 eth0: 10.0.0.2 eth0: 10.0.0.3 dplane003 dplane004 dplane001 dplane002
41. N+1 Active/Active…? 1/N Blast Radius …? eth0: 10.0.0.1 External service 119.0.0.1:80 Match Src Action 10.0.0.1 NAT 142.8.0.1:1000-1031 10.0.0.2 Redirect dplane002 10.0.0.3 Redirect dplane003 Match Src Action 10.0.0.1 Redirect dplane001 10.0.0.2 NAT 142.8.0.1:1032:1063 10.0.0.3 Redirect dplane003 dplane001 lo: 10.255.0.1 tun: 10.99.0.1 dplane002 lo: 10.255.0.2 tun: 10.99.0.1 Match Src Action 10.0.0.1 Redirect dplane001 10.0.0.2 Redirect dplane002 10.0.0.3 NAT 142.8.0.1:1064-1095 Match Src Action 10.0.0.1 Redirect dplane001 10.0.0.2 Redirect dplane002 10.0.0.3 Redirect dplane003 dplane003 lo: 10.255.0.3 tun: 10.99.0.1 dplane004 lo: 10.255.0.4 tun: 10.99.0.1 eth0: 10.0.0.2 eth0: 10.0.0.3 FabricNAT Global NAT Configuration Table ID Client Address NAT Public-IP:Port-Range Micro-active 1 10.0.0.1 142.8.0.1:1000-1031 dplane001 2 10.0.0.2 142.8.0.1:1032-1063 dplane002 3 10.0.0.3 142.8.0.1:1064-1095 dplane003 eth0: 10.0.0.1 eth0: 10.0.0.2 eth0: 10.0.0.3 dplane001 dplane002 lo: 10.255.0.2 tun: 10.99.0.1 dplane003 lo: 10.255.0.3 tun: 10.99.0.1 dplane004 lo: 10.255.0.4 tun: 10.99.0.1 External service 119.0.0.1:80 eth0: 10.0.0.1 eth0: 10.0.0.2 eth0: 10.0.0.3 1/N
 Blast Radius dplane003 dplane004 dplane002
42. N+1 Active/Active…? 1/N Blast Radius …? eth0: 10.0.0.1 External service 119.0.0.1:80 eth0: 10.0.0.2 eth0: 10.0.0.3 eth0: 10.0.0.1 eth0: 10.0.0.2 eth0: 10.0.0.3 dplane001 dplane002 dplane003 dplane004 External service 119.0.0.1:80 dplane005 dplane006 dplane007 dplane008
43. N+1 Active/Active…? 1/N Blast Radius …? eth0: 10.0.0.1 External service 119.0.0.1:80 eth0: 10.0.0.2 eth0: 10.0.0.3 eth0: 10.0.0.1 eth0: 10.0.0.2 eth0: 10.0.0.3 dplane001 dplane002 dplane003 External service 119.0.0.1:80 dplane005 dplane006 dplane007 dplane008 1/8 Blast Radius == 1/N (N=8) dplane004
44. Re-Recap FabricNAT • 2 Step Routing • 1st step: Robust with Many Stateless Active path • 2nd step: Not Robust but Blast Radius is 1/N
 • Gracefull v.s. Small Blast Radius • “Small Blast Radius and Hyperscale” >>> "Big Blast Radius and Graceful” • Getting Gracefully is Additional-Step
45. SDN Mechanism For Configuration
46. How we configure …? eth0: 10.0.0.1 External service 119.0.0.1:80 Match Src Action 10.0.0.1 NAT 142.8.0.1:1000-1031 10.0.0.2 Redirect dplane002 10.0.0.3 Redirect dplane003 Match Src Action 10.0.0.1 Redirect dplane001 10.0.0.2 NAT 142.8.0.1:1032:1063 10.0.0.3 Redirect dplane003 dplane001 lo: 10.255.0.1 tun: 10.99.0.1 dplane002 lo: 10.255.0.2 tun: 10.99.0.1 Match Src Action 10.0.0.1 Redirect dplane001 10.0.0.2 Redirect dplane002 10.0.0.3 NAT 142.8.0.1:1064-1095 Match Src Action 10.0.0.1 Redirect dplane001 10.0.0.2 Redirect dplane002 10.0.0.3 Redirect dplane003 dplane003 lo: 10.255.0.3 tun: 10.99.0.1 dplane004 lo: 10.255.0.4 tun: 10.99.0.1 eth0: 10.0.0.2 eth0: 10.0.0.3 FabricNAT Global NAT Configuration Table ID Client Address NAT Public-IP:Port-Range Worker 1 10.0.0.1 142.8.0.1:1000-1031 dplane001 2 10.0.0.2 142.8.0.1:1032-1063 dplane002 3 10.0.0.3 142.8.0.1:1064-1095 dplane003 eth0: 10.0.0.1 dplane003 dplane004 dplane001 dplane002
47. How we configure …? eth0: 10.0.0.1 External service 119.0.0.1:80 eth0: 10.0.0.2 eth0: 10.0.0.3 FabricNAT Global NAT Configuration Table ID Client Address NAT Public-IP:Port-Range Micro-active 1 10.0.0.1 142.8.0.1:1000-1031 dplane001 2 10.0.0.2 142.8.0.1:1032-1063 dplane002 3 10.0.0.3 142.8.0.1:1064-1095 dplane003 eth0: 10.0.0.1 Match Src Action 10.0.0.1 NAT 142.8.0.1:1000-1031 10.0.0.2 Redirect dplane002 10.0.0.3 Redirect dplane003 Match Src Action 10.0.0.1 Redirect dplane001 10.0.0.2 NAT 142.8.0.1:1032:1063 10.0.0.3 Redirect dplane003 dplane001 lo: 10.255.0.1 tun: 10.99.0.1 dplane002 lo: 10.255.0.2 tun: 10.99.0.1 Match Src Action 10.0.0.1 Redirect dplane001 10.0.0.2 Redirect dplane002 10.0.0.3 NAT 142.8.0.1:1064-1095 Match Src Action 10.0.0.1 Redirect dplane001 10.0.0.2 Redirect dplane002 10.0.0.3 Redirect dplane003 dplane003 lo: 10.255.0.3 tun: 10.99.0.1 dplane004 lo: 10.255.0.4 tun: 10.99.0.1 Dataplane Clustering
 with Consul dplane003 dplane004 dplane001 dplane002
48. How we configure …? eth0: 10.0.0.1 External service 119.0.0.1:80 eth0: 10.0.0.2 eth0: 10.0.0.3 Match Src Action 10.0.0.1 NAT 142.8.0.1:1000-1031 10.0.0.2 Redirect dplane002 10.0.0.3 Redirect dplane003 Match Src Action 10.0.0.1 Redirect dplane001 10.0.0.2 NAT 142.8.0.1:1032:1063 10.0.0.3 Redirect dplane003 dplane001 lo: 10.255.0.1 tun: 10.99.0.1 dplane002 lo: 10.255.0.2 tun: 10.99.0.1 Match Src Action 10.0.0.1 Redirect dplane001 10.0.0.2 Redirect dplane002 10.0.0.3 NAT 142.8.0.1:1064-1095 Match Src Action 10.0.0.1 Redirect dplane001 10.0.0.2 Redirect dplane002 10.0.0.3 Redirect dplane003 dplane003 lo: 10.255.0.3 tun: 10.99.0.1 dplane004 lo: 10.255.0.4 tun: 10.99.0.1 Distributed Routing 
 Configuration
 with Etcd FabricNAT Global NAT Configuration Table ID Client Address NAT Public-IP:Port-Range Worker 1 10.0.0.1 142.8.0.1:1000-1031 dplane001 2 10.0.0.2 142.8.0.1:1032-1063 dplane002 3 10.0.0.3 142.8.0.1:1064-1095 dplane003 dplane003 dplane004 dplane001 dplane002
49. eth0: 10.0.0.1 External service 119.0.0.1:80 eth0: 10.0.0.2 eth0: 10.0.0.3 How we configure …? dplane002 lo: 10.255.0.2 tun: 10.99.0.1 Match Src Action 10.0.0.1 Redirect dplane001 10.0.0.2 NAT 142.8.0.1:1032:1063 10.0.0.3 Redirect dplane003 Match Src Action 10.0.0.1 Redirect dplane001 10.0.0.2 Redirect dplane002 10.0.0.3 Redirect dplane003 dplane004 lo: 10.255.0.4 tun: 10.99.0.1 dplane001 lo: 10.255.0.1 tun: 10.99.0.1 dplane003 lo: 10.255.0.3 tun: 10.99.0.1 Match Src Action 10.0.0.1 NAT 142.8.0.1:1000-1031 10.0.0.2 Redirect dplane002 10.0.0.3 Redirect dplane003 Match Src Action 10.0.0.1 Redirect dplane001 10.0.0.2 Redirect dplane002 10.0.0.3 NAT 142.8.0.1:1064-1095 scheduler apiserver Natlet Natlet Natlet Natlet FabricNAT Global NAT Configuration Table ID Client Address NAT Public-IP:Port-Range Worker 1 10.0.0.1 142.8.0.1:1000-1031 dplane001 2 10.0.0.2 142.8.0.1:1032-1063 dplane002 3 10.0.0.3 142.8.0.1:1064-1095 dplane003 dplane002 dplane004 dplane001 dplane003
50. eth0: 10.0.0.1 External service 119.0.0.1:80 eth0: 10.0.0.2 eth0: 10.0.0.3 How we configure …? dplane002 lo: 10.255.0.2 tun: 10.99.0.1 Match Src Action 10.0.0.1 Redirect dplane001 10.0.0.2 NAT 142.8.0.1:1032:1063 10.0.0.3 Redirect dplane003 Match Src Action 10.0.0.1 Redirect dplane001 10.0.0.2 Redirect dplane002 10.0.0.3 Redirect dplane003 dplane004 lo: 10.255.0.4 tun: 10.99.0.1 dplane001 lo: 10.255.0.1 tun: 10.99.0.1 dplane003 lo: 10.255.0.3 tun: 10.99.0.1 Match Src Action 10.0.0.1 NAT 142.8.0.1:1000-1031 10.0.0.2 Redirect dplane002 10.0.0.3 Redirect dplane003 Match Src Action 10.0.0.1 Redirect dplane001 10.0.0.2 Redirect dplane002 10.0.0.3 NAT 142.8.0.1:1064-1095 scheduler apiserver Natlet Natlet Natlet Natlet FabricNAT Global NAT Configuration Table ID Client Address NAT Public-IP:Port-Range Micro-active 1 10.0.0.1 142.8.0.1:1000-1031 dplane001 2 10.0.0.2 142.8.0.1:1032-1063 dplane002 3 10.0.0.3 142.8.0.1:1064-1095 dplane003 dplane002 dplane004 dplane001 dplane003 dplane002 dplane004 dplane001 dplane003 Consul Client Consul Client Consul Client Consul Client Consul Server Natlet Natlet Natlet Natlet Watch
 Consul Watch
 Consul Watch
 Consul Watch
 Consul Gossip
 protocol scheduler Watch
 Consul
51. eth0: 10.0.0.1 External service 119.0.0.1:80 eth0: 10.0.0.2 eth0: 10.0.0.3 How we configure …? dplane002 Match Src Action 10.0.0.1 Redirect dplane001 10.0.0.2 NAT 142.8.0.1:1032:1063 10.0.0.3 Redirect dplane003 Match Src Action 10.0.0.1 Redirect dplane001 10.0.0.2 Redirect dplane002 10.0.0.3 Redirect dplane003 dplane004 dplane001 dplane003 Match Src Action 10.0.0.1 NAT 142.8.0.1:1000-1031 10.0.0.2 Redirect dplane002 10.0.0.3 Redirect dplane003 Match Src Action 10.0.0.1 Redirect dplane001 10.0.0.2 Redirect dplane002 10.0.0.3 NAT 142.8.0.1:1064-1095 scheduler apiserver Natlet Natlet Natlet Natlet FabricNAT Global NAT Configuration Table ID Client Address NAT Public-IP:Port-Range Micro-active 1 10.0.0.1 142.8.0.1:1000-1031 dplane001 2 10.0.0.2 142.8.0.1:1032-1063 dplane002 3 10.0.0.3 142.8.0.1:1064-1095 dplane003 Etcd-Watch Etcd-Watch Etcd-Watch Etcd-Watch Natlet Natlet Natlet Natlet apiserver Etcd-Put scheduler Etcd-Watch
52. eth0: 10.0.0.1 External service 119.0.0.1:80 eth0: 10.0.0.2 eth0: 10.0.0.3 How we configure …? dplane002 lo: 10.255.0.2 tun: 10.99.0.1 Match Src Action 10.0.0.1 Redirect dplane001 10.0.0.2 NAT 142.8.0.1:1032:1063 10.0.0.3 Redirect dplane003 Match Src Action 10.0.0.1 Redirect dplane001 10.0.0.2 Redirect dplane002 10.0.0.3 Redirect dplane003 dplane004 lo: 10.255.0.4 tun: 10.99.0.1 dplane001 lo: 10.255.0.1 tun: 10.99.0.1 dplane003 lo: 10.255.0.3 tun: 10.99.0.1 Match Src Action 10.0.0.1 NAT 142.8.0.1:1000-1031 10.0.0.2 Redirect dplane002 10.0.0.3 Redirect dplane003 Match Src Action 10.0.0.1 Redirect dplane001 10.0.0.2 Redirect dplane002 10.0.0.3 NAT 142.8.0.1:1064-1095 scheduler apiserver Natlet eth0: 10.0.0.4 Natlet Natlet Natlet 3. Etcd 
 DATA PUT 4. Notice
 New Resource 2. Port-allocation 1096-1127 FabricNAT Global NAT Configuration Table ID Client Address NAT Public-IP:Port-Range Worker 1 10.0.0.1 142.8.0.1:1000-1031 dplane001 2 10.0.0.2 142.8.0.1:1032-1063 dplane002 3 10.0.0.3 142.8.0.1:1064-1095 dplane003 4 10.0.0.4 142.8.0.1:1096-1127 NULL 1. NAT Create
 “client“: "10.0.04” “vip”: “142.2.0.1” dplane002 dplane004 dplane001 dplane003 NAT Configuration
 Client 10.0.0.4
 VIP 142.8.0.1
53. eth0: 10.0.0.1 External service 119.0.0.1:80 eth0: 10.0.0.2 eth0: 10.0.0.3 How we configure …? dplane002 Match Src Action 10.0.0.1 Redirect dplane001 10.0.0.2 NAT 142.8.0.1:1032:1063 10.0.0.3 Redirect dplane003 Match Src Action 10.0.0.1 Redirect dplane001 10.0.0.2 Redirect dplane002 10.0.0.3 Redirect dplane003 dplane004 dplane001 dplane003 Match Src Action 10.0.0.1 NAT 142.8.0.1:1000-1031 10.0.0.2 Redirect dplane002 10.0.0.3 Redirect dplane003 Match Src Action 10.0.0.1 Redirect dplane001 10.0.0.2 Redirect dplane002 10.0.0.3 NAT 142.8.0.1:1064-1095 apiserver Natlet eth0: 10.0.0.4 Natlet Natlet Natlet FabricNAT Global NAT Configuration Table ID Client Address NAT Public-IP:Port-Range Micro-active 1 10.0.0.1 142.8.0.1:1000-1031 dplane001 2 10.0.0.2 142.8.0.1:1032-1063 dplane002 3 10.0.0.3 142.8.0.1:1064-1095 dplane003 4 10.0.0.4 142.8.0.1:1096-1127 NULL Get Healthy Dplane nodes scheduler
54. eth0: 10.0.0.1 External service 119.0.0.1:80 eth0: 10.0.0.2 eth0: 10.0.0.3 How we configure …? dplane002 lo: 10.255.0.2 tun: 10.99.0.1 Match Src Action 10.0.0.1 Redirect dplane001 10.0.0.2 NAT 142.8.0.1:1032:1063 10.0.0.3 Redirect dplane003 Match Src Action 10.0.0.1 Redirect dplane001 10.0.0.2 Redirect dplane002 10.0.0.3 Redirect dplane003 dplane004 lo: 10.255.0.4 tun: 10.99.0.1 dplane001 lo: 10.255.0.1 tun: 10.99.0.1 dplane003 lo: 10.255.0.3 tun: 10.99.0.1 Match Src Action 10.0.0.1 NAT 142.8.0.1:1000-1031 10.0.0.2 Redirect dplane002 10.0.0.3 Redirect dplane003 Match Src Action 10.0.0.1 Redirect dplane001 10.0.0.2 Redirect dplane002 10.0.0.3 NAT 142.8.0.1:1064-1095 scheduler apiserver Natlet eth0: 10.0.0.4 Natlet Natlet Natlet 1. Get Healthy Dplane nodes 2. Cluster Info 4. Re-write
 Scheduled
 Resource 3. Worker
 election
 dplane004 dplane002 dplane004 dplane001 dplane003 FabricNAT Global NAT Configuration Table ID Client Address NAT Public-IP:Port-Range Worker 1 10.0.0.1 142.8.0.1:1000-1031 dplane001 2 10.0.0.2 142.8.0.1:1032-1063 dplane002 3 10.0.0.3 142.8.0.1:1064-1095 dplane003 4 10.0.0.4 142.8.0.1:1096-1127 dplane004
55. eth0: 10.0.0.1 External service 119.0.0.1:80 eth0: 10.0.0.2 eth0: 10.0.0.3 How we configure …? dplane002 Match Src Action 10.0.0.1 Redirect dplane001 10.0.0.2 NAT 142.8.0.1:1032:1063 10.0.0.3 Redirect dplane003 Match Src Action 10.0.0.1 Redirect dplane001 10.0.0.2 Redirect dplane002 10.0.0.3 Redirect dplane003 dplane004 dplane001 dplane003 Match Src Action 10.0.0.1 NAT 142.8.0.1:1000-1031 10.0.0.2 Redirect dplane002 10.0.0.3 Redirect dplane003 Match Src Action 10.0.0.1 Redirect dplane001 10.0.0.2 Redirect dplane002 10.0.0.3 NAT 142.8.0.1:1064-1095 scheduler apiserver eth0: 10.0.0.4 Natlet Natlet Natlet Natlet Client-IP=10.0.0.1
 NAT=142.8.0.1:1096-1127
 Worker=dplane4 FabricNAT Global NAT Configuration Table ID Client Address NAT Public-IP:Port-Range Worker 1 10.0.0.1 142.8.0.1:1000-1031 dplane001 2 10.0.0.2 142.8.0.1:1032-1063 dplane002 3 10.0.0.3 142.8.0.1:1064-1095 dplane003 4 10.0.0.4 142.8.0.1:1096-1127 dplane004 Etcd
 Watch Etcd
 Watch Etcd
 Watch Etcd
 Watch
56. eth0: 10.0.0.1 External service 119.0.0.1:80 eth0: 10.0.0.2 eth0: 10.0.0.3 How we configure …? dplane002 dplane004 dplane001 dplane003 scheduler apiserver eth0: 10.0.0.4 Natlet Natlet Natlet Natlet Match Src Action 10.0.0.1 Redirect dplane001 10.0.0.2 NAT 142.8.0.1:1032:1063 10.0.0.3 Redirect dplane003 10.0.0.4 Redirect dplane004 Match Src Action 10.0.0.1 NAT 142.8.0.1:1000-1031 10.0.0.2 Redirect dplane002 10.0.0.3 Redirect dplane003 10.0.0.4 Redirect dplane004 Match Src Action 10.0.0.1 Redirect dplane001 10.0.0.2 Redirect dplane002 10.0.0.3 NAT 142.8.0.1:1064-1095 10.0.0.4 Redirect dplane004 NON-WORKER FabricNAT Global NAT Configuration Table ID Client Address NAT Public-IP:Port-Range Worker 1 10.0.0.1 142.8.0.1:1000-1031 dplane001 2 10.0.0.2 142.8.0.1:1032-1063 dplane002 3 10.0.0.3 142.8.0.1:1064-1095 dplane003 4 10.0.0.4 142.8.0.1:1096-1127 dplane004 Client-IP=10.0.0.1
 NAT=142.8.0.1:1096-1127
 Worker=dplane4 WORKER Match Src Action 10.0.0.1 Redirect dplane001 10.0.0.2 Redirect dplane002 10.0.0.3 Redirect dplane003 10.0.0.4 NAT 142.8.0.1:1096-1127
57. Recap FabricNAT SDN • SDN Design Principle • Without RPC • With • Declarative Configuration • Resource-Watch • Reconciliation • Use CloudNative Parts • Use Consul for dataplane clustering • Use Etcd as Watchable Robust KVS Respecting K8s
58. eth0: 10.0.0.1 External service 119.0.0.1:80 eth0: 10.0.0.2 eth0: 10.0.0.3 How we configure …? scheduler apiserver eth0: 10.0.0.4 FabricNAT Global NAT Configuration Table ID Client Address NAT Public-IP:Port-Range Worker 1 10.0.0.1 142.8.0.1:1000-1031 dplane001 2 10.0.0.2 142.8.0.1:1032-1063 dplane002 3 10.0.0.3 142.8.0.1:1064-1095 dplane003 dplane002 lo: 10.255.0.2 tun: 10.99.0.1 Match Src Action 10.0.0.1 Redirect dplane001 10.0.0.2 NAT 142.8.0.1:1032:1063 10.0.0.3 Redirect dplane003 Match Src Action 10.0.0.1 Redirect dplane001 10.0.0.2 Redirect dplane002 10.0.0.3 Redirect dplane003 dplane004 lo: 10.255.0.4 tun: 10.99.0.1 dplane001 lo: 10.255.0.1 tun: 10.99.0.1 dplane003 lo: 10.255.0.3 tun: 10.99.0.1 Match Src Action 10.0.0.1 NAT 142.8.0.1:1000-1031 10.0.0.2 Redirect dplane002 10.0.0.3 Redirect dplane003 Match Src Action 10.0.0.1 Redirect dplane001 10.0.0.2 Redirect dplane002 10.0.0.3 NAT 142.8.0.1:1064-1095 Natlet Natlet Natlet Natlet dplane002 dplane004 dplane001 dplane003
59. eth0: 10.0.0.1 External service 119.0.0.1:80 eth0: 10.0.0.2 eth0: 10.0.0.3 How we configure …? scheduler apiserver eth0: 10.0.0.4 FabricNAT Global NAT Configuration Table ID Client Address NAT Public-IP:Port-Range Worker 1 10.0.0.1 142.8.0.1:1000-1031 dplane001 2 10.0.0.2 142.8.0.1:1032-1063 dplane002 3 10.0.0.3 142.8.0.1:1064-1095 dplane003 dplane002 lo: 10.255.0.2 tun: 10.99.0.1 Match Src Action 10.0.0.1 Redirect dplane001 10.0.0.2 NAT 142.8.0.1:1032:1063 10.0.0.3 Redirect dplane003 Match Src Action 10.0.0.1 Redirect dplane001 10.0.0.2 Redirect dplane002 10.0.0.3 Redirect dplane003 dplane004 lo: 10.255.0.4 tun: 10.99.0.1 dplane001 lo: 10.255.0.1 tun: 10.99.0.1 dplane003 lo: 10.255.0.3 tun: 10.99.0.1 Match Src Action 10.0.0.1 NAT 142.8.0.1:1000-1031 10.0.0.2 Redirect dplane002 10.0.0.3 Redirect dplane003 Match Src Action 10.0.0.1 Redirect dplane001 10.0.0.2 Redirect dplane002 10.0.0.3 NAT 142.8.0.1:1064-1095 Natlet Natlet Natlet Natlet dplane002 dplane004 dplane001 dplane003
60. eth0: 10.0.0.1 External service 119.0.0.1:80 eth0: 10.0.0.2 eth0: 10.0.0.3 How we configure …? scheduler apiserver eth0: 10.0.0.4 dplane002 lo: 10.255.0.2 tun: 10.99.0.1 Match Src Action 10.0.0.1 Redirect dplane001 10.0.0.2 NAT 142.8.0.1:1032:1063 10.0.0.3 Redirect dplane003 Match Src Action 10.0.0.1 Redirect dplane001 10.0.0.2 Redirect dplane002 10.0.0.3 Redirect dplane003 dplane004 lo: 10.255.0.4 tun: 10.99.0.1 dplane001 lo: 10.255.0.1 tun: 10.99.0.1 dplane003 lo: 10.255.0.3 tun: 10.99.0.1 Match Src Action 10.0.0.1 NAT 142.8.0.1:1000-1031 10.0.0.2 Redirect dplane002 10.0.0.3 Redirect dplane003 Match Src Action 10.0.0.1 Redirect dplane001 10.0.0.2 Redirect dplane002 10.0.0.3 NAT 142.8.0.1:1064-1095 Natlet Natlet Natlet Natlet ! ! ! ! 1. dplane001 
 was down 1. dplane001 
 was down 1. dplane001 
 was down 1. dplane001 
 was down dplane002 dplane004 dplane001 dplane003 scheduler ! 2. dplane001 
 was down FabricNAT Global NAT Configuration Table ID Client Address NAT Public-IP:Port-Range Micro-active 1 10.0.0.1 142.8.0.1:1000-1031 dplane001 2 10.0.0.2 142.8.0.1:1032-1063 dplane002 3 10.0.0.3 142.8.0.1:1064-1095 dplane003
61. eth0: 10.0.0.1 External service 119.0.0.1:80 eth0: 10.0.0.2 eth0: 10.0.0.3 How we configure …? scheduler apiserver eth0: 10.0.0.4 FabricNAT Global NAT Configuration Table ID Client Address NAT Public-IP:Port-Range Micro-active 1 10.0.0.1 142.8.0.1:1000-1031 dplane001 2 10.0.0.2 142.8.0.1:1032-1063 dplane002 3 10.0.0.3 142.8.0.1:1064-1095 dplane003 4 10.0.0.4 142.8.0.1:1096-1127 NULL dplane002 lo: 10.255.0.2 tun: 10.99.0.1 dplane004 lo: 10.255.0.4 tun: 10.99.0.1 dplane003 lo: 10.255.0.3 tun: 10.99.0.1 Match Src Action 10.0.0.1 NAT 142.8.0.1:1000-1031 10.0.0.2 Redirect dplane002 10.0.0.3 Redirect dplane003 Natlet Natlet Natlet Natlet scheduler FabricNAT Global NAT Configuration Table-1 ID Client Address NAT Public-IP:Port-Range Worker 1 10.0.0.1 142.8.0.1:1000-1031 dplane004 2 10.0.0.2 142.8.0.1:1032-1063 dplane002 3 10.0.0.3 142.8.0.1:1064-1095 dplane003 dplane002 dplane004 dplane001 dplane003 ! ! ! Match Src Action 10.0.0.1 Redirect dplane004 10.0.0.2 NAT 142.8.0.1:1032:1063 10.0.0.3 Redirect dplane003 Match Src Action 10.0.0.1 NAT 142.8.0.1:1000-1031 10.0.0.2 Redirect dplane002 10.0.0.3 Redirect dplane003 Match Src Action 10.0.0.1 Redirect dplane004 10.0.0.2 Redirect dplane002 10.0.0.3 NAT 142.8.0.1:1064-1095
62. Recap FabricNAT SDN • SDN Design Principle • Without RPC • With • Declarative Configuration • Resource-Watch • Reconciliation • Use CloudNative Parts • Use Consul for dataplane clustering • Use Etcd as Watchable Robust KVS Offload 
 Complex 
 Mechanism
63. A Software’s Stability • “Proprietary Software” v.s. “Major OSS” • Which looks be stable …?
 • Compare the Upgrade Period • Natlet: ?? • Etcd: ??
64. A Software’s Stability • “Proprietary Software” v.s. “Major OSS” • Which looks be stable …?
 • Compare the Upgrade Period • Natlet: upgrade once/day • Etcd: upgrade once/month~
65. Software will be Broken • Development Difficulty • non broken software → Really Really Difficult • reboot-able software → More Easy and Realistic
 • Basic Principle • Casual Maintenance • Frequency Upgrade
66. eth0: 10.0.0.1 External service 119.0.0.1:80 eth0: 10.0.0.2 eth0: 10.0.0.3 How we configure …? dplane002 Match Src Action 10.0.0.1 Redirect dplane001 10.0.0.2 NAT 142.8.0.1:1032:1063 10.0.0.3 Redirect dplane003 Match Src Action 10.0.0.1 Redirect dplane001 10.0.0.2 Redirect dplane002 10.0.0.3 Redirect dplane003 dplane004 dplane001 dplane003 Match Src Action 10.0.0.1 NAT 142.8.0.1:1000-1031 10.0.0.2 Redirect dplane002 10.0.0.3 Redirect dplane003 Match Src Action 10.0.0.1 Redirect dplane001 10.0.0.2 Redirect dplane002 10.0.0.3 NAT 142.8.0.1:1064-1095 scheduler apiserver eth0: 10.0.0.4 Natlet Natlet Natlet Natlet RPC FabricNAT Global NAT Configuration Table ID Client Address NAT Public-IP:Port-Range Worker 1 10.0.0.1 142.8.0.1:1000-1031 dplane001 2 10.0.0.2 142.8.0.1:1032-1063 dplane002 3 10.0.0.3 142.8.0.1:1064-1095 dplane003 4 10.0.0.4 142.8.0.1:1096-1127 dplane004 RPC RPC RPC Restarting
67. eth0: 10.0.0.1 External service 119.0.0.1:80 eth0: 10.0.0.2 eth0: 10.0.0.3 How we configure …? dplane002 Match Src Action 10.0.0.1 Redirect dplane001 10.0.0.2 NAT 142.8.0.1:1032:1063 10.0.0.3 Redirect dplane003 Match Src Action 10.0.0.1 Redirect dplane001 10.0.0.2 Redirect dplane002 10.0.0.3 Redirect dplane003 dplane004 dplane001 dplane003 Match Src Action 10.0.0.1 NAT 142.8.0.1:1000-1031 10.0.0.2 Redirect dplane002 10.0.0.3 Redirect dplane003 Match Src Action 10.0.0.1 Redirect dplane001 10.0.0.2 Redirect dplane002 10.0.0.3 NAT 142.8.0.1:1064-1095 scheduler apiserver eth0: 10.0.0.4 Natlet Natlet Natlet Natlet Watch FabricNAT Global NAT Configuration Table ID Client Address NAT Public-IP:Port-Range Worker 1 10.0.0.1 142.8.0.1:1000-1031 dplane001 2 10.0.0.2 142.8.0.1:1032-1063 dplane002 3 10.0.0.3 142.8.0.1:1064-1095 dplane003 4 10.0.0.4 142.8.0.1:1096-1127 dplane004 Watch Watch Watch Restarting Restart
 Watch &
 Reconcile
68. Software will be Broken • Development Difficulty • non broken software → Really Really Difficult • reboot-able software → More Easy and Realistic
 • Basic Principle • Casual Maintenance • Frequency Upgrade Enabled by Declarative Model and Reconciliation loop
69. Operation / Monitoring
70. Deploy/Upgrade/Restart Components with Ansible Internet Gateway Service NAT 142.8.0.1 NAT 142.8.0.2 dplane001 dplane002 dplane003 dplane004 Natlet Natlet Natlet Natlet scheduler apiserver
71. Monitoring :: Basic Metrics Internet Gateway Service NAT 142.8.0.1 NAT 142.8.0.2 Natlet dplane001 dplane002 dplane003 dplane004 Natlet Natlet Natlet scheduler apiserver k8s consul_sd_config kube_
 state_
 metrics
72. Monitoring :: Dynamic Robot Internet Gateway Service apiserver 1. OpenStack
 Token Issue 2. NAT Create 3. Replay Connection Information.
 Source IP
 Source Port etc.. 4. Check Connection
 Statement 5. NAT Delete NAT 142.8.0.1 NAT 142.8.0.2
73. Monitoring :: Static Robot Internet Gateway Service NAT 142.8.0.1 NAT 142.8.0.2 Use Blackbox Exporter consul_sd_config
74. Next Step / Future Work • NAME based ACL • Automated Robot Operation • Private Network Gateway
75. NAME based Access Control Internet Gateway Service NAT 142.8.0.1 NAT 142.8.0.2 Allow-list ACL Allow-list ACL Allow-list Dest Proto api.external.linecorp.com https, http ftp.jaist.ac.jp ftp, https github.com ssh, https DNS
76. Automated Robot Operation dplane002 Match Src Action 10.0.0.1 Redirect dplane001 10.0.0.2 NAT 142.8.0.1:1032:1063 10.0.0.3 Redirect dplane003 Match Src Action 10.0.0.1 Redirect dplane001 10.0.0.2 Redirect dplane002 10.0.0.3 Redirect dplane003 dplane004 dplane001 dplane003 Match Src Action 10.0.0.1 NAT 142.8.0.1:1000-1031 10.0.0.2 Redirect dplane002 10.0.0.3 Redirect dplane003 Match Src Action 10.0.0.1 Redirect dplane001 10.0.0.2 Redirect dplane002 10.0.0.3 NAT 142.8.0.1:1064-1095 Natlet Natlet Natlet Natlet scheduler apiserver Current Ops
77. Automated Robot Operation dplane002 Match Src Action 10.0.0.1 Redirect dplane001 10.0.0.2 NAT 142.8.0.1:1032:1063 10.0.0.3 Redirect dplane003 Match Src Action 10.0.0.1 Redirect dplane001 10.0.0.2 Redirect dplane002 10.0.0.3 Redirect dplane003 dplane004 dplane001 dplane003 Match Src Action 10.0.0.1 NAT 142.8.0.1:1000-1031 10.0.0.2 Redirect dplane002 10.0.0.3 Redirect dplane003 Match Src Action 10.0.0.1 Redirect dplane001 10.0.0.2 Redirect dplane002 10.0.0.3 NAT 142.8.0.1:1064-1095 Natlet Natlet Natlet Natlet scheduler apiserver Automated
 Operation 
 Manifest Monitor- ing
78. Private Zone Private Zone A NW-A1 NW-A2 NW-GW Private Zone B NW-B1 NW-B2 NW-GW
79. Summary • Newly Provided “Internet Gateway” • We build the new routing mechanism called “FabricNAT” for backend • Distributed NAT mech achieving “Small Blast Radius” and “Hyperscale”
 • SDN Design using CloudNative blocks • Consul for Easy/Massive clustering • Etcd for Declarative Configuration • Stateless software component can be restart casually
 • Many Network Challenge on our Private Cloud :)
80. Thank you