Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Hyperscale distributed NAT system and software engineering

Hyperscale distributed NAT system and software engineering

Hiroki Shirokura
LINE ITSC Verda Network Development Team Senior Software Engineer
https://linedevday.linecorp.com/2020/ja/sessions/2076
https://linedevday.linecorp.com/2020/en/sessions/2076

Eebedc2ee7ff95ffb9d9102c6d4a065c?s=128

LINE DevDay 2020

November 25, 2020
Tweet

Transcript

  1. None
  2. Speaker : Hiroki Shirokura Senior Software Engineer at Verda In

    charge of Design/Develop-ing Network Service on Private Cloud Love SDN, Routing, Distributed System OSS contributor around Networking
  3. Network Service Architecture 
 Optimized for LINE’s Private Cloud

  4. Verda is a Private Cloud for LINE is based on

    OpenStack. since 2016~ FaaS PaaS IaaS NAT LB Bare
 metal Hypervisor 2,000+ Virtual Machine 55,000+ Physical Machine 20,000+ Verda Only +10,000/Half
  5. Simple Networking @ Verda eBGP eBGP APPROACH
 Full L3 (No-L2,

    BGP Fabric) Protocol Reduction (No-Overlay) No Overlay Network Service We Get Simple & High Capacity
  6. Simple Networking @ Verda eBGP eBGP APPROACH
 Full L3 (No-L2,

    BGP Fabric) Protocol Reduction (No-Overlay) No Overlay Network Service We Get Simple & High Capacity Issues No Managed Network policy Public-IP High Consumption We need gateway solution !
  7. Let’s Provide Internet Gateway Service Internet Gateway Service NAT 142.8.0.1

    NAT 142.8.0.2
  8. Let’s Provide Internet Gateway Service Internet Gateway Service NAT 142.8.0.1

    NAT 142.8.0.2 Fair Balanced Flexible Config Extensibility Hyperscale 4 Requirements Service Project A
 Uses 142.8.0.1 Service Project B Uses 142.8.0.2 %JCNNGPIG Allow-list ACL Allow-list ACL Allow-list Dest Proto api.external.linecorp.com https, http ftp.jaist.ac.jp ftp, https github.com ssh, https
  9. Hyperscale Challenge…? Back to the Traditional Way Hardware
 Appliance Standby

    Hardware
 Appliance ! Active
  10. Hyperscale Challenge…? Back to the Traditional Way Hardware
 Appliance Hardware


    Appliance Dead New Active Graceful
  11. Hyperscale Challenge…? Back to the Traditional Way Hardware
 Appliance Hardware


    Appliance Dead New Active Big Blast Radius… NG#1 100% Blast Radius
  12. Hyperscale Challenge…? Back to the Traditional Way Standby Active 50%

    Blast Radius 50% Blast Radius Standby Active 2NxM Cluster Operation Cost NG#2
  13. Hyperscale Challenge…? Back to the Traditional Way Standby Active Standby

    Active L2 Network L2 Network Miss-match with Full L3 Network NG#3
  14. Hyperscale Challenge…? Back to the Traditional Way Standby Active Standby

    Active L2 Network L2 Network Miss-match with Full L3 Network NG#3 Mechanism needs (1) N+1 Active/Active Single Cluster (2) Full L3 Simple Network Aware Let’s Construct the New One
  15. FabricNAT: Hyperscale Distributed NAT System Match Src Action 10.0.0.1 NAT

    142.8.0.1:1000-1031 10.0.0.2 Redirect dplane002 10.0.0.3 Redirect dplane003 Match Src Action 10.0.0.1 Redirect dplane001 10.0.0.2 NAT 142.8.0.1:1032:1063 10.0.0.3 Redirect dplane003 Match Src Action 10.0.0.1 Redirect dplane001 10.0.0.2 Redirect dplane002 10.0.0.3 NAT 142.8.0.1:1064-1095 Match Src Action 10.0.0.1 Redirect dplane001 10.0.0.2 Redirect dplane002 10.0.0.3 Redirect dplane003 NAT 142.8.0.1 dplane001 dplane002 dplane003 dplane004
  16. FabricNAT: Hyperscale Distributed NAT System NAT 142.8.0.1 FabricNAT feature is

    N+1 Active/Active in Single Cluster Full L3 Network awareness Match Src Action 10.0.0.1 NAT 142.8.0.1:1000-1031 10.0.0.2 Redirect dplane002 10.0.0.3 Redirect dplane003 Match Src Action 10.0.0.1 Redirect dplane001 10.0.0.2 NAT 142.8.0.1:1032:1063 10.0.0.3 Redirect dplane003 Match Src Action 10.0.0.1 Redirect dplane001 10.0.0.2 Redirect dplane002 10.0.0.3 NAT 142.8.0.1:1064-1095 Match Src Action 10.0.0.1 Redirect dplane001 10.0.0.2 Redirect dplane002 10.0.0.3 Redirect dplane003 dplane001 dplane002 dplane003 dplane004 FabricNAT is structured with… BGP Anycast Policy Based Routing IPIP tunneling
  17. FabricNAT: Hyperscale Distributed NAT System Match Src Action 10.0.0.1 NAT

    142.8.0.1:1000-1031 10.0.0.2 Redirect dplane002 10.0.0.3 Redirect dplane003 Match Src Action 10.0.0.1 Redirect dplane001 10.0.0.2 NAT 142.8.0.1:1032:1063 10.0.0.3 Redirect dplane003 Match Src Action 10.0.0.1 Redirect dplane001 10.0.0.2 Redirect dplane002 10.0.0.3 NAT 142.8.0.1:1064-1095 Match Src Action 10.0.0.1 Redirect dplane001 10.0.0.2 Redirect dplane002 10.0.0.3 Redirect dplane003 dplane001 dplane002 dplane003 dplane004 eth0: 10.0.0.1 External service 119.0.0.1:80
  18. FabricNAT: Hyperscale Distributed NAT System eth0: 10.0.0.1 Match Src Action

    10.0.0.1 NAT 142.8.0.1:1000-1031 10.0.0.2 Redirect dplane002 10.0.0.3 Redirect dplane003 Match Src Action 10.0.0.1 Redirect dplane001 10.0.0.2 NAT 142.8.0.1:1032:1063 10.0.0.3 Redirect dplane003 Match Src Action 10.0.0.1 Redirect dplane001 10.0.0.2 Redirect dplane002 10.0.0.3 NAT 142.8.0.1:1064-1095 Match Src Action 10.0.0.1 Redirect dplane001 10.0.0.2 Redirect dplane002 10.0.0.3 Redirect dplane003 dplane001 dplane002 dplane003 dplane004 eth0: 10.0.0.1 FabricNAT Global NAT Configuration Table ID Client Address NAT Public-IP:Port-Range Worker 1 10.0.0.1 142.8.0.1:1000-1031 dplane001 2 10.0.0.2 142.8.0.1:1032-1063 dplane002 3 10.0.0.3 142.8.0.1:1064-1095 dplane003 External service 119.0.0.1:80 dplane003 dplane004 dplane001 dplane002 Match Src Action 10.0.0.1 Redirect dplane001 10.0.0.2 Redirect dplane002 10.0.0.3 NAT 142.8.0.1:1064-1095 Match Src Action 10.0.0.1 Redirect dplane001 10.0.0.2 Redirect dplane002 10.0.0.3 Redirect dplane003 Match Src Action 10.0.0.1 NAT 142.8.0.1:1000-1031 10.0.0.2 Redirect dplane002 10.0.0.3 Redirect dplane003 Match Src Action 10.0.0.1 Redirect dplane001 10.0.0.2 NAT 142.8.0.1:1032:1063 10.0.0.3 Redirect dplane003
  19. FabricNAT: Hyperscale Distributed NAT System eth0: 10.0.0.1 Match Src Action

    10.0.0.1 NAT 142.8.0.1:1000-1031 10.0.0.2 Redirect dplane002 10.0.0.3 Redirect dplane003 Match Src Action 10.0.0.1 Redirect dplane001 10.0.0.2 NAT 142.8.0.1:1032:1063 10.0.0.3 Redirect dplane003 Match Src Action 10.0.0.1 Redirect dplane001 10.0.0.2 Redirect dplane002 10.0.0.3 NAT 142.8.0.1:1064-1095 Match Src Action 10.0.0.1 Redirect dplane001 10.0.0.2 Redirect dplane002 10.0.0.3 Redirect dplane003 dplane001 dplane002 dplane003 dplane004 Match Src Action 10.0.0.1 NAT 142.8.0.1:1000-1031 10.0.0.2 Redirect dplane002 10.0.0.3 Redirect dplane003 Match Src Action 10.0.0.1 Redirect dplane001 10.0.0.2 NAT 142.8.0.1:1032:1063 10.0.0.3 Redirect dplane003 Match Src Action 10.0.0.1 Redirect dplane001 10.0.0.2 Redirect dplane002 10.0.0.3 NAT 142.8.0.1:1064-1095 Match Src Action 10.0.0.1 Redirect dplane001 10.0.0.2 Redirect dplane002 10.0.0.3 Redirect dplane003 dplane001 lo: 10.255.0.1 tun: 10.99.0.1 dplane002 lo: 10.255.0.2 tun: 10.99.0.1 dplane003 lo: 10.255.0.3 tun: 10.99.0.1 dplane004 lo: 10.255.0.4 tun: 10.99.0.1 Match Src Action 10.0.0.1 Redirect dplane001 10.0.0.2 Redirect dplane002 10.0.0.3 NAT 142.8.0.1:1064-1095 Match Src Action 10.0.0.1 Redirect dplane001 10.0.0.2 Redirect dplane002 10.0.0.3 Redirect dplane003 Match Src Action 10.0.0.1 NAT 142.8.0.1:1000-1031 10.0.0.2 Redirect dplane002 10.0.0.3 Redirect dplane003 Match Src Action 10.0.0.1 Redirect dplane001 10.0.0.2 NAT 142.8.0.1:1032:1063 10.0.0.3 Redirect dplane003 dplane003 dplane004 dplane001 dplane002 eth0: 10.0.0.1 FabricNAT Global NAT Configuration Table ID Client Address NAT Public-IP:Port-Range Worker 1 10.0.0.1 142.8.0.1:1000-1031 dplane001 2 10.0.0.2 142.8.0.1:1032-1063 dplane002 3 10.0.0.3 142.8.0.1:1064-1095 dplane003 External service 119.0.0.1:80
  20. FabricNAT: Hyperscale Distributed NAT System eth0: 10.0.0.1 Match Src Action

    10.0.0.1 NAT 142.8.0.1:1000-1031 10.0.0.2 Redirect dplane002 10.0.0.3 Redirect dplane003 Match Src Action 10.0.0.1 Redirect dplane001 10.0.0.2 NAT 142.8.0.1:1032:1063 10.0.0.3 Redirect dplane003 Match Src Action 10.0.0.1 Redirect dplane001 10.0.0.2 Redirect dplane002 10.0.0.3 NAT 142.8.0.1:1064-1095 Match Src Action 10.0.0.1 Redirect dplane001 10.0.0.2 Redirect dplane002 10.0.0.3 Redirect dplane003 dplane001 dplane002 dplane003 dplane004 eth0: 10.0.0.1 FabricNAT Global NAT Configuration Table ID Client Address NAT Public-IP:Port-Range Worker 1 10.0.0.1 142.8.0.1:1000-1031 dplane001 2 10.0.0.2 142.8.0.1:1032-1063 dplane002 3 10.0.0.3 142.8.0.1:1064-1095 dplane003 External service 119.0.0.1:80
  21. FabricNAT: Hyperscale Distributed NAT System eth0: 10.0.0.1 External service 119.0.0.1:80

    FabricNAT Global NAT Configuration Table ID Client Address NAT Public-IP:Port-Range Worker 1 10.0.0.1 142.8.0.1:1000-1031 dplane001 2 10.0.0.2 142.8.0.1:1032-1063 dplane002 3 10.0.0.3 142.8.0.1:1064-1095 dplane003 src: 10.0.0.1:2859 dst: 119.0.0.1:80 proto: tcp Match Src Action 10.0.0.1 NAT 142.8.0.1:1000-1031 10.0.0.2 Redirect dplane002 10.0.0.3 Redirect dplane003 Match Src Action 10.0.0.1 Redirect dplane001 10.0.0.2 NAT 142.8.0.1:1032:1063 10.0.0.3 Redirect dplane003 Match Src Action 10.0.0.1 Redirect dplane001 10.0.0.2 Redirect dplane002 10.0.0.3 NAT 142.8.0.1:1064-1095 Match Src Action 10.0.0.1 Redirect dplane001 10.0.0.2 Redirect dplane002 10.0.0.3 Redirect dplane003 dplane001 lo: 10.255.0.1 tun: 10.99.0.1 dplane002 lo: 10.255.0.2 tun: 10.99.0.1 dplane003 lo: 10.255.0.3 tun: 10.99.0.1 dplane004 lo: 10.255.0.4 tun: 10.99.0.1 dplane003 dplane004 dplane001 dplane002
  22. FabricNAT: Hyperscale Distributed NAT System eth0: 10.0.0.1 External service 119.0.0.1:80

    FabricNAT Global NAT Configuration Table ID Client Address NAT Public-IP:Port-Range Worker 1 10.0.0.1 142.8.0.1:1000-1031 dplane001 2 10.0.0.2 142.8.0.1:1032-1063 dplane002 3 10.0.0.3 142.8.0.1:1064-1095 dplane003 dst: 10.99.0.1 proto: ipip src: 10.0.0.1:2859 dst: 119.0.0.1:80 proto: tcp Match Src Action 10.0.0.1 NAT 142.8.0.1:1000-1031 10.0.0.2 Redirect dplane002 10.0.0.3 Redirect dplane003 Match Src Action 10.0.0.1 Redirect dplane001 10.0.0.2 NAT 142.8.0.1:1032:1063 10.0.0.3 Redirect dplane003 Match Src Action 10.0.0.1 Redirect dplane001 10.0.0.2 Redirect dplane002 10.0.0.3 NAT 142.8.0.1:1064-1095 Match Src Action 10.0.0.1 Redirect dplane001 10.0.0.2 Redirect dplane002 10.0.0.3 Redirect dplane003 dplane001 lo: 10.255.0.1 tun: 10.99.0.1 dplane002 lo: 10.255.0.2 tun: 10.99.0.1 dplane003 lo: 10.255.0.3 tun: 10.99.0.1 dplane004 lo: 10.255.0.4 tun: 10.99.0.1 dplane003 dplane004 dplane001 dplane002
  23. Match Src Action 10.0.0.1 Redirect dplane001 10.0.0.2 Redirect dplane002 10.0.0.3

    NAT 142.8.0.1:1064-1095 Match Src Action 10.0.0.1 Redirect dplane001 10.0.0.2 Redirect dplane002 10.0.0.3 Redirect dplane003 dplane003 lo: 10.255.0.3 tun: 10.99.0.1 dplane004 lo: 10.255.0.4 tun: 10.99.0.1 FabricNAT: Hyperscale Distributed NAT System eth0: 10.0.0.1 External service 119.0.0.1:80 FabricNAT Global NAT Configuration Table ID Client Address NAT Public-IP:Port-Range Worker 1 10.0.0.1 142.8.0.1:1000-1031 dplane001 2 10.0.0.2 142.8.0.1:1032-1063 dplane002 3 10.0.0.3 142.8.0.1:1064-1095 dplane003 BGP Advertisement 10.99.0.1/32 BGP Advertisement 10.99.0.1/32 BGP Advertisement 10.99.0.1/32 BGP Advertisement 10.99.0.1/32 Match Src Action 10.0.0.1 NAT 142.8.0.1:1000-1031 10.0.0.2 Redirect dplane002 10.0.0.3 Redirect dplane003 Match Src Action 10.0.0.1 Redirect dplane001 10.0.0.2 NAT 142.8.0.1:1032:1063 10.0.0.3 Redirect dplane003 dplane001 lo: 10.255.0.1 tun: 10.99.0.1 dplane002 lo: 10.255.0.2 tun: 10.99.0.1 dplane003 dplane004 dplane001 dplane002 dst: 10.99.0.1 proto: ipip src: 10.0.0.1:2859 dst: 119.0.0.1:80 proto: tcp
  24. Match Src Action 10.0.0.1 Redirect dplane001 10.0.0.2 Redirect dplane002 10.0.0.3

    NAT 142.8.0.1:1064-1095 Match Src Action 10.0.0.1 Redirect dplane001 10.0.0.2 Redirect dplane002 10.0.0.3 Redirect dplane003 dplane003 lo: 10.255.0.3 tun: 10.99.0.1 dplane004 lo: 10.255.0.4 tun: 10.99.0.1 FabricNAT: Hyperscale Distributed NAT System eth0: 10.0.0.1 External service 119.0.0.1:80 FabricNAT Global NAT Configuration Table ID Client Address NAT Public-IP:Port-Range Worker 1 10.0.0.1 142.8.0.1:1000-1031 dplane001 2 10.0.0.2 142.8.0.1:1032-1063 dplane002 3 10.0.0.3 142.8.0.1:1064-1095 dplane003 Match Src Action 10.0.0.1 NAT 142.8.0.1:1000-1031 10.0.0.2 Redirect dplane002 10.0.0.3 Redirect dplane003 Match Src Action 10.0.0.1 Redirect dplane001 10.0.0.2 NAT 142.8.0.1:1032:1063 10.0.0.3 Redirect dplane003 dplane001 lo: 10.255.0.1 tun: 10.99.0.1 dplane002 lo: 10.255.0.2 tun: 10.99.0.1 dplane003 dplane004 dplane001 dplane002 dst: 10.99.0.1 proto: ipip src: 10.0.0.1:2859 dst: 119.0.0.1:80 proto: tcp
  25. Match Src Action 10.0.0.1 NAT 142.8.0.1:1000-1031 10.0.0.2 Redirect dplane002 10.0.0.3

    Redirect dplane003 Match Src Action 10.0.0.1 Redirect dplane001 10.0.0.2 NAT 142.8.0.1:1032:1063 10.0.0.3 Redirect dplane003 Match Src Action 10.0.0.1 Redirect dplane001 10.0.0.2 Redirect dplane002 10.0.0.3 NAT 142.8.0.1:1064-1095 Match Src Action 10.0.0.1 Redirect dplane001 10.0.0.2 Redirect dplane002 10.0.0.3 Redirect dplane003 dplane001 lo: 10.255.0.1 tun: 10.99.0.1 dplane002 lo: 10.255.0.2 tun: 10.99.0.1 dplane003 lo: 10.255.0.3 tun: 10.99.0.1 dplane004 lo: 10.255.0.4 tun: 10.99.0.1 FabricNAT: Hyperscale Distributed NAT System eth0: 10.0.0.1 External service 119.0.0.1:80 FabricNAT Global NAT Configuration Table ID Client Address NAT Public-IP:Port-Range Worker 1 10.0.0.1 142.8.0.1:1000-1031 dplane001 2 10.0.0.2 142.8.0.1:1032-1063 dplane002 3 10.0.0.3 142.8.0.1:1064-1095 dplane003 dplane003 dplane004 dplane001 dplane002 src: 10.0.0.1:2859 dst: 119.0.0.1:80 proto: tcp
  26. Match Src Action 10.0.0.1 Redirect dplane001 10.0.0.2 Redirect dplane002 10.0.0.3

    NAT 142.8.0.1:1064-1095 Match Src Action 10.0.0.1 Redirect dplane001 10.0.0.2 Redirect dplane002 10.0.0.3 Redirect dplane003 dplane003 lo: 10.255.0.3 tun: 10.99.0.1 dplane004 lo: 10.255.0.4 tun: 10.99.0.1 FabricNAT: Hyperscale Distributed NAT System eth0: 10.0.0.1 External service 119.0.0.1:80 FabricNAT Global NAT Configuration Table ID Client Address NAT Public-IP:Port-Range Worker 1 10.0.0.1 142.8.0.1:1000-1031 dplane001 2 10.0.0.2 142.8.0.1:1032-1063 dplane002 3 10.0.0.3 142.8.0.1:1064-1095 dplane003 Match Src Action 10.0.0.1 NAT 142.8.0.1:1000-1031 10.0.0.2 Redirect dplane002 10.0.0.3 Redirect dplane003 Match Src Action 10.0.0.1 Redirect dplane001 10.0.0.2 NAT 142.8.0.1:1032:1063 10.0.0.3 Redirect dplane003 dplane001 lo: 10.255.0.1 tun: 10.99.0.1 dplane002 lo: 10.255.0.2 tun: 10.99.0.1 dplane003 dplane004 dplane001 dplane002 src: 10.0.0.1:2859 dst: 119.0.0.1:80 proto: tcp
  27. Match Src Action 10.0.0.1 NAT 142.8.0.1:1000-1031 10.0.0.2 Redirect dplane002 10.0.0.3

    Redirect dplane003 Match Src Action 10.0.0.1 Redirect dplane001 10.0.0.2 NAT 142.8.0.1:1032:1063 10.0.0.3 Redirect dplane003 dplane001 lo: 10.255.0.1 tun: 10.99.0.1 dplane002 lo: 10.255.0.2 tun: 10.99.0.1 Match Src Action 10.0.0.1 Redirect dplane001 10.0.0.2 Redirect dplane002 10.0.0.3 NAT 142.8.0.1:1064-1095 Match Src Action 10.0.0.1 Redirect dplane001 10.0.0.2 Redirect dplane002 10.0.0.3 Redirect dplane003 dplane003 lo: 10.255.0.3 tun: 10.99.0.1 dplane004 lo: 10.255.0.4 tun: 10.99.0.1 FabricNAT: Hyperscale Distributed NAT System eth0: 10.0.0.1 External service 119.0.0.1:80 FabricNAT Global NAT Configuration Table ID Client Address NAT Public-IP:Port-Range Worker 1 10.0.0.1 142.8.0.1:1000-1031 dplane001 2 10.0.0.2 142.8.0.1:1032-1063 dplane002 3 10.0.0.3 142.8.0.1:1064-1095 dplane003 dplane003 dplane004 dplane001 dplane002 dst:10.255.0.1 proto: ipip src: 10.0.0.1:2859 dst: 119.0.0.1:80 proto: tcp
  28. Match Src Action 10.0.0.1 NAT 142.8.0.1:1000-1031 10.0.0.2 Redirect dplane002 10.0.0.3

    Redirect dplane003 Match Src Action 10.0.0.1 Redirect dplane001 10.0.0.2 NAT 142.8.0.1:1032:1063 10.0.0.3 Redirect dplane003 dplane001 lo: 10.255.0.1 tun: 10.99.0.1 dplane002 lo: 10.255.0.2 tun: 10.99.0.1 FabricNAT: Hyperscale Distributed NAT System eth0: 10.0.0.1 External service 119.0.0.1:80 FabricNAT Global NAT Configuration Table ID Client Address NAT Public-IP:Port-Range Worker 1 10.0.0.1 142.8.0.1:1000-1031 dplane001 2 10.0.0.2 142.8.0.1:1032-1063 dplane002 3 10.0.0.3 142.8.0.1:1064-1095 dplane003 dst:10.255.0.1 proto: ipip src: 10.0.0.1:2859 dst: 119.0.0.1:80 proto: tcp Match Src Action 10.0.0.1 Redirect dplane001 10.0.0.2 Redirect dplane002 10.0.0.3 NAT 142.8.0.1:1064-1095 Match Src Action 10.0.0.1 Redirect dplane001 10.0.0.2 Redirect dplane002 10.0.0.3 Redirect dplane003 dplane003 lo: 10.255.0.3 tun: 10.99.0.1 dplane004 lo: 10.255.0.4 tun: 10.99.0.1 dplane003 dplane004 dplane001 dplane002
  29. FabricNAT: Hyperscale Distributed NAT System eth0: 10.0.0.1 External service 119.0.0.1:80

    FabricNAT Global NAT Configuration Table ID Client Address NAT Public-IP:Port-Range Worker 1 10.0.0.1 142.8.0.1:1000-1031 dplane001 2 10.0.0.2 142.8.0.1:1032-1063 dplane002 3 10.0.0.3 142.8.0.1:1064-1095 dplane003 Match Src Action 10.0.0.1 NAT 142.8.0.1:1000-1031 10.0.0.2 Redirect dplane002 10.0.0.3 Redirect dplane003 Match Src Action 10.0.0.1 Redirect dplane001 10.0.0.2 NAT 142.8.0.1:1032:1063 10.0.0.3 Redirect dplane003 dplane001 lo: 10.255.0.1 tun: 10.99.0.1 dplane002 lo: 10.255.0.2 tun: 10.99.0.1 src: 10.0.0.1:2859 dst: 119.0.0.1:80 proto: tcp Match Src Action 10.0.0.1 Redirect dplane001 10.0.0.2 Redirect dplane002 10.0.0.3 NAT 142.8.0.1:1064-1095 Match Src Action 10.0.0.1 Redirect dplane001 10.0.0.2 Redirect dplane002 10.0.0.3 Redirect dplane003 dplane003 lo: 10.255.0.3 tun: 10.99.0.1 dplane004 lo: 10.255.0.4 tun: 10.99.0.1 dplane003 dplane004 dplane001 dplane002
  30. FabricNAT: Hyperscale Distributed NAT System eth0: 10.0.0.1 External service 119.0.0.1:80

    FabricNAT Global NAT Configuration Table ID Client Address NAT Public-IP:Port-Range Worker 1 10.0.0.1 142.8.0.1:1000-1031 dplane001 2 10.0.0.2 142.8.0.1:1032-1063 dplane002 3 10.0.0.3 142.8.0.1:1064-1095 dplane003 src: 142.8.0.1:1010 dst: 119.0.0.1:80 proto: tcp Match Src Action 10.0.0.1 NAT 142.8.0.1:1000-1031 10.0.0.2 Redirect dplane002 10.0.0.3 Redirect dplane003 Match Src Action 10.0.0.1 Redirect dplane001 10.0.0.2 NAT 142.8.0.1:1032:1063 10.0.0.3 Redirect dplane003 dplane001 lo: 10.255.0.1 tun: 10.99.0.1 dplane002 lo: 10.255.0.2 tun: 10.99.0.1 Match Src Action 10.0.0.1 Redirect dplane001 10.0.0.2 Redirect dplane002 10.0.0.3 NAT 142.8.0.1:1064-1095 Match Src Action 10.0.0.1 Redirect dplane001 10.0.0.2 Redirect dplane002 10.0.0.3 Redirect dplane003 dplane003 lo: 10.255.0.3 tun: 10.99.0.1 dplane004 lo: 10.255.0.4 tun: 10.99.0.1 dplane003 dplane004 dplane001 dplane002
  31. FabricNAT: Hyperscale Distributed NAT System eth0: 10.0.0.1 External service 119.0.0.1:80

    FabricNAT Global NAT Configuration Table ID Client Address NAT Public-IP:Port-Range Worker 1 10.0.0.1 142.8.0.1:1000-1031 dplane001 2 10.0.0.2 142.8.0.1:1032-1063 dplane002 3 10.0.0.3 142.8.0.1:1064-1095 dplane003 src: 142.8.0.1:1010 dst: 119.0.0.1:80 proto: tcp Match Src Action 10.0.0.1 NAT 142.8.0.1:1000-1031 10.0.0.2 Redirect dplane002 10.0.0.3 Redirect dplane003 Match Src Action 10.0.0.1 Redirect dplane001 10.0.0.2 NAT 142.8.0.1:1032:1063 10.0.0.3 Redirect dplane003 dplane001 lo: 10.255.0.1 tun: 10.99.0.1 dplane002 lo: 10.255.0.2 tun: 10.99.0.1 Match Src Action 10.0.0.1 Redirect dplane001 10.0.0.2 Redirect dplane002 10.0.0.3 NAT 142.8.0.1:1064-1095 Match Src Action 10.0.0.1 Redirect dplane001 10.0.0.2 Redirect dplane002 10.0.0.3 Redirect dplane003 dplane003 lo: 10.255.0.3 tun: 10.99.0.1 dplane004 lo: 10.255.0.4 tun: 10.99.0.1 dplane003 dplane004 dplane001 dplane002
  32. Match Src Action 10.0.0.1 NAT 142.8.0.1:1000-1031 10.0.0.2 Redirect dplane002 10.0.0.3

    Redirect dplane003 dplane001 FabricNAT: Hyperscale Distributed NAT System eth0: 10.0.0.1 External service 119.0.0.1:80 src: 142.8.0.1:1010 dst: 119.0.0.1:80 proto: tcp FabricNAT Global NAT Configuration Table ID Client Address NAT Public-IP:Port-Range Worker 1 10.0.0.1 142.8.0.1:1000-1031 dplane001 2 10.0.0.2 142.8.0.1:1032-1063 dplane002 3 10.0.0.3 142.8.0.1:1064-1095 dplane003 Match Src Action 10.0.0.1 Redirect dplane001 10.0.0.2 NAT 142.8.0.1:1032:1063 10.0.0.3 Redirect dplane003 dplane002 lo: 10.255.0.2 tun: 10.99.0.1 Match Src Action 10.0.0.1 Redirect dplane001 10.0.0.2 Redirect dplane002 10.0.0.3 NAT 142.8.0.1:1064-1095 Match Src Action 10.0.0.1 Redirect dplane001 10.0.0.2 Redirect dplane002 10.0.0.3 Redirect dplane003 dplane003 lo: 10.255.0.3 tun: 10.99.0.1 dplane004 lo: 10.255.0.4 tun: 10.99.0.1 dplane003 dplane004 dplane002
  33. Match Src Action 10.0.0.1 NAT 142.8.0.1:1000-1031 10.0.0.2 Redirect dplane002 10.0.0.3

    Redirect dplane003 dplane001 FabricNAT: Hyperscale Distributed NAT System eth0: 10.0.0.1 External service 119.0.0.1:80 src: 142.8.0.1:1010 dst: 119.0.0.1:80 proto: tcp FabricNAT Global NAT Configuration Table ID Client Address NAT Public-IP:Port-Range Worker 1 10.0.0.1 142.8.0.1:1000-1031 dplane001 2 10.0.0.2 142.8.0.1:1032-1063 dplane002 3 10.0.0.3 142.8.0.1:1064-1095 dplane003 Match Dst Action 142.8.0.1:1000-1031 Red dplane001 10.0.0.2 NAT 142.8.0.1:1032: 10.0.0.3 Redirect dplane003 dplane002 lo: 10.255.0.2 tun: 10.99.0.1 Match Dst Action 142.8.0.1:1000-1031 Red dplane001 10.0.0.2 Redirect dplane002 10.0.0.3 NAT 142.8.0.1:1064- Match Dst Action 142.8.0.1:1000-1031 Red dplane001 10.0.0.2 Redirect dplane002 10.0.0.3 Redirect dplane003 dplane003 lo: 10.255.0.3 tun: 10.99.0.1 dplane004 lo: 10.255.0.4 tun: 10.99.0.1 dplane003 dplane004 dplane002
  34. Recap FabricNAT • Pre allocate Port Range for Session Quota


    • Worker Election per each Clients and No State Sync for Scalability
 • 2-Step Routing to Construct Symmetric Routing
  35. FabricNAT: Hyperscale Distributed NAT System eth0: 10.0.0.1 External service 119.0.0.1:80

    FabricNAT Global NAT Configuration Table ID Client Address NAT Public-IP:Port-Range Worker 1 10.0.0.1 142.8.0.1:1000-1031 dplane001 2 10.0.0.2 142.8.0.1:1032-1063 dplane002 3 10.0.0.3 142.8.0.1:1064-1095 dplane003 src: 142.8.0.1:1010 dst: 119.0.0.1:80 proto: tcp Match Src Action 10.0.0.1 NAT 142.8.0.1:1000-1031 10.0.0.2 Redirect dplane002 10.0.0.3 Redirect dplane003 Match Src Action 10.0.0.1 Redirect dplane001 10.0.0.2 NAT 142.8.0.1:1032:1063 10.0.0.3 Redirect dplane003 dplane001 dplane002 Match Src Action 10.0.0.1 Redirect dplane001 10.0.0.2 Redirect dplane002 10.0.0.3 NAT 142.8.0.1:1064-1095 Match Src Action 10.0.0.1 Redirect dplane001 10.0.0.2 Redirect dplane002 10.0.0.3 Redirect dplane003 dplane003 dplane004 Working as N+1 redundancy…?
  36. N+1 Active/Active…? 1/N Blast Radius …? eth0: 10.0.0.1 External service

    119.0.0.1:80 Match Src Action 10.0.0.1 NAT 142.8.0.1:1000-1031 10.0.0.2 Redirect dplane002 10.0.0.3 Redirect dplane003 Match Src Action 10.0.0.1 Redirect dplane001 10.0.0.2 NAT 142.8.0.1:1032:1063 10.0.0.3 Redirect dplane003 dplane001 lo: 10.255.0.1 tun: 10.99.0.1 dplane002 lo: 10.255.0.2 tun: 10.99.0.1 Match Src Action 10.0.0.1 Redirect dplane001 10.0.0.2 Redirect dplane002 10.0.0.3 NAT 142.8.0.1:1064-1095 Match Src Action 10.0.0.1 Redirect dplane001 10.0.0.2 Redirect dplane002 10.0.0.3 Redirect dplane003 dplane003 lo: 10.255.0.3 tun: 10.99.0.1 dplane004 lo: 10.255.0.4 tun: 10.99.0.1 eth0: 10.0.0.2 eth0: 10.0.0.3 FabricNAT Global NAT Configuration Table ID Client Address NAT Public-IP:Port-Range Worker 1 10.0.0.1 142.8.0.1:1000-1031 dplane001 2 10.0.0.2 142.8.0.1:1032-1063 dplane002 3 10.0.0.3 142.8.0.1:1064-1095 dplane003 eth0: 10.0.0.1 eth0: 10.0.0.2 eth0: 10.0.0.3 dplane003 dplane004 dplane001 dplane002
  37. N+1 Active/Active…? 1/N Blast Radius …? eth0: 10.0.0.1 External service

    119.0.0.1:80 Match Src Action 10.0.0.1 NAT 142.8.0.1:1000-1031 10.0.0.2 Redirect dplane002 10.0.0.3 Redirect dplane003 Match Src Action 10.0.0.1 Redirect dplane001 10.0.0.2 NAT 142.8.0.1:1032:1063 10.0.0.3 Redirect dplane003 dplane001 dplane002 Match Src Action 10.0.0.1 Redirect dplane001 10.0.0.2 Redirect dplane002 10.0.0.3 NAT 142.8.0.1:1064-1095 Match Src Action 10.0.0.1 Redirect dplane001 10.0.0.2 Redirect dplane002 10.0.0.3 Redirect dplane003 dplane003 dplane004 eth0: 10.0.0.2 eth0: 10.0.0.3 FabricNAT Global NAT Configuration Table ID Client Address NAT Public-IP:Port-Range Worker 1 10.0.0.1 142.8.0.1:1000-1031 dplane001 2 10.0.0.2 142.8.0.1:1032-1063 dplane002 3 10.0.0.3 142.8.0.1:1064-1095 dplane003 eth0: 10.0.0.1 eth0: 10.0.0.2 eth0: 10.0.0.3
  38. N+1 Active/Active…? 1/N Blast Radius …? eth0: 10.0.0.1 External service

    119.0.0.1:80 Match Src Action 10.0.0.1 NAT 142.8.0.1:1000-1031 10.0.0.2 Redirect dplane002 10.0.0.3 Redirect dplane003 Match Src Action 10.0.0.1 Redirect dplane001 10.0.0.2 NAT 142.8.0.1:1032:1063 10.0.0.3 Redirect dplane003 dplane001 dplane002 Match Src Action 10.0.0.1 Redirect dplane001 10.0.0.2 Redirect dplane002 10.0.0.3 NAT 142.8.0.1:1064-1095 Match Src Action 10.0.0.1 Redirect dplane001 10.0.0.2 Redirect dplane002 10.0.0.3 Redirect dplane003 dplane003 dplane004 eth0: 10.0.0.2 eth0: 10.0.0.3 FabricNAT Global NAT Configuration Table ID Client Address NAT Public-IP:Port-Range Worker 1 10.0.0.1 142.8.0.1:1000-1031 dplane001 2 10.0.0.2 142.8.0.1:1032-1063 dplane002 3 10.0.0.3 142.8.0.1:1064-1095 dplane003 eth0: 10.0.0.1 eth0: 10.0.0.2 eth0: 10.0.0.3 External service 119.0.0.1:80 eth0: 10.0.0.1
  39. N+1 Active/Active…? 1/N Blast Radius …? eth0: 10.0.0.1 External service

    119.0.0.1:80 Match Src Action 10.0.0.1 NAT 142.8.0.1:1000-1031 10.0.0.2 Redirect dplane002 10.0.0.3 Redirect dplane003 Match Src Action 10.0.0.1 Redirect dplane001 10.0.0.2 NAT 142.8.0.1:1032:1063 10.0.0.3 Redirect dplane003 dplane001 lo: 10.255.0.1 tun: 10.99.0.1 dplane002 lo: 10.255.0.2 tun: 10.99.0.1 Match Src Action 10.0.0.1 Redirect dplane001 10.0.0.2 Redirect dplane002 10.0.0.3 NAT 142.8.0.1:1064-1095 Match Src Action 10.0.0.1 Redirect dplane001 10.0.0.2 Redirect dplane002 10.0.0.3 Redirect dplane003 dplane003 lo: 10.255.0.3 tun: 10.99.0.1 dplane004 lo: 10.255.0.4 tun: 10.99.0.1 eth0: 10.0.0.2 eth0: 10.0.0.3 FabricNAT Global NAT Configuration Table ID Client Address NAT Public-IP:Port-Range Micro-active 1 10.0.0.1 142.8.0.1:1000-1031 dplane001 2 10.0.0.2 142.8.0.1:1032-1063 dplane002 3 10.0.0.3 142.8.0.1:1064-1095 dplane003 eth0: 10.0.0.1 eth0: 10.0.0.2 eth0: 10.0.0.3 BGP Advertisement 10.99.0.1/32 BGP Advertisement 10.99.0.1/32 BGP Advertisement 10.99.0.1/32 BGP Advertisement 10.99.0.1/32 dplane001 lo: 10.255.0.1 tun: 10.99.0.1 dplane002 lo: 10.255.0.2 tun: 10.99.0.1 dplane003 lo: 10.255.0.3 tun: 10.99.0.1 dplane004 lo: 10.255.0.4 tun: 10.99.0.1 Match Src Action 10.0.0.1 NAT 142.8.0.1:1000-1031 10.0.0.2 Redirect dplane002 10.0.0.3 Redirect dplane003 Match Src Action 10.0.0.1 Redirect dplane001 10.0.0.2 NAT 142.8.0.1:1032:1063 10.0.0.3 Redirect dplane003 Match Src Action 10.0.0.1 Redirect dplane001 10.0.0.2 Redirect dplane002 10.0.0.3 NAT 142.8.0.1:1064-1095 Match Src Action 10.0.0.1 Redirect dplane001 10.0.0.2 Redirect dplane002 10.0.0.3 Redirect dplane003 eth0: 10.0.0.1 dplane003 dplane004 dplane001 dplane002
  40. N+1 Active/Active…? 1/N Blast Radius …? eth0: 10.0.0.1 External service

    119.0.0.1:80 Match Src Action 10.0.0.1 NAT 142.8.0.1:1000-1031 10.0.0.2 Redirect dplane002 10.0.0.3 Redirect dplane003 Match Src Action 10.0.0.1 Redirect dplane001 10.0.0.2 NAT 142.8.0.1:1032:1063 10.0.0.3 Redirect dplane003 dplane001 lo: 10.255.0.1 tun: 10.99.0.1 dplane002 lo: 10.255.0.2 tun: 10.99.0.1 Match Src Action 10.0.0.1 Redirect dplane001 10.0.0.2 Redirect dplane002 10.0.0.3 NAT 142.8.0.1:1064-1095 Match Src Action 10.0.0.1 Redirect dplane001 10.0.0.2 Redirect dplane002 10.0.0.3 Redirect dplane003 dplane003 lo: 10.255.0.3 tun: 10.99.0.1 dplane004 lo: 10.255.0.4 tun: 10.99.0.1 eth0: 10.0.0.2 eth0: 10.0.0.3 FabricNAT Global NAT Configuration Table ID Client Address NAT Public-IP:Port-Range Micro-active 1 10.0.0.1 142.8.0.1:1000-1031 dplane001 2 10.0.0.2 142.8.0.1:1032-1063 dplane002 3 10.0.0.3 142.8.0.1:1064-1095 dplane003 eth0: 10.0.0.1 eth0: 10.0.0.2 eth0: 10.0.0.3 dplane001 lo: 10.255.0.1 tun: 10.99.0.1 dplane002 lo: 10.255.0.2 tun: 10.99.0.1 dplane003 lo: 10.255.0.3 tun: 10.99.0.1 dplane004 lo: 10.255.0.4 tun: 10.99.0.1 External service 119.0.0.1:80 eth0: 10.0.0.1 eth0: 10.0.0.2 eth0: 10.0.0.3 dplane003 dplane004 dplane001 dplane002
  41. N+1 Active/Active…? 1/N Blast Radius …? eth0: 10.0.0.1 External service

    119.0.0.1:80 Match Src Action 10.0.0.1 NAT 142.8.0.1:1000-1031 10.0.0.2 Redirect dplane002 10.0.0.3 Redirect dplane003 Match Src Action 10.0.0.1 Redirect dplane001 10.0.0.2 NAT 142.8.0.1:1032:1063 10.0.0.3 Redirect dplane003 dplane001 lo: 10.255.0.1 tun: 10.99.0.1 dplane002 lo: 10.255.0.2 tun: 10.99.0.1 Match Src Action 10.0.0.1 Redirect dplane001 10.0.0.2 Redirect dplane002 10.0.0.3 NAT 142.8.0.1:1064-1095 Match Src Action 10.0.0.1 Redirect dplane001 10.0.0.2 Redirect dplane002 10.0.0.3 Redirect dplane003 dplane003 lo: 10.255.0.3 tun: 10.99.0.1 dplane004 lo: 10.255.0.4 tun: 10.99.0.1 eth0: 10.0.0.2 eth0: 10.0.0.3 FabricNAT Global NAT Configuration Table ID Client Address NAT Public-IP:Port-Range Micro-active 1 10.0.0.1 142.8.0.1:1000-1031 dplane001 2 10.0.0.2 142.8.0.1:1032-1063 dplane002 3 10.0.0.3 142.8.0.1:1064-1095 dplane003 eth0: 10.0.0.1 eth0: 10.0.0.2 eth0: 10.0.0.3 dplane001 dplane002 lo: 10.255.0.2 tun: 10.99.0.1 dplane003 lo: 10.255.0.3 tun: 10.99.0.1 dplane004 lo: 10.255.0.4 tun: 10.99.0.1 External service 119.0.0.1:80 eth0: 10.0.0.1 eth0: 10.0.0.2 eth0: 10.0.0.3 1/N
 Blast Radius dplane003 dplane004 dplane002
  42. N+1 Active/Active…? 1/N Blast Radius …? eth0: 10.0.0.1 External service

    119.0.0.1:80 eth0: 10.0.0.2 eth0: 10.0.0.3 eth0: 10.0.0.1 eth0: 10.0.0.2 eth0: 10.0.0.3 dplane001 dplane002 dplane003 dplane004 External service 119.0.0.1:80 dplane005 dplane006 dplane007 dplane008
  43. N+1 Active/Active…? 1/N Blast Radius …? eth0: 10.0.0.1 External service

    119.0.0.1:80 eth0: 10.0.0.2 eth0: 10.0.0.3 eth0: 10.0.0.1 eth0: 10.0.0.2 eth0: 10.0.0.3 dplane001 dplane002 dplane003 External service 119.0.0.1:80 dplane005 dplane006 dplane007 dplane008 1/8 Blast Radius == 1/N (N=8) dplane004
  44. Re-Recap FabricNAT • 2 Step Routing • 1st step: Robust

    with Many Stateless Active path • 2nd step: Not Robust but Blast Radius is 1/N
 • Gracefull v.s. Small Blast Radius • “Small Blast Radius and Hyperscale” >>> "Big Blast Radius and Graceful” • Getting Gracefully is Additional-Step
  45. SDN Mechanism For Configuration

  46. How we configure …? eth0: 10.0.0.1 External service 119.0.0.1:80 Match

    Src Action 10.0.0.1 NAT 142.8.0.1:1000-1031 10.0.0.2 Redirect dplane002 10.0.0.3 Redirect dplane003 Match Src Action 10.0.0.1 Redirect dplane001 10.0.0.2 NAT 142.8.0.1:1032:1063 10.0.0.3 Redirect dplane003 dplane001 lo: 10.255.0.1 tun: 10.99.0.1 dplane002 lo: 10.255.0.2 tun: 10.99.0.1 Match Src Action 10.0.0.1 Redirect dplane001 10.0.0.2 Redirect dplane002 10.0.0.3 NAT 142.8.0.1:1064-1095 Match Src Action 10.0.0.1 Redirect dplane001 10.0.0.2 Redirect dplane002 10.0.0.3 Redirect dplane003 dplane003 lo: 10.255.0.3 tun: 10.99.0.1 dplane004 lo: 10.255.0.4 tun: 10.99.0.1 eth0: 10.0.0.2 eth0: 10.0.0.3 FabricNAT Global NAT Configuration Table ID Client Address NAT Public-IP:Port-Range Worker 1 10.0.0.1 142.8.0.1:1000-1031 dplane001 2 10.0.0.2 142.8.0.1:1032-1063 dplane002 3 10.0.0.3 142.8.0.1:1064-1095 dplane003 eth0: 10.0.0.1 dplane003 dplane004 dplane001 dplane002
  47. How we configure …? eth0: 10.0.0.1 External service 119.0.0.1:80 eth0:

    10.0.0.2 eth0: 10.0.0.3 FabricNAT Global NAT Configuration Table ID Client Address NAT Public-IP:Port-Range Micro-active 1 10.0.0.1 142.8.0.1:1000-1031 dplane001 2 10.0.0.2 142.8.0.1:1032-1063 dplane002 3 10.0.0.3 142.8.0.1:1064-1095 dplane003 eth0: 10.0.0.1 Match Src Action 10.0.0.1 NAT 142.8.0.1:1000-1031 10.0.0.2 Redirect dplane002 10.0.0.3 Redirect dplane003 Match Src Action 10.0.0.1 Redirect dplane001 10.0.0.2 NAT 142.8.0.1:1032:1063 10.0.0.3 Redirect dplane003 dplane001 lo: 10.255.0.1 tun: 10.99.0.1 dplane002 lo: 10.255.0.2 tun: 10.99.0.1 Match Src Action 10.0.0.1 Redirect dplane001 10.0.0.2 Redirect dplane002 10.0.0.3 NAT 142.8.0.1:1064-1095 Match Src Action 10.0.0.1 Redirect dplane001 10.0.0.2 Redirect dplane002 10.0.0.3 Redirect dplane003 dplane003 lo: 10.255.0.3 tun: 10.99.0.1 dplane004 lo: 10.255.0.4 tun: 10.99.0.1 Dataplane Clustering
 with Consul dplane003 dplane004 dplane001 dplane002
  48. How we configure …? eth0: 10.0.0.1 External service 119.0.0.1:80 eth0:

    10.0.0.2 eth0: 10.0.0.3 Match Src Action 10.0.0.1 NAT 142.8.0.1:1000-1031 10.0.0.2 Redirect dplane002 10.0.0.3 Redirect dplane003 Match Src Action 10.0.0.1 Redirect dplane001 10.0.0.2 NAT 142.8.0.1:1032:1063 10.0.0.3 Redirect dplane003 dplane001 lo: 10.255.0.1 tun: 10.99.0.1 dplane002 lo: 10.255.0.2 tun: 10.99.0.1 Match Src Action 10.0.0.1 Redirect dplane001 10.0.0.2 Redirect dplane002 10.0.0.3 NAT 142.8.0.1:1064-1095 Match Src Action 10.0.0.1 Redirect dplane001 10.0.0.2 Redirect dplane002 10.0.0.3 Redirect dplane003 dplane003 lo: 10.255.0.3 tun: 10.99.0.1 dplane004 lo: 10.255.0.4 tun: 10.99.0.1 Distributed Routing 
 Configuration
 with Etcd FabricNAT Global NAT Configuration Table ID Client Address NAT Public-IP:Port-Range Worker 1 10.0.0.1 142.8.0.1:1000-1031 dplane001 2 10.0.0.2 142.8.0.1:1032-1063 dplane002 3 10.0.0.3 142.8.0.1:1064-1095 dplane003 dplane003 dplane004 dplane001 dplane002
  49. eth0: 10.0.0.1 External service 119.0.0.1:80 eth0: 10.0.0.2 eth0: 10.0.0.3 How

    we configure …? dplane002 lo: 10.255.0.2 tun: 10.99.0.1 Match Src Action 10.0.0.1 Redirect dplane001 10.0.0.2 NAT 142.8.0.1:1032:1063 10.0.0.3 Redirect dplane003 Match Src Action 10.0.0.1 Redirect dplane001 10.0.0.2 Redirect dplane002 10.0.0.3 Redirect dplane003 dplane004 lo: 10.255.0.4 tun: 10.99.0.1 dplane001 lo: 10.255.0.1 tun: 10.99.0.1 dplane003 lo: 10.255.0.3 tun: 10.99.0.1 Match Src Action 10.0.0.1 NAT 142.8.0.1:1000-1031 10.0.0.2 Redirect dplane002 10.0.0.3 Redirect dplane003 Match Src Action 10.0.0.1 Redirect dplane001 10.0.0.2 Redirect dplane002 10.0.0.3 NAT 142.8.0.1:1064-1095 scheduler apiserver Natlet Natlet Natlet Natlet FabricNAT Global NAT Configuration Table ID Client Address NAT Public-IP:Port-Range Worker 1 10.0.0.1 142.8.0.1:1000-1031 dplane001 2 10.0.0.2 142.8.0.1:1032-1063 dplane002 3 10.0.0.3 142.8.0.1:1064-1095 dplane003 dplane002 dplane004 dplane001 dplane003
  50. eth0: 10.0.0.1 External service 119.0.0.1:80 eth0: 10.0.0.2 eth0: 10.0.0.3 How

    we configure …? dplane002 lo: 10.255.0.2 tun: 10.99.0.1 Match Src Action 10.0.0.1 Redirect dplane001 10.0.0.2 NAT 142.8.0.1:1032:1063 10.0.0.3 Redirect dplane003 Match Src Action 10.0.0.1 Redirect dplane001 10.0.0.2 Redirect dplane002 10.0.0.3 Redirect dplane003 dplane004 lo: 10.255.0.4 tun: 10.99.0.1 dplane001 lo: 10.255.0.1 tun: 10.99.0.1 dplane003 lo: 10.255.0.3 tun: 10.99.0.1 Match Src Action 10.0.0.1 NAT 142.8.0.1:1000-1031 10.0.0.2 Redirect dplane002 10.0.0.3 Redirect dplane003 Match Src Action 10.0.0.1 Redirect dplane001 10.0.0.2 Redirect dplane002 10.0.0.3 NAT 142.8.0.1:1064-1095 scheduler apiserver Natlet Natlet Natlet Natlet FabricNAT Global NAT Configuration Table ID Client Address NAT Public-IP:Port-Range Micro-active 1 10.0.0.1 142.8.0.1:1000-1031 dplane001 2 10.0.0.2 142.8.0.1:1032-1063 dplane002 3 10.0.0.3 142.8.0.1:1064-1095 dplane003 dplane002 dplane004 dplane001 dplane003 dplane002 dplane004 dplane001 dplane003 Consul Client Consul Client Consul Client Consul Client Consul Server Natlet Natlet Natlet Natlet Watch
 Consul Watch
 Consul Watch
 Consul Watch
 Consul Gossip
 protocol scheduler Watch
 Consul
  51. eth0: 10.0.0.1 External service 119.0.0.1:80 eth0: 10.0.0.2 eth0: 10.0.0.3 How

    we configure …? dplane002 Match Src Action 10.0.0.1 Redirect dplane001 10.0.0.2 NAT 142.8.0.1:1032:1063 10.0.0.3 Redirect dplane003 Match Src Action 10.0.0.1 Redirect dplane001 10.0.0.2 Redirect dplane002 10.0.0.3 Redirect dplane003 dplane004 dplane001 dplane003 Match Src Action 10.0.0.1 NAT 142.8.0.1:1000-1031 10.0.0.2 Redirect dplane002 10.0.0.3 Redirect dplane003 Match Src Action 10.0.0.1 Redirect dplane001 10.0.0.2 Redirect dplane002 10.0.0.3 NAT 142.8.0.1:1064-1095 scheduler apiserver Natlet Natlet Natlet Natlet FabricNAT Global NAT Configuration Table ID Client Address NAT Public-IP:Port-Range Micro-active 1 10.0.0.1 142.8.0.1:1000-1031 dplane001 2 10.0.0.2 142.8.0.1:1032-1063 dplane002 3 10.0.0.3 142.8.0.1:1064-1095 dplane003 Etcd-Watch Etcd-Watch Etcd-Watch Etcd-Watch Natlet Natlet Natlet Natlet apiserver Etcd-Put scheduler Etcd-Watch
  52. eth0: 10.0.0.1 External service 119.0.0.1:80 eth0: 10.0.0.2 eth0: 10.0.0.3 How

    we configure …? dplane002 lo: 10.255.0.2 tun: 10.99.0.1 Match Src Action 10.0.0.1 Redirect dplane001 10.0.0.2 NAT 142.8.0.1:1032:1063 10.0.0.3 Redirect dplane003 Match Src Action 10.0.0.1 Redirect dplane001 10.0.0.2 Redirect dplane002 10.0.0.3 Redirect dplane003 dplane004 lo: 10.255.0.4 tun: 10.99.0.1 dplane001 lo: 10.255.0.1 tun: 10.99.0.1 dplane003 lo: 10.255.0.3 tun: 10.99.0.1 Match Src Action 10.0.0.1 NAT 142.8.0.1:1000-1031 10.0.0.2 Redirect dplane002 10.0.0.3 Redirect dplane003 Match Src Action 10.0.0.1 Redirect dplane001 10.0.0.2 Redirect dplane002 10.0.0.3 NAT 142.8.0.1:1064-1095 scheduler apiserver Natlet eth0: 10.0.0.4 Natlet Natlet Natlet 3. Etcd 
 DATA PUT 4. Notice
 New Resource 2. Port-allocation 1096-1127 FabricNAT Global NAT Configuration Table ID Client Address NAT Public-IP:Port-Range Worker 1 10.0.0.1 142.8.0.1:1000-1031 dplane001 2 10.0.0.2 142.8.0.1:1032-1063 dplane002 3 10.0.0.3 142.8.0.1:1064-1095 dplane003 4 10.0.0.4 142.8.0.1:1096-1127 NULL 1. NAT Create
 “client“: "10.0.04” “vip”: “142.2.0.1” dplane002 dplane004 dplane001 dplane003 NAT Configuration
 Client 10.0.0.4
 VIP 142.8.0.1
  53. eth0: 10.0.0.1 External service 119.0.0.1:80 eth0: 10.0.0.2 eth0: 10.0.0.3 How

    we configure …? dplane002 Match Src Action 10.0.0.1 Redirect dplane001 10.0.0.2 NAT 142.8.0.1:1032:1063 10.0.0.3 Redirect dplane003 Match Src Action 10.0.0.1 Redirect dplane001 10.0.0.2 Redirect dplane002 10.0.0.3 Redirect dplane003 dplane004 dplane001 dplane003 Match Src Action 10.0.0.1 NAT 142.8.0.1:1000-1031 10.0.0.2 Redirect dplane002 10.0.0.3 Redirect dplane003 Match Src Action 10.0.0.1 Redirect dplane001 10.0.0.2 Redirect dplane002 10.0.0.3 NAT 142.8.0.1:1064-1095 apiserver Natlet eth0: 10.0.0.4 Natlet Natlet Natlet FabricNAT Global NAT Configuration Table ID Client Address NAT Public-IP:Port-Range Micro-active 1 10.0.0.1 142.8.0.1:1000-1031 dplane001 2 10.0.0.2 142.8.0.1:1032-1063 dplane002 3 10.0.0.3 142.8.0.1:1064-1095 dplane003 4 10.0.0.4 142.8.0.1:1096-1127 NULL Get Healthy Dplane nodes scheduler
  54. eth0: 10.0.0.1 External service 119.0.0.1:80 eth0: 10.0.0.2 eth0: 10.0.0.3 How

    we configure …? dplane002 lo: 10.255.0.2 tun: 10.99.0.1 Match Src Action 10.0.0.1 Redirect dplane001 10.0.0.2 NAT 142.8.0.1:1032:1063 10.0.0.3 Redirect dplane003 Match Src Action 10.0.0.1 Redirect dplane001 10.0.0.2 Redirect dplane002 10.0.0.3 Redirect dplane003 dplane004 lo: 10.255.0.4 tun: 10.99.0.1 dplane001 lo: 10.255.0.1 tun: 10.99.0.1 dplane003 lo: 10.255.0.3 tun: 10.99.0.1 Match Src Action 10.0.0.1 NAT 142.8.0.1:1000-1031 10.0.0.2 Redirect dplane002 10.0.0.3 Redirect dplane003 Match Src Action 10.0.0.1 Redirect dplane001 10.0.0.2 Redirect dplane002 10.0.0.3 NAT 142.8.0.1:1064-1095 scheduler apiserver Natlet eth0: 10.0.0.4 Natlet Natlet Natlet 1. Get Healthy Dplane nodes 2. Cluster Info 4. Re-write
 Scheduled
 Resource 3. Worker
 election
 dplane004 dplane002 dplane004 dplane001 dplane003 FabricNAT Global NAT Configuration Table ID Client Address NAT Public-IP:Port-Range Worker 1 10.0.0.1 142.8.0.1:1000-1031 dplane001 2 10.0.0.2 142.8.0.1:1032-1063 dplane002 3 10.0.0.3 142.8.0.1:1064-1095 dplane003 4 10.0.0.4 142.8.0.1:1096-1127 dplane004
  55. eth0: 10.0.0.1 External service 119.0.0.1:80 eth0: 10.0.0.2 eth0: 10.0.0.3 How

    we configure …? dplane002 Match Src Action 10.0.0.1 Redirect dplane001 10.0.0.2 NAT 142.8.0.1:1032:1063 10.0.0.3 Redirect dplane003 Match Src Action 10.0.0.1 Redirect dplane001 10.0.0.2 Redirect dplane002 10.0.0.3 Redirect dplane003 dplane004 dplane001 dplane003 Match Src Action 10.0.0.1 NAT 142.8.0.1:1000-1031 10.0.0.2 Redirect dplane002 10.0.0.3 Redirect dplane003 Match Src Action 10.0.0.1 Redirect dplane001 10.0.0.2 Redirect dplane002 10.0.0.3 NAT 142.8.0.1:1064-1095 scheduler apiserver eth0: 10.0.0.4 Natlet Natlet Natlet Natlet Client-IP=10.0.0.1
 NAT=142.8.0.1:1096-1127
 Worker=dplane4 FabricNAT Global NAT Configuration Table ID Client Address NAT Public-IP:Port-Range Worker 1 10.0.0.1 142.8.0.1:1000-1031 dplane001 2 10.0.0.2 142.8.0.1:1032-1063 dplane002 3 10.0.0.3 142.8.0.1:1064-1095 dplane003 4 10.0.0.4 142.8.0.1:1096-1127 dplane004 Etcd
 Watch Etcd
 Watch Etcd
 Watch Etcd
 Watch
  56. eth0: 10.0.0.1 External service 119.0.0.1:80 eth0: 10.0.0.2 eth0: 10.0.0.3 How

    we configure …? dplane002 dplane004 dplane001 dplane003 scheduler apiserver eth0: 10.0.0.4 Natlet Natlet Natlet Natlet Match Src Action 10.0.0.1 Redirect dplane001 10.0.0.2 NAT 142.8.0.1:1032:1063 10.0.0.3 Redirect dplane003 10.0.0.4 Redirect dplane004 Match Src Action 10.0.0.1 NAT 142.8.0.1:1000-1031 10.0.0.2 Redirect dplane002 10.0.0.3 Redirect dplane003 10.0.0.4 Redirect dplane004 Match Src Action 10.0.0.1 Redirect dplane001 10.0.0.2 Redirect dplane002 10.0.0.3 NAT 142.8.0.1:1064-1095 10.0.0.4 Redirect dplane004 NON-WORKER FabricNAT Global NAT Configuration Table ID Client Address NAT Public-IP:Port-Range Worker 1 10.0.0.1 142.8.0.1:1000-1031 dplane001 2 10.0.0.2 142.8.0.1:1032-1063 dplane002 3 10.0.0.3 142.8.0.1:1064-1095 dplane003 4 10.0.0.4 142.8.0.1:1096-1127 dplane004 Client-IP=10.0.0.1
 NAT=142.8.0.1:1096-1127
 Worker=dplane4 WORKER Match Src Action 10.0.0.1 Redirect dplane001 10.0.0.2 Redirect dplane002 10.0.0.3 Redirect dplane003 10.0.0.4 NAT 142.8.0.1:1096-1127
  57. Recap FabricNAT SDN • SDN Design Principle • Without RPC

    • With • Declarative Configuration • Resource-Watch • Reconciliation • Use CloudNative Parts • Use Consul for dataplane clustering • Use Etcd as Watchable Robust KVS Respecting K8s
  58. eth0: 10.0.0.1 External service 119.0.0.1:80 eth0: 10.0.0.2 eth0: 10.0.0.3 How

    we configure …? scheduler apiserver eth0: 10.0.0.4 FabricNAT Global NAT Configuration Table ID Client Address NAT Public-IP:Port-Range Worker 1 10.0.0.1 142.8.0.1:1000-1031 dplane001 2 10.0.0.2 142.8.0.1:1032-1063 dplane002 3 10.0.0.3 142.8.0.1:1064-1095 dplane003 dplane002 lo: 10.255.0.2 tun: 10.99.0.1 Match Src Action 10.0.0.1 Redirect dplane001 10.0.0.2 NAT 142.8.0.1:1032:1063 10.0.0.3 Redirect dplane003 Match Src Action 10.0.0.1 Redirect dplane001 10.0.0.2 Redirect dplane002 10.0.0.3 Redirect dplane003 dplane004 lo: 10.255.0.4 tun: 10.99.0.1 dplane001 lo: 10.255.0.1 tun: 10.99.0.1 dplane003 lo: 10.255.0.3 tun: 10.99.0.1 Match Src Action 10.0.0.1 NAT 142.8.0.1:1000-1031 10.0.0.2 Redirect dplane002 10.0.0.3 Redirect dplane003 Match Src Action 10.0.0.1 Redirect dplane001 10.0.0.2 Redirect dplane002 10.0.0.3 NAT 142.8.0.1:1064-1095 Natlet Natlet Natlet Natlet dplane002 dplane004 dplane001 dplane003
  59. eth0: 10.0.0.1 External service 119.0.0.1:80 eth0: 10.0.0.2 eth0: 10.0.0.3 How

    we configure …? scheduler apiserver eth0: 10.0.0.4 FabricNAT Global NAT Configuration Table ID Client Address NAT Public-IP:Port-Range Worker 1 10.0.0.1 142.8.0.1:1000-1031 dplane001 2 10.0.0.2 142.8.0.1:1032-1063 dplane002 3 10.0.0.3 142.8.0.1:1064-1095 dplane003 dplane002 lo: 10.255.0.2 tun: 10.99.0.1 Match Src Action 10.0.0.1 Redirect dplane001 10.0.0.2 NAT 142.8.0.1:1032:1063 10.0.0.3 Redirect dplane003 Match Src Action 10.0.0.1 Redirect dplane001 10.0.0.2 Redirect dplane002 10.0.0.3 Redirect dplane003 dplane004 lo: 10.255.0.4 tun: 10.99.0.1 dplane001 lo: 10.255.0.1 tun: 10.99.0.1 dplane003 lo: 10.255.0.3 tun: 10.99.0.1 Match Src Action 10.0.0.1 NAT 142.8.0.1:1000-1031 10.0.0.2 Redirect dplane002 10.0.0.3 Redirect dplane003 Match Src Action 10.0.0.1 Redirect dplane001 10.0.0.2 Redirect dplane002 10.0.0.3 NAT 142.8.0.1:1064-1095 Natlet Natlet Natlet Natlet dplane002 dplane004 dplane001 dplane003
  60. eth0: 10.0.0.1 External service 119.0.0.1:80 eth0: 10.0.0.2 eth0: 10.0.0.3 How

    we configure …? scheduler apiserver eth0: 10.0.0.4 dplane002 lo: 10.255.0.2 tun: 10.99.0.1 Match Src Action 10.0.0.1 Redirect dplane001 10.0.0.2 NAT 142.8.0.1:1032:1063 10.0.0.3 Redirect dplane003 Match Src Action 10.0.0.1 Redirect dplane001 10.0.0.2 Redirect dplane002 10.0.0.3 Redirect dplane003 dplane004 lo: 10.255.0.4 tun: 10.99.0.1 dplane001 lo: 10.255.0.1 tun: 10.99.0.1 dplane003 lo: 10.255.0.3 tun: 10.99.0.1 Match Src Action 10.0.0.1 NAT 142.8.0.1:1000-1031 10.0.0.2 Redirect dplane002 10.0.0.3 Redirect dplane003 Match Src Action 10.0.0.1 Redirect dplane001 10.0.0.2 Redirect dplane002 10.0.0.3 NAT 142.8.0.1:1064-1095 Natlet Natlet Natlet Natlet ! ! ! ! 1. dplane001 
 was down 1. dplane001 
 was down 1. dplane001 
 was down 1. dplane001 
 was down dplane002 dplane004 dplane001 dplane003 scheduler ! 2. dplane001 
 was down FabricNAT Global NAT Configuration Table ID Client Address NAT Public-IP:Port-Range Micro-active 1 10.0.0.1 142.8.0.1:1000-1031 dplane001 2 10.0.0.2 142.8.0.1:1032-1063 dplane002 3 10.0.0.3 142.8.0.1:1064-1095 dplane003
  61. eth0: 10.0.0.1 External service 119.0.0.1:80 eth0: 10.0.0.2 eth0: 10.0.0.3 How

    we configure …? scheduler apiserver eth0: 10.0.0.4 FabricNAT Global NAT Configuration Table ID Client Address NAT Public-IP:Port-Range Micro-active 1 10.0.0.1 142.8.0.1:1000-1031 dplane001 2 10.0.0.2 142.8.0.1:1032-1063 dplane002 3 10.0.0.3 142.8.0.1:1064-1095 dplane003 4 10.0.0.4 142.8.0.1:1096-1127 NULL dplane002 lo: 10.255.0.2 tun: 10.99.0.1 dplane004 lo: 10.255.0.4 tun: 10.99.0.1 dplane003 lo: 10.255.0.3 tun: 10.99.0.1 Match Src Action 10.0.0.1 NAT 142.8.0.1:1000-1031 10.0.0.2 Redirect dplane002 10.0.0.3 Redirect dplane003 Natlet Natlet Natlet Natlet scheduler FabricNAT Global NAT Configuration Table-1 ID Client Address NAT Public-IP:Port-Range Worker 1 10.0.0.1 142.8.0.1:1000-1031 dplane004 2 10.0.0.2 142.8.0.1:1032-1063 dplane002 3 10.0.0.3 142.8.0.1:1064-1095 dplane003 dplane002 dplane004 dplane001 dplane003 ! ! ! Match Src Action 10.0.0.1 Redirect dplane004 10.0.0.2 NAT 142.8.0.1:1032:1063 10.0.0.3 Redirect dplane003 Match Src Action 10.0.0.1 NAT 142.8.0.1:1000-1031 10.0.0.2 Redirect dplane002 10.0.0.3 Redirect dplane003 Match Src Action 10.0.0.1 Redirect dplane004 10.0.0.2 Redirect dplane002 10.0.0.3 NAT 142.8.0.1:1064-1095
  62. Recap FabricNAT SDN • SDN Design Principle • Without RPC

    • With • Declarative Configuration • Resource-Watch • Reconciliation • Use CloudNative Parts • Use Consul for dataplane clustering • Use Etcd as Watchable Robust KVS Offload 
 Complex 
 Mechanism
  63. A Software’s Stability • “Proprietary Software” v.s. “Major OSS” •

    Which looks be stable …?
 • Compare the Upgrade Period • Natlet: ?? • Etcd: ??
  64. A Software’s Stability • “Proprietary Software” v.s. “Major OSS” •

    Which looks be stable …?
 • Compare the Upgrade Period • Natlet: upgrade once/day • Etcd: upgrade once/month~
  65. Software will be Broken • Development Difficulty • non broken

    software → Really Really Difficult • reboot-able software → More Easy and Realistic
 • Basic Principle • Casual Maintenance • Frequency Upgrade
  66. eth0: 10.0.0.1 External service 119.0.0.1:80 eth0: 10.0.0.2 eth0: 10.0.0.3 How

    we configure …? dplane002 Match Src Action 10.0.0.1 Redirect dplane001 10.0.0.2 NAT 142.8.0.1:1032:1063 10.0.0.3 Redirect dplane003 Match Src Action 10.0.0.1 Redirect dplane001 10.0.0.2 Redirect dplane002 10.0.0.3 Redirect dplane003 dplane004 dplane001 dplane003 Match Src Action 10.0.0.1 NAT 142.8.0.1:1000-1031 10.0.0.2 Redirect dplane002 10.0.0.3 Redirect dplane003 Match Src Action 10.0.0.1 Redirect dplane001 10.0.0.2 Redirect dplane002 10.0.0.3 NAT 142.8.0.1:1064-1095 scheduler apiserver eth0: 10.0.0.4 Natlet Natlet Natlet Natlet RPC FabricNAT Global NAT Configuration Table ID Client Address NAT Public-IP:Port-Range Worker 1 10.0.0.1 142.8.0.1:1000-1031 dplane001 2 10.0.0.2 142.8.0.1:1032-1063 dplane002 3 10.0.0.3 142.8.0.1:1064-1095 dplane003 4 10.0.0.4 142.8.0.1:1096-1127 dplane004 RPC RPC RPC Restarting
  67. eth0: 10.0.0.1 External service 119.0.0.1:80 eth0: 10.0.0.2 eth0: 10.0.0.3 How

    we configure …? dplane002 Match Src Action 10.0.0.1 Redirect dplane001 10.0.0.2 NAT 142.8.0.1:1032:1063 10.0.0.3 Redirect dplane003 Match Src Action 10.0.0.1 Redirect dplane001 10.0.0.2 Redirect dplane002 10.0.0.3 Redirect dplane003 dplane004 dplane001 dplane003 Match Src Action 10.0.0.1 NAT 142.8.0.1:1000-1031 10.0.0.2 Redirect dplane002 10.0.0.3 Redirect dplane003 Match Src Action 10.0.0.1 Redirect dplane001 10.0.0.2 Redirect dplane002 10.0.0.3 NAT 142.8.0.1:1064-1095 scheduler apiserver eth0: 10.0.0.4 Natlet Natlet Natlet Natlet Watch FabricNAT Global NAT Configuration Table ID Client Address NAT Public-IP:Port-Range Worker 1 10.0.0.1 142.8.0.1:1000-1031 dplane001 2 10.0.0.2 142.8.0.1:1032-1063 dplane002 3 10.0.0.3 142.8.0.1:1064-1095 dplane003 4 10.0.0.4 142.8.0.1:1096-1127 dplane004 Watch Watch Watch Restarting Restart
 Watch &
 Reconcile
  68. Software will be Broken • Development Difficulty • non broken

    software → Really Really Difficult • reboot-able software → More Easy and Realistic
 • Basic Principle • Casual Maintenance • Frequency Upgrade Enabled by Declarative Model and Reconciliation loop
  69. Operation / Monitoring

  70. Deploy/Upgrade/Restart Components with Ansible Internet Gateway Service NAT 142.8.0.1 NAT

    142.8.0.2 dplane001 dplane002 dplane003 dplane004 Natlet Natlet Natlet Natlet scheduler apiserver
  71. Monitoring :: Basic Metrics Internet Gateway Service NAT 142.8.0.1 NAT

    142.8.0.2 Natlet dplane001 dplane002 dplane003 dplane004 Natlet Natlet Natlet scheduler apiserver k8s consul_sd_config kube_
 state_
 metrics
  72. Monitoring :: Dynamic Robot Internet Gateway Service apiserver 1. OpenStack


    Token Issue 2. NAT Create 3. Replay Connection Information.
 Source IP
 Source Port etc.. 4. Check Connection
 Statement 5. NAT Delete NAT 142.8.0.1 NAT 142.8.0.2
  73. Monitoring :: Static Robot Internet Gateway Service NAT 142.8.0.1 NAT

    142.8.0.2 Use Blackbox Exporter consul_sd_config
  74. Next Step / Future Work • NAME based ACL •

    Automated Robot Operation • Private Network Gateway
  75. NAME based Access Control Internet Gateway Service NAT 142.8.0.1 NAT

    142.8.0.2 Allow-list ACL Allow-list ACL Allow-list Dest Proto api.external.linecorp.com https, http ftp.jaist.ac.jp ftp, https github.com ssh, https DNS
  76. Automated Robot Operation dplane002 Match Src Action 10.0.0.1 Redirect dplane001

    10.0.0.2 NAT 142.8.0.1:1032:1063 10.0.0.3 Redirect dplane003 Match Src Action 10.0.0.1 Redirect dplane001 10.0.0.2 Redirect dplane002 10.0.0.3 Redirect dplane003 dplane004 dplane001 dplane003 Match Src Action 10.0.0.1 NAT 142.8.0.1:1000-1031 10.0.0.2 Redirect dplane002 10.0.0.3 Redirect dplane003 Match Src Action 10.0.0.1 Redirect dplane001 10.0.0.2 Redirect dplane002 10.0.0.3 NAT 142.8.0.1:1064-1095 Natlet Natlet Natlet Natlet scheduler apiserver Current Ops
  77. Automated Robot Operation dplane002 Match Src Action 10.0.0.1 Redirect dplane001

    10.0.0.2 NAT 142.8.0.1:1032:1063 10.0.0.3 Redirect dplane003 Match Src Action 10.0.0.1 Redirect dplane001 10.0.0.2 Redirect dplane002 10.0.0.3 Redirect dplane003 dplane004 dplane001 dplane003 Match Src Action 10.0.0.1 NAT 142.8.0.1:1000-1031 10.0.0.2 Redirect dplane002 10.0.0.3 Redirect dplane003 Match Src Action 10.0.0.1 Redirect dplane001 10.0.0.2 Redirect dplane002 10.0.0.3 NAT 142.8.0.1:1064-1095 Natlet Natlet Natlet Natlet scheduler apiserver Automated
 Operation 
 Manifest Monitor- ing
  78. Private Zone Private Zone A NW-A1 NW-A2 NW-GW Private Zone

    B NW-B1 NW-B2 NW-GW
  79. Summary • Newly Provided “Internet Gateway” • We build the

    new routing mechanism called “FabricNAT” for backend • Distributed NAT mech achieving “Small Blast Radius” and “Hyperscale”
 • SDN Design using CloudNative blocks • Consul for Easy/Massive clustering • Etcd for Declarative Configuration • Stateless software component can be restart casually
 • Many Network Challenge on our Private Cloud :)
  80. Thank you