"Change" our private cloud infrastructures from single-AZ to multi-AZs Backbone Network part

Slide 1

Slide 1 text

"Change" our private cloud infrastructures from single-AZ to multi-AZs Backbone Network part Shu Mukai LINE Service Network 1 Team 2022/07/13@JANOG50

Slide 2

Slide 2 text

Agenda • Data Center Network概要 • Multi-AZs対応前のBackbone Networkと課題 • Multi-AZs対応後のBackbone Network

Slide 3

Slide 3 text

l プロフィール l 名前：向井脩 (Shu Mukai) l 居住地：福岡県福岡市 l 職歴 l 2015年〜2020年：KDDI Backbone Network, AS2516の設計・構築・運⽤ピアリング担当 l 2021年〜 : LINE ITSC Service Network 1 Team Inter-DC Backbone Network, AS38631, IP CLOSの設計・構築・運⽤ SONiC導⼊検討 ABOUT ME

Slide 4

Slide 4 text

Data Center Network概要

Slide 5

Slide 5 text

Data Center Network Legacy CLOS ref:https://www.janog.gr.jp/meeting/janog43/application/files/7915/4823/1858/janog43-line-kobayashi.pdf

Slide 6

Slide 6 text

Data Center Network External Spine Leaf Leaf ToR ToR Spine 0.0.0.0/0 Leaf Leaf ToR ToR Other DC routes Server Server Core DC Network 192.0.2.1/32 192.0.2.2/32 192.0.2.0/24 192.0.2.2/32 192.0.2.1/32 NW-POP Router Backbone Router

Slide 7

Slide 7 text

Data Center Network External Spine Leaf Leaf ToR ToR Spine Leaf Leaf ToR ToR Server Server Internet Traffic Inter-DC Traffic Core DC Network Internet Traffic, Inter-DC Traffic, East-West Traffic 全て、各種LINEサービスの正常稼働に必要不可⽋ East-West Traffic

Slide 8

Slide 8 text

Multi-AZs対応前の Backbone Networkと課題

Slide 9

Slide 9 text

The Internet Backbone network before multi-AZs CLOS CLOS Tokyo DC-1 Legacy Osaka DC Singapore DC Mainly serve JP/TW user DR site of Tokyo The Internet The Internet Mainly serve TH user

Slide 10

Slide 10 text

The Internet Routing for Internet Traffic CLOS CLOS Legacy The Internet The Internet Default route Default route Full routes Full routes Default route Full Routes Tokyo DC-1 Osaka DC Singapore DC

Slide 11

Slide 11 text

The Internet Routing for Internet Traffic CLOS CLOS Legacy The Internet The Internet Osaka Routes Tokyo Routes Tokyo Routes Osaka Routes SG Routes SG Routes Tokyo DC-1 Osaka DC Singapore DC

Slide 12

Slide 12 text

The Internet Routing for Inter-DC Traffic CLOS CLOS Legacy The Internet The Internet Osaka routes Tokyo routes SG routes SR-MPLS + MP-iBGP VRF for Inter-DC Traffic Tokyo DC-1 Osaka DC Singapore DC

Slide 13

Slide 13 text

The Internet Routing for Inter-DC Traffic CLOS CLOS Legacy The Internet The Internet Other DC routes Other DC routes Other DC routes SR-MPLS + MP-iBGP VRF for Inter-DC Traffic Tokyo DC-1 Osaka DC Singapore DC

Slide 14

Slide 14 text

The Internet Traffic flow CLOS CLOS Legacy Internet Traffic flow Inter-DC Traffic flow The Internet The Internet SR-MPLS + MP-iBGP VRF for Inter-DC Traffic Tokyo DC-1 Osaka DC Singapore DC

Slide 15

Slide 15 text

The Internet Traffic flow CLOS CLOS Legacy Internet Traffic flow Inter-DC Traffic flow The Internet The Internet Tokyo DC-1のネットワーク機器で障害が発⽣すると、 JP/TWで、LINE各種サービスに⼤きな影響が発⽣する SR-MPLS + MP-iBGP VRF for Inter-DC Traffic Tokyo DC-1 Osaka DC Singapore DC

Slide 16

Slide 16 text

Multi-AZs対応後の Backbone Network

Slide 17

Slide 17 text

l Tokyo Region内に複数のAZを構築 l 1つのAZで障害が発⽣しても、その他AZの Internet Traffic, Inter-DC Traffic, East-West Trafficは、影響を受けない構成 Design policy for multi-AZs network

Slide 18

Slide 18 text

Backbone network after multi-AZs CLOS CLOS CLOS Tokyo DC-3 AZ3 Tokyo DC-2 AZ2 Tokyo Region Tokyo DC-1 AZ1 CLOS CLOS The Internet The Internet The Internet Osaka DC Singapore DC

Slide 19

Slide 19 text

Backbone network after multi-AZs CLOS CLOS CLOS Tokyo DC-3 AZ3 Tokyo DC-2 AZ2 Tokyo Region Tokyo DC-1 AZ1 CLOS CLOS The Internet The Internet The Internet l 関東地⽅に新たに2つのDCを構築し、3つのDCを三⾓形で接続 DC間を接続する専⽤線は、複数社から異経路で調達 l 関東地⽅の3つのDCをAZ1〜AZ3と定義 Osaka DC Singapore DC

Slide 20

Slide 20 text

Backbone network after multi-AZs CLOS CLOS CLOS Tokyo DC-3 AZ3 Tokyo DC-2 AZ2 Tokyo Region Tokyo DC-1 AZ1 CLOS CLOS The Internet The Internet The Internet Osaka DC Singapore DC l Tokyo DC-1の⼀部サーバルームを、Legacy→CLOSへリノベーション NFVのmulti-AZs対応のためにCLOS構成が必要であるため。 ※Legacy構成のサーバールーム（non-AZサーバールーム）も引き続き運⽤中

Slide 21

Slide 21 text

Backbone network after multi-AZs CLOS CLOS CLOS Tokyo DC-3 AZ3 Tokyo DC-2 AZ2 Tokyo Region Tokyo DC-1 AZ1 CLOS CLOS The Internet The Internet The Internet l NW-POP RouterをBackbone Routerに接続 l Backbone Networkで、Internet Traffic⽤VRFを新規に作成 l Internet Traffic⽤のRoutingを是正 Osaka DC Singapore DC

Slide 22

Slide 22 text

Routing for Internet Traffic CLOS CLOS CLOS Tokyo DC-3 AZ3 Tokyo DC-2 AZ2 Tokyo Region Tokyo DC-1 AZ1 CLOS CLOS The Internet The Internet The Internet VRF for Internet Traffic Tokyo & Osaka Routes Tokyo & Osaka Routes Tokyo & Osaka Routes Tokyo & Osaka Routes SG Routes SG Routes Osaka DC Singapore DC

Slide 23

Slide 23 text

Routing for Internet Traffic CLOS CLOS CLOS Tokyo DC-3 AZ3 Tokyo DC-2 AZ2 Tokyo Region Tokyo DC-1 AZ1 CLOS CLOS The Internet The Internet The Internet VRF for Internet Traffic Full routes Default route Full routes Default route Default route Full Routes Osaka DC Singapore DC

Slide 24

Slide 24 text

Internet Traffic flow CLOS CLOS CLOS Tokyo DC-3 AZ3 Tokyo DC-2 AZ2 Tokyo Region Tokyo DC-1 AZ1 CLOS CLOS The Internet The Internet The Internet VRF for Internet Traffic l 平常時は、IGP COSTに従い東京NW-POPを利⽤ Osaka DC Singapore DC

Slide 25

Slide 25 text

Internet Traffic flow CLOS CLOS CLOS Tokyo DC-3 AZ3 Tokyo DC-2 AZ2 Tokyo Region Tokyo DC-1 AZ1 CLOS CLOS The Internet The Internet The Internet VRF for Internet Traffic Osaka DC Singapore DC

Slide 26

Slide 26 text

Internet Traffic flow CLOS CLOS CLOS Tokyo DC-3 AZ3 Tokyo DC-2 AZ2 Tokyo Region Tokyo DC-1 AZ1 CLOS CLOS The Internet The Internet The Internet VRF for Internet Traffic Osaka DC Singapore DC

Slide 27

Slide 27 text

Internet Traffic flow CLOS CLOS CLOS Tokyo DC-3 AZ3 Tokyo DC-2 AZ2 Tokyo Region Tokyo DC-1 AZ1 CLOS CLOS The Internet The Internet The Internet l 東京NW-POP障害時は、⼤阪NW-POPを利⽤ VRF for Internet Traffic Osaka DC Singapore DC

Slide 28

Slide 28 text

Osaka NW-POPを利⽤ Tokyo Region内に新規にNW-POPを構築 Pros Cons • 既存のネットワーク機器を流用できる • 東日本ユーザ向けのlatencyの増加がない • 東日本ユーザ向けのlatencyが増加する • ネットワーク機器を新規に調達する必要がある • 関東圏内での冗長では、NW-POP 冗長として不十分ではないか？ NW-POP Redundancy

Slide 29

Slide 29 text

Routing for Inter-DC Traffic CLOS CLOS CLOS Tokyo DC-3 AZ3 Tokyo DC-2 AZ2 Tokyo Region Tokyo DC-1 AZ1 CLOS CLOS The Internet The Internet The Internet Osaka routes AZ1 routes SG routes AZ3 routes AZ2 routes VRF for Inter-DC Traffic Osaka DC Singapore DC

Slide 30

Slide 30 text

Routing for Inter-DC Traffic CLOS CLOS CLOS Tokyo DC-3 AZ3 Tokyo DC-2 AZ2 Tokyo Region Tokyo DC-1 AZ1 CLOS CLOS The Internet The Internet The Internet Other DC routes Other DC routes Other DC routes Other DC routes Other DC routes VRF for Inter-DC Traffic Osaka DC Singapore DC

Slide 31

Slide 31 text

Inter-DC Traffic flow CLOS CLOS CLOS Tokyo DC-3 AZ3 Tokyo DC-2 AZ2 Tokyo Region Tokyo DC-1 AZ1 CLOS CLOS The Internet The Internet The Internet VRF for Inter-DC Traffic Osaka DC Singapore DC

Slide 32

Slide 32 text

Inter-DC Traffic flow CLOS CLOS CLOS Tokyo DC-3 AZ3 Tokyo DC-2 AZ2 Tokyo Region Tokyo DC-1 AZ1 CLOS CLOS The Internet The Internet The Internet VRF for Inter-DC Traffic Osaka DC Singapore DC

Slide 33

Slide 33 text

Inter-DC Traffic flow CLOS CLOS CLOS Tokyo DC-3 AZ3 Tokyo DC-2 AZ2 Tokyo Region Tokyo DC-1 AZ1 CLOS CLOS The Internet The Internet The Internet VRF for Inter-DC Traffic Osaka DC Singapore DC

Slide 34

Slide 34 text

l ⼤阪からインターネットへ流すと、東⽇本ユーザ向けのlatencyが増加 l オリジンが東⽇本にあるため、仕⽅がない l トランジットコスト増加 l ⼤阪でも、東京と同容量のトランジット確保が必要 →コミット値の上昇→トランジットコスト増加 l ピアリングで捌けるトラフィック量を少しでも増やしたい課題

Slide 35

Slide 35 text

l 関東地⽅に新たに2つのDCを構築し、それぞれのDCをAZとして定義 l 1つのAZで障害が発⽣しても、その他AZの Internet Traffic, Inter-DC Traffic, East-West Trafficは影響を受けない Backbone Network l NW-POP TokyoのBackupとして、NW-POP Osakaを利⽤ Conclusion

Slide 36

Slide 36 text

THANK YOU