Upgrade to Pro — share decks privately, control downloads, hide ads and more …

"Change" our private cloud infrastructures from single-AZ to multi-AZs Backbone Network part

"Change" our private cloud infrastructures from single-AZ to multi-AZs Backbone Network part

JANOG50での登壇資料です

LINE Developers
PRO

July 13, 2022
Tweet

More Decks by LINE Developers

Other Decks in Technology

Transcript

  1. "Change" our private cloud infrastructures from single-AZ to multi-AZs Backbone

    Network part Shu Mukai LINE Service Network 1 Team 2022/07/13@JANOG50
  2. Agenda • Data Center Network概要 • Multi-AZs対応前のBackbone Networkと課題 • Multi-AZs対応後のBackbone

    Network
  3. l プロフィール l 名前:向井脩 (Shu Mukai) l 居住地:福岡県福岡市 l 職歴

    l 2015年〜2020年:KDDI Backbone Network, AS2516の設計・構築・運⽤ ピアリング担当 l 2021年〜 : LINE ITSC Service Network 1 Team Inter-DC Backbone Network, AS38631, IP CLOSの設計・構築・運⽤ SONiC導⼊検討 ABOUT ME
  4. Data Center Network概要

  5. Data Center Network Legacy CLOS ref:https://www.janog.gr.jp/meeting/janog43/application/files/7915/4823/1858/janog43-line-kobayashi.pdf

  6. Data Center Network External Spine Leaf Leaf ToR ToR Spine

    0.0.0.0/0 Leaf Leaf ToR ToR Other DC routes Server Server Core DC Network 192.0.2.1/32 192.0.2.2/32 192.0.2.0/24 192.0.2.2/32 192.0.2.1/32 NW-POP Router Backbone Router
  7. Data Center Network External Spine Leaf Leaf ToR ToR Spine

    Leaf Leaf ToR ToR Server Server Internet Traffic Inter-DC Traffic Core DC Network Internet Traffic, Inter-DC Traffic, East-West Traffic 全て、各種LINEサービスの正常稼働に必要不可⽋ East-West Traffic
  8. Multi-AZs対応前の Backbone Networkと課題

  9. The Internet Backbone network before multi-AZs CLOS CLOS Tokyo DC-1

    Legacy Osaka DC Singapore DC Mainly serve JP/TW user DR site of Tokyo The Internet The Internet Mainly serve TH user
  10. The Internet Routing for Internet Traffic CLOS CLOS Legacy The

    Internet The Internet Default route Default route Full routes Full routes Default route Full Routes Tokyo DC-1 Osaka DC Singapore DC
  11. The Internet Routing for Internet Traffic CLOS CLOS Legacy The

    Internet The Internet Osaka Routes Tokyo Routes Tokyo Routes Osaka Routes SG Routes SG Routes Tokyo DC-1 Osaka DC Singapore DC
  12. The Internet Routing for Inter-DC Traffic CLOS CLOS Legacy The

    Internet The Internet Osaka routes Tokyo routes SG routes SR-MPLS + MP-iBGP VRF for Inter-DC Traffic Tokyo DC-1 Osaka DC Singapore DC
  13. The Internet Routing for Inter-DC Traffic CLOS CLOS Legacy The

    Internet The Internet Other DC routes Other DC routes Other DC routes SR-MPLS + MP-iBGP VRF for Inter-DC Traffic Tokyo DC-1 Osaka DC Singapore DC
  14. The Internet Traffic flow CLOS CLOS Legacy Internet Traffic flow

    Inter-DC Traffic flow The Internet The Internet SR-MPLS + MP-iBGP VRF for Inter-DC Traffic Tokyo DC-1 Osaka DC Singapore DC
  15. The Internet Traffic flow CLOS CLOS Legacy Internet Traffic flow

    Inter-DC Traffic flow The Internet The Internet Tokyo DC-1のネットワーク機器で障害が発⽣すると、 JP/TWで、LINE各種サービスに⼤きな影響が発⽣する SR-MPLS + MP-iBGP VRF for Inter-DC Traffic Tokyo DC-1 Osaka DC Singapore DC
  16. Multi-AZs対応後の Backbone Network

  17. l Tokyo Region内に複数のAZを構築 l 1つのAZで障害が発⽣しても、その他AZの Internet Traffic, Inter-DC Traffic, East-West

    Trafficは、 影響を受けない構成 Design policy for multi-AZs network
  18. Backbone network after multi-AZs CLOS CLOS CLOS Tokyo DC-3 AZ3

    Tokyo DC-2 AZ2 Tokyo Region Tokyo DC-1 AZ1 CLOS CLOS The Internet The Internet The Internet Osaka DC Singapore DC
  19. Backbone network after multi-AZs CLOS CLOS CLOS Tokyo DC-3 AZ3

    Tokyo DC-2 AZ2 Tokyo Region Tokyo DC-1 AZ1 CLOS CLOS The Internet The Internet The Internet l 関東地⽅に新たに2つのDCを構築し、3つのDCを三⾓形で接続 DC間を接続する専⽤線は、複数社から異経路で調達 l 関東地⽅の3つのDCをAZ1〜AZ3と定義 Osaka DC Singapore DC
  20. Backbone network after multi-AZs CLOS CLOS CLOS Tokyo DC-3 AZ3

    Tokyo DC-2 AZ2 Tokyo Region Tokyo DC-1 AZ1 CLOS CLOS The Internet The Internet The Internet Osaka DC Singapore DC l Tokyo DC-1の⼀部サーバルームを、Legacy→CLOSへリノベーション NFVのmulti-AZs対応のためにCLOS構成が必要であるため。 ※Legacy構成のサーバールーム(non-AZサーバールーム)も引き続き運⽤中
  21. Backbone network after multi-AZs CLOS CLOS CLOS Tokyo DC-3 AZ3

    Tokyo DC-2 AZ2 Tokyo Region Tokyo DC-1 AZ1 CLOS CLOS The Internet The Internet The Internet l NW-POP RouterをBackbone Routerに接続 l Backbone Networkで、Internet Traffic⽤VRFを新規に作成 l Internet Traffic⽤のRoutingを是正 Osaka DC Singapore DC
  22. Routing for Internet Traffic CLOS CLOS CLOS Tokyo DC-3 AZ3

    Tokyo DC-2 AZ2 Tokyo Region Tokyo DC-1 AZ1 CLOS CLOS The Internet The Internet The Internet VRF for Internet Traffic Tokyo & Osaka Routes Tokyo & Osaka Routes Tokyo & Osaka Routes Tokyo & Osaka Routes SG Routes SG Routes Osaka DC Singapore DC
  23. Routing for Internet Traffic CLOS CLOS CLOS Tokyo DC-3 AZ3

    Tokyo DC-2 AZ2 Tokyo Region Tokyo DC-1 AZ1 CLOS CLOS The Internet The Internet The Internet VRF for Internet Traffic Full routes Default route Full routes Default route Default route Full Routes Osaka DC Singapore DC
  24. Internet Traffic flow CLOS CLOS CLOS Tokyo DC-3 AZ3 Tokyo

    DC-2 AZ2 Tokyo Region Tokyo DC-1 AZ1 CLOS CLOS The Internet The Internet The Internet VRF for Internet Traffic l 平常時は、IGP COSTに従い東京NW-POPを利⽤ Osaka DC Singapore DC
  25. Internet Traffic flow CLOS CLOS CLOS Tokyo DC-3 AZ3 Tokyo

    DC-2 AZ2 Tokyo Region Tokyo DC-1 AZ1 CLOS CLOS The Internet The Internet The Internet VRF for Internet Traffic Osaka DC Singapore DC
  26. Internet Traffic flow CLOS CLOS CLOS Tokyo DC-3 AZ3 Tokyo

    DC-2 AZ2 Tokyo Region Tokyo DC-1 AZ1 CLOS CLOS The Internet The Internet The Internet VRF for Internet Traffic Osaka DC Singapore DC
  27. Internet Traffic flow CLOS CLOS CLOS Tokyo DC-3 AZ3 Tokyo

    DC-2 AZ2 Tokyo Region Tokyo DC-1 AZ1 CLOS CLOS The Internet The Internet The Internet l 東京NW-POP障害時は、⼤阪NW-POPを利⽤ VRF for Internet Traffic Osaka DC Singapore DC
  28. Osaka NW-POPを利⽤ Tokyo Region内に新規 にNW-POPを構築 Pros Cons • 既存のネットワーク機器を流用で きる

    • 東日本ユーザ向けのlatencyの増加 がない • 東日本ユーザ向けのlatencyが増加 する • ネットワーク機器を新規に調達す る必要がある • 関東圏内での冗長では、NW-POP 冗長として不十分ではないか? NW-POP Redundancy
  29. Routing for Inter-DC Traffic CLOS CLOS CLOS Tokyo DC-3 AZ3

    Tokyo DC-2 AZ2 Tokyo Region Tokyo DC-1 AZ1 CLOS CLOS The Internet The Internet The Internet Osaka routes AZ1 routes SG routes AZ3 routes AZ2 routes VRF for Inter-DC Traffic Osaka DC Singapore DC
  30. Routing for Inter-DC Traffic CLOS CLOS CLOS Tokyo DC-3 AZ3

    Tokyo DC-2 AZ2 Tokyo Region Tokyo DC-1 AZ1 CLOS CLOS The Internet The Internet The Internet Other DC routes Other DC routes Other DC routes Other DC routes Other DC routes VRF for Inter-DC Traffic Osaka DC Singapore DC
  31. Inter-DC Traffic flow CLOS CLOS CLOS Tokyo DC-3 AZ3 Tokyo

    DC-2 AZ2 Tokyo Region Tokyo DC-1 AZ1 CLOS CLOS The Internet The Internet The Internet VRF for Inter-DC Traffic Osaka DC Singapore DC
  32. Inter-DC Traffic flow CLOS CLOS CLOS Tokyo DC-3 AZ3 Tokyo

    DC-2 AZ2 Tokyo Region Tokyo DC-1 AZ1 CLOS CLOS The Internet The Internet The Internet VRF for Inter-DC Traffic Osaka DC Singapore DC
  33. Inter-DC Traffic flow CLOS CLOS CLOS Tokyo DC-3 AZ3 Tokyo

    DC-2 AZ2 Tokyo Region Tokyo DC-1 AZ1 CLOS CLOS The Internet The Internet The Internet VRF for Inter-DC Traffic Osaka DC Singapore DC
  34. l ⼤阪からインターネットへ流すと、東⽇本ユーザ向けのlatencyが増加 l オリジンが東⽇本にあるため、仕⽅がない l トランジットコスト増加 l ⼤阪でも、東京と同容量のトランジット確保が必要 →コミット値の上昇→トランジットコスト増加 l

    ピアリングで捌けるトラフィック量を少しでも増やしたい 課題
  35. l 関東地⽅に新たに2つのDCを構築し、それぞれのDCをAZとして定義 l 1つのAZで障害が発⽣しても、その他AZの Internet Traffic, Inter-DC Traffic, East-West Trafficは影響を受けない

    Backbone Network l NW-POP TokyoのBackupとして、NW-POP Osakaを利⽤ Conclusion
  36. THANK YOU