LINEのネットワーク / Network of LINE

Transcript

1. -*/&ͷωοτϫʔΫ
   :PIFJ ,BOFNBSV
   -*/&$PSQPSBUJPO
   
   

   View Slide

2. ࣗݾ঺հ
   ؙۚ ༸ฏ
   ωοτϫʔΫνʔϜ d
   
   σʔληϯλωοτϫʔΫͷӡ༻ɺϐΞϦϯάௐ੔
   ΦϑΟεແઢ-"/ͷߏஙɾӡ༻
   0QFO4UBDLϕʔεͷϓϥΠϕʔτΫϥ΢υ7FSEBͷ։ൃνʔϜ d
   
   -PBE#BMBODFSBTB4FSWJDF -#BB4
   ͷઃܭɾ։ൃɾߏஙɾӡ༻
   2019-02-06 2
   

   View Slide

3. "HFOEB
   -*/&ͷωοτϫʔΫͷ֓ཁ
   ࠷ۙͷऔΓ૊Έࣄྫ
   ϩʔυόϥϯαͷࣗࣾ։ൃ
   σʔληϯλωοτϫʔΫͷ࡮৽
   ࠓޙͷऔΓ૊Έ༧ఆ
   43WʹΑΔॊೈͳωοτϫʔΫͷ࣮ݱ
   2019-02-06 3
   

   View Slide

4. -*/&ͷωοτϫʔΫͷ֓ཁ
   2019-02-06 4
   

   View Slide

5. lωοτϫʔΫz
   2019-02-06
   https://syllabus.kyoto-su.ac.jp/syllabus/html/2018/3783.html
   /
   5
   

   View Slide

6. -*/&ʹ͓͚ΔlωοτϫʔΫz
   2019-02-06
   /$!D9
   %2D9
   <5B@
   &+
   -"%)/ NAT
   E6@
   :H'+
   ?@LAN
   IP",C5
   (#*/
   NIC
   TCP
   TLS
   + /
   
   +
   7G14
   =I3
   F>;0$!.A8
   /!-)
   DNS
   6
   

   View Slide

7. Datacenter Network
   Datacenter Network
   యܕతͳΦϯϓϨϛεωοτϫʔΫ
   2019-02-06
   Datacenter Interconnect
   Internet
   Internet
   North-South
   traffic
   East-West traffic East-West traffic
   North-South
   traffic
   7
   

   View Slide

8. ωοτϫʔΫͷಛ৭
   ಛ௃తͳ/PSUI4PVUIτϥϑΟοΫ
   ಛఆͷ஍Ҭ΍*41ʹภ͍ͬͯΔ
   ৗʹ5$1ίωΫγϣϯΛҡ࣋͢Δ
   Πϕϯτੑͷ͋ΔτϥϑΟοΫύλʔϯ
   ৗʹ߈ܸʹࡽ͞Ε͍ͯΔ
   ലେͳ&BTU8FTUτϥϑΟοΫ
   σʔλ෼ੳʹ൐͏αʔόؒτϥϑΟοΫ
   ϚΠΫϩαʔϏε
   2019-02-06 8
   

   View Slide

9. ωοτϫʔΫʹٻΊΒΕΔ͜ͱ
   ๲େͳτϥϑΟοΫΛॲཧͰ͖Δ͜ͱ
   ਺ඦ(CQTن໛ͷ/PSUI4PVUIτϥϑΟοΫ
   ਺5CQTن໛ͷ&BTU8FTUτϥϑΟοΫ
   ؆୯͔ͭਝ଎ʹεέʔϧͰ͖Δ͜ͱ
   ͢͹΍͍ࣄۀల։ʹ଱͑͏Δߏ੒
   ҆ఆͨ͠ӡ༻͕Ͱ͖Δ͜ͱ
   ͋ΒΏΔ΋ͷ͸յΕΔ
   2019-02-06 9
   

   View Slide

10. Πϯϑϥͷ՝୊ղܾʹର͢Δߟ͑ํ
    ࠜຊ͔Βղܾ͢Δ͜ͱΛࢼΈΔ
    ର঱ྍ๏తͳ΋ͷΑΓ͸ɺΞʔΩςΫνϟ͔Βݟ௚͢
    ՄೳͳݶΓγϯϓϧʹอͭ
    ࠷খݶͷߏ੒ཁૉ΍Φʔϓϯͳϓϩτίϧ
    εςʔτϑϧΑΓ͸εςʔτϨεߏ੒
    ద੾ͳλΠϛϯάͰ౤ೖ͢Δ
    Πϯϑϥߏ੒ཁૉͷϥΠϑαΠΫϧ͸ԟʑʹͯ͠௕͍
    2019-02-06 10
    

    View Slide

11. ࠷ۙͷऔΓ૊Έࣄྫ
    ϩʔυόϥϯαͷࣗࣾ։ൃ
    ΞʔΩςΫνϟ͔Βࠜຊతʹมߋ
    ࠾༻ࣄྫͷແ͍ٕज़Ͱ͋ͬͯ΋ඞཁͰ͋Ε͹બ୒
    ͚͓͋ΊτϥϑΟοΫʹ଱͑ΔύϑΥʔϚϯεΛ࣮ݱ
    σʔληϯλωοτϫʔΫͷ׬શ-Խ
    ΞʔΩςΫνϟ͔Βࠜຊతʹมߋ
    ωοτϫʔΫɾαʔόͷ૒ํΛแׅతʹઃܭ
    εέʔϥϏϦςΟͱӡ༻ίετܰݮΛཱ྆
    2019-02-06 11
    

    View Slide

12. ࣄྫ ϩʔυόϥϯα։ൃ
    2019-02-06 12
    

    View Slide

13. -#ͷ໾ׂ
    7JSUVBM*1 7*1
    Ѽͷ௨৴ΛԿΒ͔ͷϙϦγʔʹै࣮ͬͯαʔόʹసૹ
    ෛՙ෼ࢄ΍৑௕ੑ֬อ͕ओͳ༻్
    *1ϒϥοΫϦετ΍5-4ऴ୺ͱ͍ͬͨ෇ՃػೳΛ࣋ͭ΋ͷ΋
    2019-02-06
    Real Server (RS)
    LB
    Client
    VIP RSIP
    i
    • RS
    
    • IP
    • TLS (TLS Offload)
    • …
    13
    

    View Slide

14. ΦϯϓϨϛε؀ڥͰͷయܕతͳ-#αʔϏεߏ੒
    2019-02-06
    OpenStack Octavia
    
    
    HAProxy
    LVS (IPVS)
    LBaaS Frontend LBaaS Backend
    
    
    LB
    
    API / Web UI API
    API / Web UI / CLI
    14
    

    View Slide

15. -#ͷมભͱ௚໘ͨ͠՝୊
    ॳظͷஈ֊ΑΓϋʔυ΢ΣΞΞϓϥΠΞϯε੡඼Λ࢖༻
    εςʔτϑϧʹಈ࡞͢Δ୆ηοτߏ੒
    εςʔτΛอଘ͢Δςʔϒϧ͕࣌ંރׇ͢ΔΑ͏ʹ
    ಉ࣌઀ଓ਺͕ଟ͍ͷʹՃ͑ɺಈ࡞ํࣜݻ༗ͷ໰୊΋ൃੜ
    -#୆਺͕૿͑ΔʹͭΕίετύϑΥʔϚϯεѱԽ
    ඞཁͳΩϟύγςΟͷഒΛ༻ҙ͢Δඞཁ͕͋Δ
    2019-02-06 15
    

    View Slide

16. / .VMUJUJFSߏ੒ 4UBUFMFTTԽ
    (PPHMF 'BDFCPPLͳͲͰ࠾༻͞Ε͍ͯΔΞʔΩςΫνϟɾํࣜ <>
    2019-02-06
    /.VMUJUJFS
    Client
    L4LB
    (Hardware)
    Real Server
    : In service
    : Not in service
    L4LB
    (Software)
    Real Server
    L7LB
    (Software)
    L3 Switch
    (Hardware)
    Client
    Stateful L3DSR
    TCP/HTTP(S) Proxy
    Stateless L3DSR
    ECMP
    [1]
    http://yunazuno.hatenablog.com/entry/2016/02/29/090001
    16
    

    View Slide

17. / .VMUJUJFS
    /ϝϯςφϯε༰қԽɺো֐υϝΠϯͷ࠷খԽ
    .VMUJUJFSಛఆͷॲཧʹूதͤ͞Δ͜ͱͰεέʔϧΞ΢τΛ༰қԽ
    2019-02-06
    L3 Switch L4LB L7LB
    Whitebox Switch
    100G x 32 ports
    JPY 2,000,000
    x86 Linux Server
    25G x 1 ports
    JPY 800,000
    x86 Linux Server
    25G x 1 ports
    JPY 800,000
    
    
    
    
    
    
    
    17
    

    View Slide

18. L4LB Program
    -*/&ͷ-#։ൃͱύέοτॲཧͷؔ܎
    --#ͷύέοτॲཧ෦෼͸ιϑτ΢ΣΞ 9%1
    Ͱ࣮૷
    2019-02-06
    Rx
    Tx
    Match
    Lookup
    Rewrite
    Table
    VIP-Backend
    table
    Statistics
    table
    From
    L3 Switch
    To L7 LB
    NIC
    1. Match
    Dst. IP/Port
    
    
    2. Lookup
    Hash
    
    Real Server
    
    
    3. Rewrite
    IP
    
    
    Dst. IP, DSCP
    
    
    
    18
    

    View Slide

19. 9%1͸ͳͥߴ଎ͳύέοτॲཧ͕Մೳ͔
    ௨ৗͷந৅Խ͞ΕͨωοτϫʔΫελοΫʹύέοτ͕౉Δલʹॲཧ
    %1%,΍OFUNBQͱ͍ͬͨྨࣅٕज़Ͱ͋ͬͯ΋جຊతͳߟ͑ํ͸ಉ͡
    2019-02-06
    
    XDP
    NIC
    Kernel
    Kernel
    User
    NIC
    Driver
    Ethernet
    IPv4
    TCP
    App Socket
    NIC
    Driver
    XDP
    App
    Ether
    net
    IPv4
    TCP
    App Socket
    19
    

    View Slide

20. ࣄྫ σʔληϯλωοτϫʔΫ
    2019-02-06 20
    

    View Slide

21. σʔληϯλωοτϫʔΫͷมભͱ՝୊
    ϨΠϠϕʔεͷωοτϫʔΫ
    ن໛֦େʹͭΕෆ҆ఆԽ
    Λલఏͱͨ͠ߏ੒ཁૉ
    ΩϟύγςΟͷෆ଍
    ߴ͍ӡ༻ෛՙ
    ಛఆϕϯμʹറΒΕΔ
    2019-02-06 21
    

    View Slide

22. ղܾࡦ -MFTT #(1 $-04
    ωοτϫʔΫػثɾαʔό૒ํ͕-ϧʔςΟϯά #(1
    Λ࢖༻
    /ߏ੒ʹΑΓεέʔϥϏϦςΟͱΩϟύγςΟΛ֬อ
    2019-02-06
    
    
    7,200 servers / POD
    7,200 servers / POD
    100G x 720 links
    10G x 14,400 links
    100G x 80 links
    100G x 1,440 links
    72,000Gbps Capacity
    L3
    22
    

    View Slide

23. 2019-02-06 23
    

    View Slide

24. ࠓޙͷऔΓ૊Έ 43W
    2019-02-06 24
    

    View Slide

25. γϯϓϧͳ-ωοτϫʔΫ্ͰॊೈͳॲཧΛ
    ৽ཁ݅ϚϧνςφϯγʔɺαʔϏενΣΠχϯά
    -ͷٕज़ 79-"/
    Ͱ࣮ݱ͢Δͷ͕௨ྫ
    ʜ͕ɺ-MFTTͷԸܙΛாফ͠ʹ͢Δଆ໘΋
    ొ৔ͨ͠͹͔Γͷ৽ͨͳٕज़Ͱ΋ݕ౼
    43W*1W4FHNFOU3PVUJOH
    ݱࡏݕূΛ࣮ࢪத
    2019-02-06
    SRv6 Underlay
    Tenant A
    Tenant B
    Tenant C
    25
    

    View Slide

26. ·ͱΊ
    2019-02-06 26
    

    View Slide

27. جຊΛॏࢹͭͭ͠՝୊Λղܾ
    ͞·͟·ͳٕज़త՝୊Λͻͱͭͻͱͭղܾ
    σʔληϯλωοτϫʔΫɺϩʔυόϥϯαɺʜ
    ϕʔεͱͳΔߟ͑ํΛॏࢹ͠ͳ͕ΒखஈΛݕ౼
    ৽͍͠ɾݹ͍ɺͰ͸ͳ͘ɺద੾͔Ͳ͏͔Λॏࢹ
    ࠓޙ΋৽ͨͳ෼໺΁ͷ௅ઓΛܧଓ
    ίϯςφωοτϫʔΩϯά΋ͦͷͻͱͭ
    2019-02-06 27
    

    View Slide