Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Kamuee: A Pure-Software-Approach Router

Kamuee: A Pure-Software-Approach Router

APRICOT2020で発表した「Kamuee: A Pure-Software-Approach Router」の講演資料です. 講演詳細についてはこちらをご覧ください.(https://2020.apricot.net/program/presentations/#kamuee-a-pure-software-approach-router)

NTT Communications

December 05, 2023
Tweet

More Decks by NTT Communications

Other Decks in Technology

Transcript

  1. 9 _haXVWc k CII 9 d XRPcX b 9 a_ aPcX ' 7 aXVWcb aTbTaeTS'
    BB NB 9LCPS NB NL E
    8L PBN
    CII 9 d XRPcX b 9 a_ aPcX
    ITRW Vh :TeT _ T c :XeXbX
    NPbdWXa DWPaP

    View full-size slide

  2. 9 _haXVWc k CII 9 d XRPcX b 9 a_ aPcX ' 7 aXVWcb aTbTaeTS'
    n :E:@ QPbTS b UcfPaT a dcTa TgcT STS Ua ,UfS
    n Ea _aXTcPah b daRT R ST c _T b daRT(UaTT b Uc
    n EdaT b UcfPaT P__a PRW aT dXaTb STSXRPcTS WPaSfPaT
    • 9E R aT X cT bXeT
    n >XVW _TaU a P RT a dcX V d_ P V aXcW 3 E _caXT
    n d 8=E a dcX V cPQ T RP_PQX Xch P__a g' +B a dcTb
    n D_T cTRW Vh UP X XPaXchpFdPVVP( GGq
    n >XVW _TaU a P RT ,/)=Q_b 6. +8% +))=Q_b ,))B__b
    6/-8
    n :TbXV _ XRh3 HRP T dc PaVT caPUUXR P V P h 9E
    R aTb'
    9 NU
    +

    View full-size slide

  3. 9 _haXVWc k CII 9 d XRPcX b 9 a_ aPcX ' 7 aXVWcb aTbTaeTS'
    1 / T BNCLN K B BK E N
    Spirent
    Kamuee
    (DUT)
    100G
    GP S STbcX PcX _PR Tcb
    p2)=Q_b g - 4 ,/)=Q_b X YTRcX q
    sorted packets
    measure
    received
    612,927
    routes
    table
    ,

    View full-size slide

  4. 9 _haXVWc k CII 9 d XRPcX b 9 a_ aPcX ' 7 aXVWcb aTbTaeTS'
    BNCLN K B U AAFK LNBO
    0
    100
    200
    300
    400
    500
    600
    1 2 3 4 5 6 7 8 9 10 11 12
    Throughput (Gbps)
    Cores per Port
    Skylake Platinum x 2 100GbE x 4 Performance
    360Gbps load, 612,927 routes
    360Gbps
    1518B
    512B
    256B
    128B
    64B
    222.11Gbps
    330.52Mpps
    -

    View full-size slide

  5. 9 _haXVWc k CII 9 d XRPcX b 9 a_ aPcX ' 7 aXVWcb aTbTaeTS'
    8BOL N B LKPNLI U KBBAO
    0
    100
    200
    300
    400
    500
    600
    64 128 256 512 1024 1518
    Throughput (Gbps)
    Ethernet Frame Size (Bytes)
    Skylake Platinum x 2 100GbE x 4 Performance
    360Gbps load, 612,927 routes
    360Gbps
    12cpp
    8cpp
    4cpp
    1cpp
    222.11Gbps
    330.52Mpps
    .

    View full-size slide

  6. 9 _haXVWc k CII 9 d XRPcX b 9 a_ aPcX ' 7 aXVWcb aTbTaeTS'
    2KPBNK I OPN P NB LC PEB OLCPS NB NL PBN BB
    kernel
    bgpd zebra
    NIC NIC NIC NIC
    NIC NIC
    CPU socket CPU socket
    0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19
    master forwarder
    lthread
    scheduler
    D P D K
    FIB
    RIB
    FIB
    RIB manager
    tap manager
    arp manager
    vty-server
    tap device
    arp
    netlink manager
    netlink socket
    scale-out
    RCU(liburcu)
    O
    S
    vtys
    acl manager
    routing table
    Scale-out among CPU Cores
    run-to-completion
    OSS

    View full-size slide

  7. 9 _haXVWc k CII 9 d XRPcX b 9 a_ aPcX ' 7 aXVWcb aTbTaeTS'
    n d h Ee/ RP_PQ T% Ud 8=E a dcX V RP_PQ T a dcTa
    Qh P E9 bTaeTa'
    n Ee-( Ee/ 8=E d cXW T
    • Ud a dcTb3 Ee-3 1))@( Ee/ 2)@
    n E9 bTaeTa
    • + cT MT 9E . +/+) ( ,+=8 BT ah ( + )=Q _ acb cT
    M0 ) H E
    n H UcfPaT
    • Qd cd 1')-'+ ( :E:@ 1' ( @P dTT ( GG
    OB OB -1 1 PBS U CLN KFRBNOFPU
    0

    View full-size slide

  8. 9 _haXVWc k CII 9 d XRPcX b 9 a_ aPcX ' 7 aXVWcb aTbTaeTS'
    Use Case 2: Multicast
    Router for events
    • PIM-SM capability using FRR-7.2 pimd
    • 100GbE x 4 NIC ports
    • 8K (Dual Green) x 2 or Full 8K: 51.2Gbps
    • Supermicro XXX
    • Intel Scalable Platinum 8180 x 2
    • Mellanox ConnectX-5 VPI x 4
    8

    View full-size slide

  9. 9 _haXVWc k CII 9 d XRPcX b 9 a_ aPcX ' 7 aXVWcb aTbTaeTS'
    n +)+) +(- IdT j +( IdT DS aX HXcT
    n HcaTP X V3 +(- j +(0
    9 LNL 9KLS 0BOPFR I
    2

    View full-size slide

  10. 9 _haXVWc k CII 9 d XRPcX b 9 a_ aPcX ' 7 aXVWcb aTbTaeTS'
    NPF F KPO ) 6N KFV PFLKO
    • Organized by
    • National Institute of Information and Communications Technology (NICT)
    • Cooperating Organizations (in no particular order)
    • Hokkaido Television Broadcasting Co., Ltd. (HTB)
    • GAORA Inc.
    • Mainichi Broadcasting System
    • NTT Communications Corporation
    • NTT TechnoCross Corporation
    • Nippon Telegraph and Telephone West Corporation
    • KDDI CORPORATION
    • Hokkaido Telecommunication Network Co., Inc.(HOTnet)
    • KYOWA EXEO CORPORATION
    • OBIS, Inc. (OBIS)
    • Trust Communication Inc.
    • Fatware Inc.
    • Arista Networks Japan Limited.
    • FXC Inc.
    • Keysight Technologies Inc.
    • Cisco Systems G.K
    • Juniper Networks, Inc.
    • Palo Alto Networks
    • Fortinet, Inc.
    • NEC Corporation
    • NEC Networks & System Integration Corporation
    • Seiko Solutions Inc.
    • Tohoku Infomation Systems Company, Incorporated
    • Furukawa Network Solution Corp.
    • SHARP Corporation
    • ASTRODESIGN, Inc.
    • Alpha Code Inc.
    • IKEGAMI TSUSHINKI CO.,LTD.
    • VILLAGE island Co., Ltd.
    • AIM Electronics Co., Ltd.
    • Embrionix Design Inc.
    • Keisoku Giken Co., Ltd.
    • Servants International Corporation
    • T-SPOTs Co., Ltd.
    • Techno House Inc.
    • Digital Research Inc.
    • PacketLight Networks Ltd.
    • Hibino Corporation
    • Purelogic Co.,Ltd.
    • FUJIFILM Corporation
    • Micro Research Ltd.(MRL)
    • Miharu Communications Inc.
    • Kanagawa Institute of Technology
    • Kyoto Sangyo University
    • Keio University
    • Daido University
    • The University of Tokyo
    • Nara Institute of Science and Technology
    • National Institute of Informatics
    • Information Technology Promotion Agency
    • Nago City, Okinawa Pref.
    • Union-wide Municipal Affairs of Northern Area of Okinawa Prefecture
    • Specified NPO NDA
    • Knowledge Capital Association
    • VisLab OSAKA
    • Cyber Kansai Project(CKP)
    • WIDE Project

    View full-size slide

  11. 9 _haXVWc k CII 9 d XRPcX b 9 a_ aPcX ' 7 aXVWcb aTbTaeTS'
    LPB
    T P
    LPB
    T P
    KLPB
    T P
    KALGF
    T P
    Tc )( () Tc )( ()
    Tc )( ()
    Tc )( (
    Tc )( (,
    EL NF
    T ) P
    Tc )( (+
    KLPB ON
    Wd )()()()
    Wd )()()(
    925/:
    Wd )()()(+
    9P N BA Tc ( ()
    vux kamuee–y o
    →IPAmrwst/StarBed→rwst
    Tc )()()
    0
    -0
    )
    Tc )( (0
    kamuee



    Tc ,(,()
    Tc )( ()
    Tc )( (/
    Wd -2(
    Wd .)(
    Wd . ( Wd .+(
    e P -)
    e P +-
    e P ,2
    B FP
    T
    4/
    4/
    8
    4
    rwst9F
    0
    NTT
    KDDI
    W b
    de
    4/
    LPB T)(
    .(
    2 2(
    CDABG ) E 44 E
    O LNL
    T ) P
    4/
    Hotnet
    rwst2F
    B FP
    19

    dec
    dec
    )=l /
    / NFLKF
    T
    ))=
    ))=
    ))=
    ))=
    ))=



    )=



    )=
    #$!
    "
    50
    af50
    2
    :
    ( G ) E 44 E
    ,(
    cost100
    cost100
    cost100
    cost200

    View full-size slide

  12. 9 _haXVWc k CII 9 d XRPcX b 9 a_ aPcX ' 7 aXVWcb aTbTaeTS'
    vux
    mx480
    PIM-R
    kote mx2010
    PIM-R
    kote mx10k
    PIM-R
    (Lsys A16016)
    StarBED
    o
    RP:192.168.255.2
    vlan40
    192.168.166.0/24
    .2 .10
    PIM
    PIM
    PIM
    PIM-R
    (Lsys A19004)
    z (L)
    z (R)
    PIM
    IGMP
    PIM
    PIM-R
    (Lsys A19004)
    sapporo-mx480-t
    PIM-R
    (Lsys A16016)
    kamuee
    PIM-R
    IPA
    IGMP
    IGMP
    PIM
    vlan39
    192.168.165.0/24
    vlan39
    192.168.169.0/24
    vlan35
    192.168.207.0/24
    vlan3540
    192.168.212.0/24
    vlan35
    192.168.211.0/24
    vlan3549
    192.168.49.0/24
    vlan3544
    192.168.44.0/24
    IGMP
    vlan3548
    192.168.48.0/24
    vlan3546
    192.168.46.0/24
    vlan3547
    192.168.47.0/24
    vlan35
    192.168.201.0/24
    NAIST
    vlan3634
    192.168.34.0/24
    IGMP
    vlan35
    192.168.210.0/24
    vlan3541(L2)
    vlan3541(L2)
    vlan3542
    192.168.42.0/24
    IGMP
    A16016nKAIT PJ
    A19004nkamuee PJ
    PIM
    HTB
    IGM
    P
    vlan3543
    192.168.43.0/24
    IGMP
    vlan1118
    192.168.118.0/24
    mx10k
    PIM-R
    (Lsys A16016)
    rwst
    vlan3545
    192.168.45.0/24
    IGMP

    View full-size slide

  13. 9 _haXVWc k CII 9 d XRPcX b 9 a_ aPcX ' 7 aXVWcb aTbTaeTS'
    n eXbdP h
    bb Tbb
    n bXV U
    _PR Tc bb
    n . %+2-'+0
    BQ_b
    02+' 0
    @__b
    ) C II OFVB IPF OP OPNB
    ,
    0
    10000
    20000
    30000
    40000
    50000
    60000
    70000
    Feb 3 0am Feb 4 0am Feb 5 0am Feb 6 0am Feb 7 0am
    Bandwidth (Mbps)
    Port-0: In: Upstream: Sapporo/Odori
    0
    10000
    20000
    30000
    40000
    50000
    60000
    70000
    Feb 3 0am Feb 4 0am Feb 5 0am Feb 6 0am Feb 7 0am
    Bandwidth (Mbps)
    Port-4: Out: Downstream: Osaka/Umekita
    0
    10000
    20000
    30000
    40000
    50000
    60000
    70000
    Feb 3 0am Feb 4 0am Feb 5 0am Feb 6 0am Feb 7 0am
    Bandwidth (Mbps)
    Port-6: Out: Downstream: Akihabara/UDX

    View full-size slide

  14. 9 _haXVWc k CII 9 d XRPcX b 9 a_ aPcX ' 7 aXVWcb aTbTaeTS' -
    c _ /3.+3++ d_ -. SPhb% /3 -% 0 dbTab% PS PeTaPVT3 .+' /% .+' -% .+' )
    IPb b3 /1 c cP % + ad X V% , ) b TT_X V% ) bc __TS% ) i QXT
    9_d) 3 ))') db% )') bh% )') X% )') XS% )') fP% )') WX% )') bX% )') bc
    9_d 3 )') db% )', bh% )') X% 22'0 XS% )') fP% )') WX% )') bX% )') bc
    9_d+ 3 )') db% )') bh% )') X% ))') XS% )') fP% )') WX% )') bX% )') bc
    9_d, 3 .-'1 db% -+'2 bh% )') X% +', XS% )') fP% )') WX% )') bX% )') bc
    9_d- 3 ))') db% )') bh% )') X% )') XS% )') fP% )') WX% )') bX% )') bc
    9_d. 3 ))') db% )') bh% )') X% )') XS% )') fP% )') WX% )') bX% )') bc
    9_d/ 3 ))') db% )') bh% )') X% )') XS% )') fP% )') WX% )') bX% )') bc
    9_d0 3 ))') db% )') bh% )') X% )') XS% )') fP% )') WX% )') bX% )') bc
    9_d1 3 ))') db% )') bh% )') X% )') XS% )') fP% )') WX% )') bX% )') bc
    9_d2 3 ))') db% )') bh% )') X% )') XS% )') fP% )') WX% )') bX% )') bc
    9_d ) 3 ))') db% )') bh% )') X% )') XS% )') fP% )') WX% )') bX% )') bc
    9_d 3 ))') db% )') bh% )') X% )') XS% )') fP% )') WX% )') bX% )') bc
    9_d + 3 ))') db% )') bh% )') X% )') XS% )') fP% )') WX% )') bX% )') bc
    9_d , 3 ))') db% )') bh% )') X% )') XS% )') fP% )') WX% )') bX% )') bc
    9_d - 3 ))') db% )') bh% )') X% )') XS% )') fP% )') WX% )') bX% )') bc
    9_d . 3 ))') db% )') bh% )') X% )') XS% )') fP% )') WX% )') bX% )') bc
    9_d / 3 ))') db% )') bh% )') X% )') XS% )') fP% )') WX% )') bX% )') bc
    9_d 0 3 ))') db% )') bh% )') X% )') XS% )') fP% )') WX% )') bX% )') bc
    9_d 1 3 ))') db% )') bh% )') X% )') XS% )') fP% )') WX% )') bX% )') bc
    9_d 2 3 ))') db% )') bh% )') X% )') XS% )') fP% )') WX% )') bX% )') bc
    9_d+) 3 ))') db% )') bh% )') X% )') XS% )') fP% )') WX% )') bX% )') bc
    9_d+ 3 ))') db% )') bh% )') X% )') XS% )') fP% )') WX% )') bX% )') bc
    9_d++ 3 ))') db% )') bh% )') X% )') XS% )') fP% )') WX% )') bX% )') bc
    9_d+, 3 ))') db% )') bh% )') X% )') XS% )') fP% )') WX% )') bX% )') bc
    9_d+- 3 ))') db% )') bh% )') X% )') XS% )') fP% )') WX% )') bX% )') bc
    9_d+. 3 ))') db% )') bh% )') X% )') XS% )') fP% )') WX% )') bX% )') bc
    9_d+/ 3 ))') db% )') bh% )') X% )') XS% )') fP% )') WX% )') bX% )') bc
    9_d+0 3 ))') db% )') bh% )') X% )') XS% )') fP% )') WX% )') bX% )') bc
    9_d+1 3 ))') db% )') bh% )') X% )') XS% )') fP% )') WX% )') bX% )') bc
    9_d+2 3 ,1'2 db% / ' bh% )') X% )') XS% )') fP% )') WX% )') bX% )') bc
    9_d,) 3 ))') db% )') bh% )') X% )') XS% )') fP% )') WX% )') bX% )') bc
    9_d, 3 ))') db% )') bh% )') X% )') XS% )') fP% )') WX% )') bX% )') bc
    9_d,+ 3 ))') db% )') bh% )') X% )') XS% )') fP% )') WX% )') bX% )') bc
    9_d,, 3 ))') db% )') bh% )') X% )') XS% )') fP% )') WX% )') bX% )') bc
    9_d,- 3 ))') db% )') bh% )') X% )') XS% )') fP% )') WX% )') bX% )') bc

    View full-size slide

  15. 9 _haXVWc k CII 9 d XRPcX b 9 a_ aPcX ' 7 aXVWcb aTbTaeTS'
    n 9PeTPcb3 D_cX XiPcX TeT U a TPRW U :E:@ XQaPah P S
    P__ XRPcX ! UcT R UdbTS Qh cWT _cX XiPcX TeT
    U cWT R _X TS cX T'''
    n D_cX XiPcX Xb Th3 /B _(bTR U a jD) P S B U a jD,
    n fXcW P bX V T 1@ bcaTP fWT c QaP RWX V 3
    6 PF FV PFLK
    .
    core[4]: 11,920,554
    core[5]: 11,633,171
    core[6]: 11,100,561
    core[7]: 12,452,232
    core[8]: 12,573,650
    core[9]: 12,566,363
    core[10]: 11,834,071

    View full-size slide

  16. 9 _haXVWc k CII 9 d XRPcX b 9 a_ aPcX ' 7 aXVWcb aTbTaeTS'
    n Ea QPQ h fXcW aT cPb X cWT PX _
    n aTRTXeT QdUUTa Ud bTT TS c RRda3 Ud ,+ _ cb aTPS'
    n IWT _PR Tc bb bTT TS c RRda cWT aTRTXeT QdUUTa Ud
    cX X V'
    n X RaTPbT cWT aTRTXeT QdUUTa3 B7MOE@IO8 GHI3 ,+ 5 /-
    n _PR Tc bb RRda fWT aTRTXeT Xb Ud /-'
    n X RaTPbT cWT aTRTXeT QdUUTa3 B7MOE@IO8 GHI3 /- 5 . +
    n bcX acTOTcWOagOQdabc bTT b c aTcda /- Pc PgX d '
    n acTOTcWOagOQdabc PVPX d cX B7MOE@IO8 GHI'
    n _PR Tc bb WPb V T'
    0FTFK PEB BP ILOO
    /

    View full-size slide

  17. 9 _haXVWc k CII 9 d XRPcX b 9 a_ aPcX ' 7 aXVWcb aTbTaeTS'
    n D_cX XiPcX U a BAM.3 EDGIO:E:@OC BOIMO: H9 . +
    5 -)2/
    n IWT cT M..)!b SaXeTa S Tb c bTT c UaTT QdU Pc P
    V S aPcT'
    n HT U aXVX PcTS _PR Tcb Ua W bc T'V'% 7GE fTaT
    P RPcTS Ua cWT bT_PaPcTS T _ bXiT -)2/ '
    n cWT 7GE T _ P fPhb TgWPdbcTS' T TP bXcdPcX
    n b_TRd PcX 3 cT C 9 SaXeTa UaTTb P d QTa U QdU X
    _a _ acX c cWT IMO: H9 bXiT' :E:@ S R bPhb b c
    n bTc IMO: H9 QPR c . + U a Q cW cT (BAM RPbTb'
    n aTb daRT QP P RT'
    0FTFK B LNU IB
    0

    View full-size slide

  18. 9 _haXVWc k CII 9 d XRPcX b 9 a_ aPcX ' 7 aXVWcb aTbTaeTS'
    n cWP b c G9 ' EPR Tcb PaT TXcWTa a dcTS P V fXcW cWT
    S a Tf 8 cPQ T' CD RPbT fT bT _PR Tcb X QTcfTT
    cWT S P S Tf 8'
    n LT f Xc h PUcTa fT R d S T X X PcT cWT _PR Tc bb
    U d f RPdbT '
    KL BP ILOO OBA U NL PB E K BO
    1

    View full-size slide

  19. 9 _haXVWc k CII 9 d XRPcX b 9 a_ aPcX ' 7 aXVWcb aTbTaeTS'
    n c!b f a X V HTT X V c P T P aTP QdbX Tbb dc U cWXb
    b UcfPaT a dcTa'
    n LT _a eTS
    • :E:@ b UcfPaT a dcTa RP _TaU a bXV XUXRP c h WXVW h'
    • c R d S QT TgcT STS U a cWT _aPRcXRP dbT3 8=E R aT
    a dcTa ( Bd cXRPbc a dcTa'
    n Ea b_TRcb3
    • C K ( 9 dS a dcTa
    • E PcU a U a b UcfPaT STeT _ T c
    • XSS TQ gTb bdRW Pb A8( L(:E (KEC
    • RP .=b% HGe/
    N
    +-

    View full-size slide