Upgrade to Pro — share decks privately, control downloads, hide ads and more …

10GbE時代のネットワークI/O高速化

 10GbE時代のネットワークI/O高速化

Takuya ASADA

June 07, 2013
Tweet

More Decks by Takuya ASADA

Other Decks in Technology

Transcript

  1. 10GbE࣌୅ͷωοτ
    ϫʔΫI/Oߴ଎Խ
    Takuya ASADA

    View full-size slide

  2. Slide URL
    •http://slidesha.re/16OV9Yx

    View full-size slide

  3. ͸͡Ίʹ
    • 10GbEɺ40GbEͳͲͷۃΊͯߴ଎ͳ௨৴Λ
    αϙʔτ͢ΔNIC͕ɺPCαʔόͷྖҬͰ
    ΋࢖ΘΕΔΑ͏ʹͳ͖͍ͬͯͯΔ

    • ͜ͷΑ͏ͳ଎౓ͷ௨৴Λιϑτ΢ΣΞ
    ʢOSʣͰॲཧ͠ߴ͍ੑೳΛಘΔʹ͸༷ʑ
    ͳো֐͕͋Γɺϋʔυ΢ΣΞɾιϑτ΢Σ
    Ξ྆໘ͷ࣮૷Λݟ௚͢ඞཁ͕͋Δ

    View full-size slide

  4. ࠓ೔ͷτϐοΫ
    1. ׂΓࠐΈ͕ଟ͗͢Δ

    2. ϓϩτίϧॲཧ͕ॏ͍

    3. ෳ਺ͷCPUͰύέοτॲཧ͍ͨ͠

    4. σʔλҠಈʹ൐͏ϨΠςϯγͷ࡟ݮ

    5. ϓϩτίϧελοΫΛܦ༝͠ͳ͍ωοτϫʔ
    ΫIO

    View full-size slide

  5. 1. ׂΓࠐΈ͕ଟ͗͢Δ
    Process(User)
    Process(Kernel)
    HW Intr Handler
    SW Intr Handler
    ύέοτड৴
    ϓϩτίϧॲཧ
    ιέοτ
    ड৴ॲཧ
    Ϣʔβ
    ϓϩάϥϜ
    VTFS
    CV⒎FS
    input
    queue
    socket
    queue
    ύέοτ
    γεςϜίʔϧ
    ϓϩηεىচ
    ιϑτ΢ΣΞׂΓࠐΈεέδϡʔϧ
    ϋʔυ΢ΣΞׂΓࠐΈ
    Ϣʔβۭؒ΁ίϐʔ

    View full-size slide

  6. ׂΓࠐΈ͕ଟ͗͢Δ
    • NICͷੑೳ޲্ʹΑͬͯɺҰఆ࣌ؒʹ
    NIC͕ॲཧͰ͖Δύέοτ਺͕ඈ༂తʹ
    ૿Ճ

    • ̍ύέοτຖʹׂΓࠐΈ͕དྷΔͱɺ௨
    ৴ྔ͕ଟ͍ͱ͖ʹίϯςΩετεΠο
    νճ਺͕૿͑͗͢ੑೳ͕ྼԽ

    View full-size slide

  7. چདྷͷύέοτड৴ॲཧ
    Process(User)
    Process(Kernel)
    HW Intr Handler
    SW Intr Handler
    ύέοτड৴
    ϓϩτίϧॲཧ
    ιέοτ
    ड৴ॲཧ
    Ϣʔβ
    ϓϩάϥϜ
    VTFS
    CV⒎FS
    input
    queue
    socket
    queue
    ύέοτ
    γεςϜίʔϧ
    ϓϩηεىচ
    ιϑτ΢ΣΞׂΓࠐΈεέδϡʔϧ
    ϋʔυ΢ΣΞׂΓࠐΈ
    Ϣʔβۭؒ΁ίϐʔ
    ϋʔυ΢ΣΞׂΓࠐΈ

    ˣ

    ड৴ΩϡʔʹΩϡʔ
    Πϯά

    ˣ

    ιϑτ΢ΣΞׂΓࠐ
    Έεέδϡʔϧ

    View full-size slide

  8. چདྷͷύέοτड৴ॲཧ
    • ̍ύέοτड৴͢ΔͨͼʹׂΓࠐΈΛ
    ड͚ͯॲཧΛߦ͍ͬͯΔ

    • 64byte frameͷ࠷େड৴Մೳ਺ɿ

    • GbEɿ໿1.5Mppsʢ150ສʣ

    • 10GbEɿ໿15Mppsʢ1500ສʣ

    View full-size slide

  9. ׂΓࠐΈΛແޮʹ͢Δʁ
    • ϙʔϦϯάํࣜ

    • NICͷׂΓࠐΈΛېࢭ͠ɺ୅ΘΓʹΫϩοΫׂΓࠐΈΛ༻
    ͍ͯఆظతʹड৴ΩϡʔΛνΣοΫ

    • σϝϦοτɿϨΠςϯγ্͕͕ΔɾఆظతʹCPUΛى͜͢
    ඞཁ͕͋Δ

    • ϋΠϒϦουํࣜ

    • ௨৴ྔ͕ଟ͘࿈ଓͯ͠ύέοτॲཧΛߦ͍ͬͯΔ࣌ͷΈׂ
    ΓࠐΈΛແޮԽͯ͠ϙʔϦϯάͰಈ࡞


    View full-size slide

  10. NAPIʢϋΠϒϦουํ
    ࣜʣ
    Process(User)
    Process(Kernel)
    HW Intr Handler
    SW Intr Handler
    ׂΓࠐΈແޮԽ
    ϓϩτίϧॲཧ
    ιέοτ
    ड৴ॲཧ
    Ϣʔβ
    ϓϩάϥϜ
    VTFS
    CV⒎FS
    socket
    queue
    ύέοτ
    γεςϜίʔϧ
    ϓϩηεىচ
    ϋʔυ΢ΣΞׂΓࠐΈ
    Ϣʔβۭؒ΁ίϐʔ
    ύέοτ
    ύέοτ
    ιϑτ΢ΣΞׂΓࠐΈεέδϡʔϧ
    ύέοτड৴
    ύέοτ͕ແ͘ͳΔ
    ·Ͱ܁Γฦ͠
    ϋʔυ΢ΣΞׂΓࠐΈ

    ˣ

    ׂΓࠐΈແޮԽˍ

    ϙʔϦϯά։࢝



    ύέοτ͕ແ͘ͳͬ
    ͨΒׂΓࠐΈ༗ޮԽ

    View full-size slide

  11. Interrupt Coalescing
    • NIC͕OSෛՙΛߟׂྀͯ͠ΓࠐΈΛؒ
    Ҿ͘

    • ύέοτ਺ݸʹҰճׂΓࠐΉɺ

    ͍҃͸Ұఆظؒ଴͔ͬͯΒׂΓࠐΉ

    • σϝϦοτɿϨΠςϯγ্͕͕Δ

    View full-size slide

  12. Interrupt CoalescingͷޮՌ
    • Intel 82599(ixgbe)ͰInterrupt Coalescingແޮɺ

    ༗ޮʢׂΓࠐΈස౓ࣗಈௐ੔ʣͰൺֱ

    • MultiQueue, GRO, LRO౳͸ແޮԽ

    • iperfͷTCPϞʔυͰܭଌ
    interrupts throughput packets CPU%(sy+si)
    ແޮ 46687 int/s 7.82 Gbps 660386 pkt/s 97.6%
    ༗ޮ 7994 int/s 8.24 Gbps 711132 pkt/s 79.6%

    View full-size slide

  13. Process(User)
    Process(Kernel)
    HW Intr Handler
    SW Intr Handler
    ׂΓࠐΈແޮԽ
    ϓϩτίϧॲཧ
    ιέοτ
    ड৴ॲཧ
    Ϣʔβ
    ϓϩάϥϜ
    VTFS
    CV⒎FS
    socket
    queue
    ύέοτ
    γεςϜίʔϧ
    ϓϩηεىচ
    ϋʔυ΢ΣΞׂΓࠐΈ
    Ϣʔβۭؒ΁ίϐʔ
    ύέοτ
    ύέοτ
    ιϑτ΢ΣΞׂΓࠐΈεέδϡʔϧ
    ύέοτड৴
    ύέοτ͕ແ͘ͳΔ
    ·Ͱ܁Γฦ͠
    2.ϓϩτίϧॲཧ͕ॏ͍

    View full-size slide

  14. ϓϩτίϧॲཧ͕ॏ͍
    • ಛʹখ͞ͳύέοτ͕େྔʹಧ͘৔߹ʹ
    ϓϩτίϧॲཧͰCPU࣌ؒΛେྔʹ࢖ͬͯ
    ͠·͏

    • ύέοτ਺෼ϓϩτίϧελοΫ͕ݺͼग़
    ͞ΕΔ

    ྫɿ64byte frameͷ৔߹

    ˠཧ࿦্ͷ࠷େ஋͸1500ສճ/s

    View full-size slide

  15. TOE
    (TCP Offload Engine)
    • OSͰϓϩτίϧॲཧ͢ΔͷΛ΍ΊͯɺNICͰॲཧ͢Δ

    • σϝϦοτ

    • ηΩϡϦςΟɿTOEʹηΩϡϦςΟϗʔϧ͕ੜͯ͡΋ɺOS
    ଆ͔Βରॲ͕ग़དྷͳ͍

    • ෳࡶੑɿOSͷωοτϫʔΫελοΫΛTOEͰஔ͖׵͑Δʹ
    ͸͔ͳΓ޿ൣғͷมߋ͕ඞཁ

    ϝʔΧʹΑͬͯTOEͷ࣮૷͕ҟͳΓڞ௨ΠϯλϑΣʔεఆ
    ͕ٛࠔ೉

    • Linuxɿαϙʔτ༧ఆແ͠

    View full-size slide

  16. Checksum Offloading

    • IPɾTCPɾUDP checksumͷܭࢉΛNICͰ
    ߦ͏

    View full-size slide

  17. Checksum Offloading
    ͷޮՌ
    • Intel 82599(ixgbe)Ͱൺֱ

    • iperfͷTCPϞʔυͰܭଌ

    • MultiQueue͸ແޮԽ

    • ethtool -K ix0 rx off
    throughput CPU%(sy+si)
    ແޮ 8.27 Gbps 86
    ༗ޮ 8.27 Gbps 85.2

    View full-size slide

  18. LRO
    (Large Receive Offload)
    • NIC͕ड৴ͨ͠TCPύέοτΛ݁߹͠ɺ

    େ͖ͳύέοτʹ͔ͯ͠ΒOS΁౉͢

    • ϓϩτίϧελοΫͷݺͼग़͠ճ਺Λ
    ࡟ݮ

    • LinuxͰ͸ιϑτ΢ΣΞʹΑΔLRO͕࣮
    ૷͞Ε͍ͯΔʢGROʣ

    View full-size slide

  19. LRO͕ແ͍৔߹
    • ύέοτຖʹωοτϫʔΫελοΫΛ
    ࣮ߦ
    seq 10000 seq 10001 seq 10002 seq 10003
    ←1500bytes→
    To network stack

    View full-size slide

  20. LRO͕༗Δ৔߹
    • ύέοτΛ݁߹͔ͯ͠ΒωοτϫʔΫελοΫΛ
    ࣮ߦɺωοτϫʔΫελοΫͷ࣮ߦճ਺Λ࡟ݮ
    seq 10000 seq 10001 seq 10002 seq 10003
    ←1500bytes→
    To network stack
    big one packet

    View full-size slide

  21. GROͷޮՌ
    • Intel 82599(ixgbe)Ͱൺֱ

    • MultiQueue͸ແޮԽ

    • iperfͷTCPϞʔυͰܭଌ

    • ethtool -K ix0 gro off
    packets network stack
    called count
    throughput CPU%(sy+si)
    ແޮ 632139 pkt/s 632139 call/s 7.30 Gbps 97.6%
    ༗ޮ 712387 pkt/s 47957 call/s 8.25 Gbps 79.6%

    View full-size slide

  22. TSO
    (TCP Segmentation Offload)
    • LROͷٯ

    • ύέοτΛϑϥάϝϯτԽͤͣʹૹ৴

    NIC͕ύέοτΛMTUαΠζʹ෼ׂ

    • OS͸ύέοτ෼ׂॲཧΛলུग़དྷΔ

    • LinuxͰ͸ιϑτ΢ΣΞʹΑΔGSOɺ

    ϋʔυ΢ΣΞʹΑΔTSOʗUFOΛαϙʔτ

    View full-size slide

  23. TSOͷޮՌ
    • Intel 82599(ixgbe)Ͱൺֱ

    • MultiQueue͸ແޮԽ

    • iperfͷTCPϞʔυͰܭଌ

    • ethtool -K ix0 gso off tso off
    packets throughput CPU%(sy+si)
    ແޮ 247794 pkt/s 2.87 Gbps 53.5%
    ༗ޮ 713127 pkt/s 8.16 Gbps 26.8%

    View full-size slide

  24. 3.ෳ਺ͷCPUͰύέοτॲཧ͍ͨ͠
    cpu0
    Process(User)
    Process(Kernel)
    HW Intr Handler
    SW Intr Handler
    ׂΓࠐΈແޮԽ
    ϓϩτίϧॲཧ
    ιέοτ
    ड৴ॲཧ
    Ϣʔβ
    ϓϩάϥϜ
    VTFS
    CV⒎FS
    socket
    queue
    ύέοτ
    γεςϜίʔϧ
    ϓϩηεىচ
    ϋʔυ΢ΣΞׂΓࠐΈ
    Ϣʔβۭؒ΁ίϐʔ
    ύέοτ
    ύέοτ
    ιϑτ΢ΣΞׂΓࠐΈεέδϡʔϧ
    ύέοτड৴
    ύέοτ͕ແ͘ͳΔ
    ·Ͱ܁Γฦ͠
    cpu1
    Process(User)
    Process(Kernel)
    HW Intr Handler
    SW Intr Handler
    ׂΓࠐΈແޮԽ
    ϓϩτίϧॲཧ
    ιέοτ
    ड৴ॲཧ
    Ϣʔβ
    ϓϩάϥϜ
    VTFS
    CV⒎FS
    socket
    queue
    ύέοτ
    γεςϜίʔϧ
    ϓϩηεىচ
    ϋʔυ΢ΣΞׂΓࠐΈ
    Ϣʔβۭؒ΁ίϐʔ
    ύέοτ
    ύέοτ
    ιϑτ΢ΣΞׂΓࠐΈεέδϡʔϧ
    ύέοτड৴
    ύέοτ͕ແ͘ͳΔ
    ·Ͱ܁Γฦ͠

    View full-size slide

  25. ιϑτׂΓࠐΈ͕
    ̍ͭͷίΞʹภΔ

    View full-size slide

  26. ιϑτׂΓࠐΈͱ͸ʁ
    Process(User)
    Process(Kernel)
    HW Intr Handler
    SW Intr Handler
    ׂΓࠐΈແޮԽ
    ϓϩτίϧॲཧ
    ιέοτ
    ड৴ॲཧ
    Ϣʔβ
    ϓϩάϥϜ
    VTFS
    CV⒎FS
    socket
    queue
    ύέοτ
    γεςϜίʔϧ
    ϓϩηεىচ
    ϋʔυ΢ΣΞׂΓࠐΈ
    Ϣʔβۭؒ΁ίϐʔ
    ύέοτ
    ύέοτ
    ιϑτ΢ΣΞׂΓࠐΈεέδϡʔϧ
    ύέοτड৴
    ύέοτ͕ແ͘ͳΔ
    ·Ͱ܁Γฦ͠
    ϙʔϦϯά͔Β

    ϓϩτίϧॲཧ·Ͱ

    →ωοτϫʔΫIOͷ
    େ൒෦෼

    View full-size slide

  27. ԿނภΔʁ
    ιϑτׂΓࠐΈ͸NICͷׂΓࠐΈ͕͔͔ͬͨCPU΁

    εέδϡʔϧ͞ΕΔ

    ˣ

    ϙʔϦϯά͔ΒϓϩτίϧελοΫͷ࣮ߦ·Ͱ

    ιϑτׂΓࠐΈ಺Ͱ࣮ߦ͞ΕΔ



    NICͷׂΓࠐΈ͕͔͔͍ͬͯΔCPU͚ͩʹ

    ෛՙ͕͔͔Δ

    View full-size slide

  28. ιϑτׂΓࠐΈ͕̍ͭͷ
    ίΞʹภͬͯੑೳ͕ग़ͳ͍
    • memcachedͳͲγϣʔτύέοτΛେ
    ྔʹࡹ͘ϫʔΫϩʔυͰݦࡏԽ

    • ιϑτ΢ΣΞׂΓࠐΈΛ࣮ߦ͍ͯ͠Δ
    CPU͕ϘτϧωοΫʹͳΓɺੑೳ͕ε
    έʔϧ͠ͳ͘ͳΔ

    View full-size slide

  29. ղܾํ๏
    • ύέοτΛෳ਺ͷCPU΁෼ࢄ͔ͤͯ͞Βϓ
    ϩτίϧॲཧ͢Δ࢓૊Έ͕͋Ε͹ྑ͍

    • ୠ͠ɺTCPʹ͸ॱংอূ͕༗ΔͷͰฒྻʹ
    ॲཧ͞ΕΔͱύέοτͷฒ΂௚͠ʢϦΦʔ
    μʣ͕ൃੜͯ͠ύϑΥʔϚϯε͕མͪΔ


    View full-size slide

  30. TCP Reordering
    • γʔέϯεφϯόʔ௨ΓͷॱংͰύέο
    τ͕ண৴͍ͯ͠Ε͹ॱʹόοϑΝ΁ίϐʔ
    ͍͚ͯͩ͘͠ͰΑ͍͕…
    ̍
    ̍
    protocol
    processing
    user buffer

    View full-size slide

  31. TCP Reordering
    ̍
    ̍
    protocol
    processing
    user buffer
    SFPSEFS
    RVFVF

    • ॱং͕ཚΕ͍ͯΔͱύέοτͷฒ΂௚
    ͠ʢϦΦʔμʣ࡞ۀ͕ඞཁʹͳΔ

    View full-size slide

  32. ղܾํ๏ʢଓʣ
    • ̍ͭͷϑϩʔ͸̍ͭͷCPUͰॲཧ͞Ε
    Δํ͕౎߹͕ྑ͍

    View full-size slide

  33. RSS
    ʢReceive Side Scalingʣ
    • CPU͝ͱʹผʑͷड৴ΩϡʔΛ࣋ͭNIC

    ʢMultiQueue NICͱݺ͹ΕΔʣ

    • ड৴Ωϡʔ͝ͱʹಠׂཱͨ͠ΓࠐΈΛ࣋ͭ

    • ಉ͡ϑϩʔʹଐ͢Δύέοτ͸ಉ͡Ωϡʔ΁ɺ

    ҟͳΔϑϩʔʹଐ͢Δύέοτ͸ͳΔ΂͘ผͷ
    Ωϡʔ΁෼ࢄ

    ˠύέοτϔομͷϋογϡ஋Λܭࢉ͢ΔࣄʹΑ
    ΓѼઌΩϡʔΛܾఆ

    View full-size slide

  34. MSI-XׂΓࠐΈ
    • PCI ExpressͰαϙʔτ

    • σόΠε͋ͨΓ2048ݸͷIRQΛ࣋ͯΔ

    • ͦΕͧΕͷIRQͷׂΓࠐΈઌCPUΛબ΂
    Δ

    ˠ1ͭͷNIC͕CPUίΞ਺෼ͷIRQΛ࣋
    ͯΔ

    View full-size slide

  35. RSSʹΑΔ
    ύέοτৼΓ෼͚
    NIC
    ύέοτ
    ύέοτ
    ύέοτ
    ϋογϡܭࢉ
    ύέοτண৴
    hash queue
    σΟεύον
    ࢀর
    RX
    Queue
    #0
    RX
    Queue
    #1
    RX
    Queue
    #2
    RX
    Queue
    #3
    cpu0 cpu1 cpu2 cpu3
    ड৴ॲཧ
    ׂΓࠐΈ
    ड৴ॲཧ


    0
    1

    View full-size slide

  36. Ωϡʔબ୒ͷखॱ
    indirection_table[64] = initial_value

    input[12] = 

    {src_addr, dst_addr, src_port, dst_port}

    key = toeplitz_hash(input, 12)

    index = key & 0x3f

    queue = indirection_table[index]

    View full-size slide

  37. RSSಋೖલ

    View full-size slide

  38. RPS
    • RSSඇରԠͷΦϯϘʔυNICΛ͏·͔ͭͬͯ͘αʔ
    όͷੑೳΛ޲্͍ͤͨ͞

    • ιϑτͰRSSΛ࣮૷ͯ͠͠·͓͏

    • ιϑτׂΓࠐΈͷஈ֊ͰύέοτΛ֤CPU΁͹Β
    ·͘

    • CPUׂؒΓࠐΈΛ࢖ͬͯଞͷCPUΛՔಈͤ͞Δ

    • RSSͷιϑτ΢ΤΞʹΑΔΤϛϡϨʔγϣϯ

    View full-size slide

  39. cpu3
    cpu2
    cpu1
    cpu0
    ׂΓࠐΈແޮԽ
    ϓϩτίϧॲཧ
    ιέοτ
    ड৴ॲཧ
    Ϣʔβ
    ϓϩάϥϜ
    VTFS
    CV⒎FS
    socket
    queue
    ύέοτ
    γεςϜ
    ίʔϧ
    ϓϩηεىচ
    ϋʔυ΢ΣΞׂΓࠐΈ
    Ϣʔβۭؒ΁ίϐʔ
    ύέοτ
    ύέοτ
    ιϑτ΢ΣΞׂΓࠐΈ
    ύέοτड৴
    ϋογϡܭࢉ
    σΟεύον
    ϓϩτίϧॲཧ
    ιέοτ
    ड৴ॲཧ
    Ϣʔβ
    ϓϩάϥϜ
    VTFS
    CV⒎FS
    socket
    queue
    backlog
    #1
    hash queue
    ࢀর


    0
    1
    $16ؒ
    ׂΓࠐΈ
    backlog
    #2
    backlog
    #3

    View full-size slide

  40. RPSͷ࢖͍ํ
    # echo "f" > /sys/class/net/eth0/queues/rx-0/rps_cpus

    # echo 4096 > /sys/class/net/eth0/queues/rx-0/
    rps_flow_cnt

    View full-size slide

  41. RPSಋೖલ

    View full-size slide

  42. RPS netperf result
    netperf benchmark result on lwn.net:

    e1000e on 8 core Intel

    Without RPS: 90K tps at 33% CPU

    With RPS: 239K tps at 60% CPU

    !
    foredeth on 16 core AMD

    Without RPS: 103K tps at 15% CPU

    With RPS: 285K tps at 49% CPU

    View full-size slide

  43. RFS
    • ϓϩηε௥੻ػೳΛRPSʹ௥Ճ

    View full-size slide

  44. RFS
    ϑϩʔʹׂΓ౰ͯΒ
    ΕͨΩϡʔ͕Ѽઌϓ
    ϩηεͷCPUͱҟͳ
    ΔͱΦʔόϔου͕
    ൃੜ͢Δ


    View full-size slide

  45. RFS
    ϋογϡςʔϒϧͷ
    ઃఆ஋Λมߋ͢Δ
    ࣄͰCPUΛҰக͞
    ͤΔࣄ͕Ͱ͖Δ

    View full-size slide

  46. RFSͷ࢖͍ํ
    # echo "f" > /sys/class/net/eth0/queues/rx-0/rps_cpus

    # echo 4096 > /sys/class/net/eth0/queues/rx-0/rps_flow_cnt

    # echo 32768 > /proc/sys/net/core/rps_sock_flow_entries

    View full-size slide

  47. RFS netperf result
    netperf benchmark result on lwn.net:

    e1000e on 8 core Intel

    No RFS or RPS 104K tps at 30% CPU

    No RFS (best RPS config): 290K tps at 63% CPU

    RFS 303K tps at 61% CPU

    !
    RPC test tps CPU% 50/90/99% usec latency StdDev

    No RFS or RPS 103K 48% 757/900/3185 4472.35

    RPS only: 174K 73% 415/993/2468 491.66

    RFS 223K 73% 379/651/1382 315.61

    View full-size slide

  48. Accelerated RFS
    • RFSΛMultiQueue NICͰ΋࣮ݱ͢ΔͨΊ
    ͷNICυϥΠό֦ு

    • Linux kernel͸ϓϩηεͷ࣮ߦதCPUΛ
    NICυϥΠόʹ௨஌

    • NICυϥΠό͸௨஌Λड͚ͯϑϩʔͷ
    ΩϡʔׂΓ౰ͯΛߋ৽

    View full-size slide

  49. Receive Side Scalingͷ੍ݶ
    • 32bitͷϋογϡ஋Λͦͷ··࢖༻ͯ͠
    ͍Ε͹ϋογϡিಥ͠ʹ͍͕͘ɺ
    Indirection Table͕খ͍͞ͷͰগͳ͍Ϗο
    τ਺Ͱindex஋ΛϚεΫ͍ͯ͠Δ

    ˠϑϩʔ͕ଟ͍࣌ʹϋογϡিಥ͢Δ

    • Accelerated RFSʹ͸ෆ޲͖

    View full-size slide

  50. Flow Steering
    • ϑϩʔͱΩϡʔͷରԠ৘ใΛهԱ

    4tupleɿΩϡʔ൪߸ͷΑ͏ͳܗࣜͰઃఆ

    • RSSͷΑ͏ͳ໌֬ͳڞ௨࢓༷͸ແ͍͕ɺ
    ֤ࣾͷ10GbEʹ࣮૷͞Ε͍ͯΔ

    • Accelerated RFS͸Flow SteeringΛલఏͱ
    ͍ͯ͠Δ

    View full-size slide

  51. Flow SteeringͰ
    खಈϑΟϧλઃఆ
    # ethtool --config-nfc ix00 flow-type tcp4
    src-ip 10.0.0.1 dst-ip 10.0.0.2 src-port 10000
    dst-port 10001 action 6

    Added rule with ID 2045

    View full-size slide

  52. XPS
    • MultiQueue NIC͸ૹ৴Ωϡʔ΋ෳ਺
    ͍࣋ͬͯΔ

    • XPS͸CPUͱૹ৴ΩϡʔͷׂΓ౰ͯΛܾ
    ΊΔΠϯλϑΣʔε

    View full-size slide

  53. XPSͷ࢖͍ํ
    # echo 1 > /sys/class/net/eth0/queues/tx-0/xps_cpus

    # echo 2 > /sys/class/net/eth0/queues/tx-1/xps_cpus

    # echo 4 > /sys/class/net/eth0/queues/tx-2/xps_cpus

    # echo 8 > /sys/class/net/eth0/queues/tx-3/xps_cpus

    View full-size slide

  54. 4.σʔλҠಈʹ൐͏
    ϨΠςϯγͷ࡟ݮ

    View full-size slide

  55. σʔλҠಈʹ൐͏
    ϨΠςϯγͷ࡟ݮ
    • ϓϩτίϧॲཧΑΓ΋Ή͠ΖNIC㲗ϝϞ
    Ϧ㲗CPUΩϟογϡͷؒͰͷσʔλҠ
    ಈʹ൐͏Φʔόϔουͷํ͕ॏ͍έʔε
    ͕͋Δ

    • ಛʹϝϞϦΞΫηε͕௿଎

    View full-size slide

  56. Intel Data Direct I/O
    Technology
    • NIC͕DMAͨ͠ύέοτͷσʔλ͸ɺ࠷ॳʹCPU
    ͕ΞΫηεͨ࣌͠ʹඞͣΩϟογϡώοτϛεΛ
    ى͜͢

    ɹɹɹɹɹɹɹɹɹˣ

    • CPUͷLLCʢࡾ࣍ΩϟογϡʣʹDMAͯ͠͠·͑ʂ

    • ৽͍͠XeonͱIntel 10GbEͰαϙʔτ

    • OSରԠ͸ෆཁʢHW͕ಁաతʹఏڙ͢Δػೳʣ

    View full-size slide

  57. ίϐʔ͕ॏ͍
    Process(User)
    Process(Kernel)
    HW Intr Handler
    SW Intr Handler
    ύέοτड৴
    ϓϩτίϧॲཧ
    ιέοτ
    ड৴ॲཧ
    Ϣʔβ
    ϓϩάϥϜ
    VTFS
    CV⒎FS
    input
    queue
    socket
    queue
    ύέοτ
    γεςϜίʔϧ
    ϓϩηεىচ
    ιϑτ΢ΣΞׂΓࠐΈεέδϡʔϧ
    ϋʔυ΢ΣΞׂΓࠐΈ
    Ϣʔβۭؒ΁ίϐʔ

    View full-size slide

  58. ίϐʔ͕ॏ͍͕
    θϩίϐʔԽ͸ࠔ೉
    • NICͷDMAόοϑΝ͸ΩϡʔຖʹઃఆͰ͖Δ͕ϑ
    ϩʔຖͰ͸ͳ͍

    ˠͦ΋ͦ΋ΩϡʔΛҰͭͷΞϓϦͰઐ༗ग़དྷΔલ
    ఏͰͳ͍ͱແཧ

    • όοϑΝ͕ϖʔδαΠζʹΞϥΠϯɾΞϩέʔτ
    ͞Εͯͳ͍ͱແཧ

    • ύέοτϔομͱϖΠϩʔυ͕෼཭͞Εͯͳ͍ͱ
    όοϑΝʹύέοτϔομ·Ͱॻ͔Εͯ͠·͏

    View full-size slide

  59. • ʢIntel I/O ATͱ΋ݺ͹ΕΔʣ

    • NICͷόοϑΝˠΞϓϦέʔγϣϯͷόο
    ϑΝ΁DMAసૹ

    • CPUෛՙΛ࡟ݮ

    • νοϓηοτʹ࣮૷

    • CONFIG_NET_DMA=y in Linux
    Intel QuickData Technology

    View full-size slide

  60. 5.ϓϩτίϧελοΫΛ
    ܦ༝͠ͳ͍ωοτϫʔΫIO

    View full-size slide

  61. ϓϩτίϧελοΫΛ
    ܦ༝͠ͳ͍ωοτϫʔΫIO
    • ϓϩτίϧॲཧΛ͢Δඞཁ΋Socket APIͰ͋Δඞཁ
    ΋ແ͍ͳΒɺωοτϫʔΫIO͸΋ͬͱ଎͘ग़དྷΔ

    • ಛఆ༻్޲͚

    • ϓϩτίϧॲཧΛඞཁͱ͠ͳ͍ΞϓϦέʔγϣϯ

    ˠsnortɺOpenvSwitchͳͲ

    • ϓϩτίϧॲཧΛࣗલͰߦͳͬͯͰ΋ੑೳΛ্͛
    ͍ͨΞϓϦέʔγϣϯ

    View full-size slide

  62. جຊతͳ࢓૊Έ
    • ઐ༻NICυϥΠόͱઐ༻
    ϥΠϒϥϦΛ༻͍ͯɺ
    NICͷड৴όοϑΝΛ
    MMAP

    • ύέοτΛϙʔϦϯά

    • ΞϓϦݻ༗ͷύέοτ
    ʹର͢ΔॲཧΛ࣮ߦ
    NIC
    RX1 RX2 RX3
    Kernel Driver
    App
    RX1 RX2 RX3
    MMAP
    1BD
    LFUT
    Polling
    Do some
    work

    View full-size slide

  63. RAWιέοτɾBPF
    ͱͷҧ͍ʁ
    • θϩίϐʔ͕جຊ

    • ϚϧνΩϡʔͷड৴όοϑΝΛͦͷ··Ϣʔ
    βϥϯυʹΤΫεϙʔτ͍ͯ͠Δ

    • ↑ʹΑΓɺϚϧνεϨουੑೳ͕ߴ͍

    ʢRAWιέοτɾBPF͸γϯάϧεϨουʣ

    • ্ड़ͷػೳΛ࣮ݱ͢ΔͨΊNICͷυϥΠόΛվ

    View full-size slide

  64. Intel DPDK
    • ׂΓࠐΈΛ΍ΊͯϙʔϦϯάΛ࢖༻͠Φʔόϔου࡟ݮ

    • ड৴όοϑΝʹHugePageΛ࢖͏ࣄʹΑΓTLB missΛ௿ݮ

    • 64 byte packetͷL3ϑΥϫʔσΟϯάੑೳʢIntelࢿྉΑΓʣ

    • Linux network stackɿXeon E5645 x 2 → 12.2Mpps

    • DPDKɿXeon E5645 x 1 → 35.2Mpps

    • DPDK : Next generation Intel Processor x 1 → 80Mpps

    • OpenvSwitchରԠ

    • ରԠNICɿIntel


    View full-size slide

  65. ྨࣅͷ࣮૷
    • PF_RING DNA

    ntopͷ࣮૷ɺLinux޲͚

    libpcapαϙʔτ

    ରԠNICɿIntel

    • Netmap

    FreeBSD޲͚ͷ࣮૷ɺҰԠLinux൛͋Γ

    libpcap, OpenvSwitchαϙʔτ

    ରԠNICɿIntel, Realtek...

    View full-size slide

  66. ·ͱΊ
    • ߴ଎ͳωοτϫʔΫIOΛࡹͨ͘Ίʹ༷ʑͳվળ͕
    ߦΘΕ͍ͯΔࣄΛ঺հ

    • ϋʔυ΢ΣΞɾιϑτ΢ΣΞͷ྆໘Ͱ࣮૷ͷݟ௚
    ͕͠ཁٻ͞Ε͓ͯΓɺͦͷൣғ͸ωοτϫʔΫʹ
    ௚઀ؔ܎ͳ͍Α͏ͳॴʹ·ͰٴͿ

    • औΓ׶͑ͣ໌೔͔Βग़དྷΔ͜ͱɿ

    ·ͣ͸αʔόʹऔΓ෇͚ΔNICΛ

    ʮϚϧνΩϡʔNICʯʮRSSରԠʯʹ͠Α͏

    View full-size slide