10GbE時代のネットワークI/O高速化

Transcript

1. 10GbE࣌୅ͷωοτ ϫʔΫI/Oߴ଎Խ Takuya ASADA<syuu@freebsd.org>

2. Slide URL •http://slidesha.re/16OV9Yx

3. ͸͡Ίʹ • 10GbEɺ40GbEͳͲͷۃΊͯߴ଎ͳ௨৴Λ αϙʔτ͢ΔNIC͕ɺPCαʔόͷྖҬͰ ΋࢖ΘΕΔΑ͏ʹͳ͖͍ͬͯͯΔ • ͜ͷΑ͏ͳ଎౓ͷ௨৴Λιϑτ΢ΣΞ ʢOSʣͰॲཧ͠ߴ͍ੑೳΛಘΔʹ͸༷ʑ ͳো֐͕͋Γɺϋʔυ΢ΣΞɾιϑτ΢Σ Ξ྆໘ͷ࣮૷Λݟ௚͢ඞཁ͕͋Δ

4. ࠓ೔ͷτϐοΫ 1. ׂΓࠐΈ͕ଟ͗͢Δ 2. ϓϩτίϧॲཧ͕ॏ͍ 3. ෳ਺ͷCPUͰύέοτॲཧ͍ͨ͠ 4. σʔλҠಈʹ൐͏ϨΠςϯγͷ࡟ݮ 5.

   ϓϩτίϧελοΫΛܦ༝͠ͳ͍ωοτϫʔ ΫIO

5. 1. ׂΓࠐΈ͕ଟ͗͢Δ Process(User) Process(Kernel) HW Intr Handler SW Intr Handler

   ύέοτड৴ ϓϩτίϧॲཧ ιέοτ ड৴ॲཧ Ϣʔβ ϓϩάϥϜ VTFS
   CV⒎FS input queue socket queue ύέοτ γεςϜίʔϧ ϓϩηεىচ ιϑτ΢ΣΞׂΓࠐΈεέδϡʔϧ ϋʔυ΢ΣΞׂΓࠐΈ Ϣʔβۭؒ΁ίϐʔ

6. ׂΓࠐΈ͕ଟ͗͢Δ • NICͷੑೳ޲্ʹΑͬͯɺҰఆ࣌ؒʹ NIC͕ॲཧͰ͖Δύέοτ਺͕ඈ༂తʹ ૿Ճ • ̍ύέοτຖʹׂΓࠐΈ͕དྷΔͱɺ௨ ৴ྔ͕ଟ͍ͱ͖ʹίϯςΩετεΠο νճ਺͕૿͑͗͢ੑೳ͕ྼԽ

7. چདྷͷύέοτड৴ॲཧ Process(User) Process(Kernel) HW Intr Handler SW Intr Handler ύέοτड৴

   ϓϩτίϧॲཧ ιέοτ ड৴ॲཧ Ϣʔβ ϓϩάϥϜ VTFS
   CV⒎FS input queue socket queue ύέοτ γεςϜίʔϧ ϓϩηεىচ ιϑτ΢ΣΞׂΓࠐΈεέδϡʔϧ ϋʔυ΢ΣΞׂΓࠐΈ Ϣʔβۭؒ΁ίϐʔ ϋʔυ΢ΣΞׂΓࠐΈ
 ˣ
 ड৴ΩϡʔʹΩϡʔ Πϯά
 ˣ
 ιϑτ΢ΣΞׂΓࠐ Έεέδϡʔϧ

8. چདྷͷύέοτड৴ॲཧ • ̍ύέοτड৴͢ΔͨͼʹׂΓࠐΈΛ ड͚ͯॲཧΛߦ͍ͬͯΔ • 64byte frameͷ࠷େड৴Մೳ਺ɿ • GbEɿ໿1.5Mppsʢ150ສʣ •

   10GbEɿ໿15Mppsʢ1500ສʣ

9. ׂΓࠐΈΛແޮʹ͢Δʁ • ϙʔϦϯάํࣜ • NICͷׂΓࠐΈΛېࢭ͠ɺ୅ΘΓʹΫϩοΫׂΓࠐΈΛ༻ ͍ͯఆظతʹड৴ΩϡʔΛνΣοΫ • σϝϦοτɿϨΠςϯγ্͕͕ΔɾఆظతʹCPUΛى͜͢ ඞཁ͕͋Δ •

   ϋΠϒϦουํࣜ • ௨৴ྔ͕ଟ͘࿈ଓͯ͠ύέοτॲཧΛߦ͍ͬͯΔ࣌ͷΈׂ ΓࠐΈΛແޮԽͯ͠ϙʔϦϯάͰಈ࡞


10. NAPIʢϋΠϒϦουํ ࣜʣ Process(User) Process(Kernel) HW Intr Handler SW Intr Handler

    ׂΓࠐΈແޮԽ ϓϩτίϧॲཧ ιέοτ ड৴ॲཧ Ϣʔβ ϓϩάϥϜ VTFS
    CV⒎FS socket queue ύέοτ γεςϜίʔϧ ϓϩηεىচ ϋʔυ΢ΣΞׂΓࠐΈ Ϣʔβۭؒ΁ίϐʔ ύέοτ ύέοτ ιϑτ΢ΣΞׂΓࠐΈεέδϡʔϧ ύέοτड৴ ύέοτ͕ແ͘ͳΔ ·Ͱ܁Γฦ͠ ϋʔυ΢ΣΞׂΓࠐΈ
 ˣ
 ׂΓࠐΈແޮԽˍ
 ϙʔϦϯά։࢝ ↓ ύέοτ͕ແ͘ͳͬ ͨΒׂΓࠐΈ༗ޮԽ

11. Interrupt Coalescing • NIC͕OSෛՙΛߟׂྀͯ͠ΓࠐΈΛؒ Ҿ͘ • ύέοτ਺ݸʹҰճׂΓࠐΉɺ
 ͍҃͸Ұఆظؒ଴͔ͬͯΒׂΓࠐΉ • σϝϦοτɿϨΠςϯγ্͕͕Δ

12. Interrupt CoalescingͷޮՌ • Intel 82599(ixgbe)ͰInterrupt Coalescingແޮɺ
 ༗ޮʢׂΓࠐΈස౓ࣗಈௐ੔ʣͰൺֱ • MultiQueue, GRO,

    LRO౳͸ແޮԽ • iperfͷTCPϞʔυͰܭଌ interrupts throughput packets CPU%(sy+si) ແޮ 46687 int/s 7.82 Gbps 660386 pkt/s 97.6% ༗ޮ 7994 int/s 8.24 Gbps 711132 pkt/s 79.6%

13. Process(User) Process(Kernel) HW Intr Handler SW Intr Handler ׂΓࠐΈແޮԽ ϓϩτίϧॲཧ

    ιέοτ ड৴ॲཧ Ϣʔβ ϓϩάϥϜ VTFS
    CV⒎FS socket queue ύέοτ γεςϜίʔϧ ϓϩηεىচ ϋʔυ΢ΣΞׂΓࠐΈ Ϣʔβۭؒ΁ίϐʔ ύέοτ ύέοτ ιϑτ΢ΣΞׂΓࠐΈεέδϡʔϧ ύέοτड৴ ύέοτ͕ແ͘ͳΔ ·Ͱ܁Γฦ͠ 2.ϓϩτίϧॲཧ͕ॏ͍

14. ϓϩτίϧॲཧ͕ॏ͍ • ಛʹখ͞ͳύέοτ͕େྔʹಧ͘৔߹ʹ ϓϩτίϧॲཧͰCPU࣌ؒΛେྔʹ࢖ͬͯ ͠·͏ • ύέοτ਺෼ϓϩτίϧελοΫ͕ݺͼग़ ͞ΕΔ
 ྫɿ64byte frameͷ৔߹


    ˠཧ࿦্ͷ࠷େ஋͸1500ສճ/s

15. TOE
    (TCP Offload Engine) • OSͰϓϩτίϧॲཧ͢ΔͷΛ΍ΊͯɺNICͰॲཧ͢Δ • σϝϦοτ • ηΩϡϦςΟɿTOEʹηΩϡϦςΟϗʔϧ͕ੜͯ͡΋ɺOS

    ଆ͔Βରॲ͕ग़དྷͳ͍ • ෳࡶੑɿOSͷωοτϫʔΫελοΫΛTOEͰஔ͖׵͑Δʹ ͸͔ͳΓ޿ൣғͷมߋ͕ඞཁ
 ϝʔΧʹΑͬͯTOEͷ࣮૷͕ҟͳΓڞ௨ΠϯλϑΣʔεఆ ͕ٛࠔ೉ • Linuxɿαϙʔτ༧ఆແ͠

16. Checksum Offloading
 • IPɾTCPɾUDP checksumͷܭࢉΛNICͰ ߦ͏

17. Checksum Offloading ͷޮՌ • Intel 82599(ixgbe)Ͱൺֱ • iperfͷTCPϞʔυͰܭଌ • MultiQueue͸ແޮԽ

    • ethtool -K ix0 rx off throughput CPU%(sy+si) ແޮ 8.27 Gbps 86 ༗ޮ 8.27 Gbps 85.2

18. LRO
    (Large Receive Offload) • NIC͕ड৴ͨ͠TCPύέοτΛ݁߹͠ɺ
 େ͖ͳύέοτʹ͔ͯ͠ΒOS΁౉͢ • ϓϩτίϧελοΫͷݺͼग़͠ճ਺Λ ࡟ݮ

    • LinuxͰ͸ιϑτ΢ΣΞʹΑΔLRO͕࣮ ૷͞Ε͍ͯΔʢGROʣ

19. LRO͕ແ͍৔߹ • ύέοτຖʹωοτϫʔΫελοΫΛ ࣮ߦ seq 10000 seq 10001 seq 10002

    seq 10003 ←1500bytes→ To network stack

20. LRO͕༗Δ৔߹ • ύέοτΛ݁߹͔ͯ͠ΒωοτϫʔΫελοΫΛ ࣮ߦɺωοτϫʔΫελοΫͷ࣮ߦճ਺Λ࡟ݮ seq 10000 seq 10001 seq 10002

    seq 10003 ←1500bytes→ To network stack big one packet

21. GROͷޮՌ • Intel 82599(ixgbe)Ͱൺֱ • MultiQueue͸ແޮԽ • iperfͷTCPϞʔυͰܭଌ • ethtool

    -K ix0 gro off packets network stack called count throughput CPU%(sy+si) ແޮ 632139 pkt/s 632139 call/s 7.30 Gbps 97.6% ༗ޮ 712387 pkt/s 47957 call/s 8.25 Gbps 79.6%

22. TSO
    (TCP Segmentation Offload) • LROͷٯ • ύέοτΛϑϥάϝϯτԽͤͣʹૹ৴
 NIC͕ύέοτΛMTUαΠζʹ෼ׂ •

    OS͸ύέοτ෼ׂॲཧΛলུग़དྷΔ • LinuxͰ͸ιϑτ΢ΣΞʹΑΔGSOɺ
 ϋʔυ΢ΣΞʹΑΔTSOʗUFOΛαϙʔτ

23. TSOͷޮՌ • Intel 82599(ixgbe)Ͱൺֱ • MultiQueue͸ແޮԽ • iperfͷTCPϞʔυͰܭଌ • ethtool

    -K ix0 gso off tso off packets throughput CPU%(sy+si) ແޮ 247794 pkt/s 2.87 Gbps 53.5% ༗ޮ 713127 pkt/s 8.16 Gbps 26.8%

24. 3.ෳ਺ͷCPUͰύέοτॲཧ͍ͨ͠ cpu0 Process(User) Process(Kernel) HW Intr Handler SW Intr Handler

    ׂΓࠐΈແޮԽ ϓϩτίϧॲཧ ιέοτ ड৴ॲཧ Ϣʔβ ϓϩάϥϜ VTFS
    CV⒎FS socket queue ύέοτ γεςϜίʔϧ ϓϩηεىচ ϋʔυ΢ΣΞׂΓࠐΈ Ϣʔβۭؒ΁ίϐʔ ύέοτ ύέοτ ιϑτ΢ΣΞׂΓࠐΈεέδϡʔϧ ύέοτड৴ ύέοτ͕ແ͘ͳΔ ·Ͱ܁Γฦ͠ cpu1 Process(User) Process(Kernel) HW Intr Handler SW Intr Handler ׂΓࠐΈແޮԽ ϓϩτίϧॲཧ ιέοτ ड৴ॲཧ Ϣʔβ ϓϩάϥϜ VTFS
    CV⒎FS socket queue ύέοτ γεςϜίʔϧ ϓϩηεىচ ϋʔυ΢ΣΞׂΓࠐΈ Ϣʔβۭؒ΁ίϐʔ ύέοτ ύέοτ ιϑτ΢ΣΞׂΓࠐΈεέδϡʔϧ ύέοτड৴ ύέοτ͕ແ͘ͳΔ ·Ͱ܁Γฦ͠

25. ιϑτׂΓࠐΈ͕
    ̍ͭͷίΞʹภΔ

26. ιϑτׂΓࠐΈͱ͸ʁ Process(User) Process(Kernel) HW Intr Handler SW Intr Handler ׂΓࠐΈແޮԽ

    ϓϩτίϧॲཧ ιέοτ ड৴ॲཧ Ϣʔβ ϓϩάϥϜ VTFS
    CV⒎FS socket queue ύέοτ γεςϜίʔϧ ϓϩηεىচ ϋʔυ΢ΣΞׂΓࠐΈ Ϣʔβۭؒ΁ίϐʔ ύέοτ ύέοτ ιϑτ΢ΣΞׂΓࠐΈεέδϡʔϧ ύέοτड৴ ύέοτ͕ແ͘ͳΔ ·Ͱ܁Γฦ͠ ϙʔϦϯά͔Β
 ϓϩτίϧॲཧ·Ͱ →ωοτϫʔΫIOͷ େ൒෦෼

27. ԿނภΔʁ ιϑτׂΓࠐΈ͸NICͷׂΓࠐΈ͕͔͔ͬͨCPU΁
 εέδϡʔϧ͞ΕΔ
 ˣ ϙʔϦϯά͔ΒϓϩτίϧελοΫͷ࣮ߦ·Ͱ
 ιϑτׂΓࠐΈ಺Ͱ࣮ߦ͞ΕΔ ↓ NICͷׂΓࠐΈ͕͔͔͍ͬͯΔCPU͚ͩʹ
 ෛՙ͕͔͔Δ

28. ιϑτׂΓࠐΈ͕̍ͭͷ ίΞʹภͬͯੑೳ͕ग़ͳ͍ • memcachedͳͲγϣʔτύέοτΛେ ྔʹࡹ͘ϫʔΫϩʔυͰݦࡏԽ • ιϑτ΢ΣΞׂΓࠐΈΛ࣮ߦ͍ͯ͠Δ CPU͕ϘτϧωοΫʹͳΓɺੑೳ͕ε έʔϧ͠ͳ͘ͳΔ

29. ղܾํ๏ • ύέοτΛෳ਺ͷCPU΁෼ࢄ͔ͤͯ͞Βϓ ϩτίϧॲཧ͢Δ࢓૊Έ͕͋Ε͹ྑ͍ • ୠ͠ɺTCPʹ͸ॱংอূ͕༗ΔͷͰฒྻʹ ॲཧ͞ΕΔͱύέοτͷฒ΂௚͠ʢϦΦʔ μʣ͕ൃੜͯ͠ύϑΥʔϚϯε͕མͪΔ


30. TCP Reordering • γʔέϯεφϯόʔ௨ΓͷॱংͰύέο τ͕ண৴͍ͯ͠Ε͹ॱʹόοϑΝ΁ίϐʔ ͍͚ͯͩ͘͠ͰΑ͍͕… ̍   

      ̍      protocol processing user buffer

31. TCP Reordering ̍      ̍ 

        protocol processing user buffer SFPSEFS RVFVF    • ॱং͕ཚΕ͍ͯΔͱύέοτͷฒ΂௚ ͠ʢϦΦʔμʣ࡞ۀ͕ඞཁʹͳΔ

32. ղܾํ๏ʢଓʣ • ̍ͭͷϑϩʔ͸̍ͭͷCPUͰॲཧ͞Ε Δํ͕౎߹͕ྑ͍

33. RSS
    ʢReceive Side Scalingʣ • CPU͝ͱʹผʑͷड৴ΩϡʔΛ࣋ͭNIC
 ʢMultiQueue NICͱݺ͹ΕΔʣ • ड৴Ωϡʔ͝ͱʹಠׂཱͨ͠ΓࠐΈΛ࣋ͭ

    • ಉ͡ϑϩʔʹଐ͢Δύέοτ͸ಉ͡Ωϡʔ΁ɺ
 ҟͳΔϑϩʔʹଐ͢Δύέοτ͸ͳΔ΂͘ผͷ Ωϡʔ΁෼ࢄ
 ˠύέοτϔομͷϋογϡ஋Λܭࢉ͢ΔࣄʹΑ ΓѼઌΩϡʔΛܾఆ

34. MSI-XׂΓࠐΈ • PCI ExpressͰαϙʔτ • σόΠε͋ͨΓ2048ݸͷIRQΛ࣋ͯΔ • ͦΕͧΕͷIRQͷׂΓࠐΈઌCPUΛબ΂ Δ
 ˠ1ͭͷNIC͕CPUίΞ਺෼ͷIRQΛ࣋

    ͯΔ

35. RSSʹΑΔ
    ύέοτৼΓ෼͚ NIC ύέοτ ύέοτ ύέοτ ϋογϡܭࢉ ύέοτண৴ hash queue

    σΟεύον ࢀর RX Queue #0 RX Queue #1 RX Queue #2 RX Queue #3 cpu0 cpu1 cpu2 cpu3 ड৴ॲཧ ׂΓࠐΈ ड৴ॲཧ ▪ ▪ 0 1

36. Ωϡʔબ୒ͷखॱ indirection_table[64] = initial_value input[12] = 
 {src_addr, dst_addr, src_port,

    dst_port} key = toeplitz_hash(input, 12) index = key & 0x3f queue = indirection_table[index]

37. RSSಋೖલ

38. RSSಋೖޙ

39. RPS • RSSඇରԠͷΦϯϘʔυNICΛ͏·͔ͭͬͯ͘αʔ όͷੑೳΛ޲্͍ͤͨ͞ • ιϑτͰRSSΛ࣮૷ͯ͠͠·͓͏ • ιϑτׂΓࠐΈͷஈ֊ͰύέοτΛ֤CPU΁͹Β ·͘ •

    CPUׂؒΓࠐΈΛ࢖ͬͯଞͷCPUΛՔಈͤ͞Δ • RSSͷιϑτ΢ΤΞʹΑΔΤϛϡϨʔγϣϯ

40. cpu3 cpu2 cpu1 cpu0 ׂΓࠐΈແޮԽ ϓϩτίϧॲཧ ιέοτ ड৴ॲཧ Ϣʔβ ϓϩάϥϜ

    VTFS
    CV⒎FS socket queue ύέοτ γεςϜ ίʔϧ ϓϩηεىচ ϋʔυ΢ΣΞׂΓࠐΈ Ϣʔβۭؒ΁ίϐʔ ύέοτ ύέοτ ιϑτ΢ΣΞׂΓࠐΈ ύέοτड৴ ϋογϡܭࢉ σΟεύον ϓϩτίϧॲཧ ιέοτ ड৴ॲཧ Ϣʔβ ϓϩάϥϜ VTFS
    CV⒎FS socket queue backlog #1 hash queue ࢀর ▪ ▪ 0 1 $16ؒ ׂΓࠐΈ backlog #2 backlog #3

41. RPSͷ࢖͍ํ # echo "f" > /sys/class/net/eth0/queues/rx-0/rps_cpus # echo 4096 >

    /sys/class/net/eth0/queues/rx-0/ rps_flow_cnt

42. RPSಋೖલ

43. RPSಋೖޙ

44. RPS netperf result netperf benchmark result on lwn.net: e1000e on

    8 core Intel Without RPS: 90K tps at 33% CPU With RPS: 239K tps at 60% CPU ! foredeth on 16 core AMD Without RPS: 103K tps at 15% CPU With RPS: 285K tps at 49% CPU

45. RFS • ϓϩηε௥੻ػೳΛRPSʹ௥Ճ

46. RFS ϑϩʔʹׂΓ౰ͯΒ ΕͨΩϡʔ͕Ѽઌϓ ϩηεͷCPUͱҟͳ ΔͱΦʔόϔου͕ ൃੜ͢Δ


47. RFS ϋογϡςʔϒϧͷ ઃఆ஋Λมߋ͢Δ ࣄͰCPUΛҰக͞ ͤΔࣄ͕Ͱ͖Δ

48. RFSͷ࢖͍ํ # echo "f" > /sys/class/net/eth0/queues/rx-0/rps_cpus # echo 4096 >

    /sys/class/net/eth0/queues/rx-0/rps_flow_cnt # echo 32768 > /proc/sys/net/core/rps_sock_flow_entries

49. RFS netperf result netperf benchmark result on lwn.net: e1000e on

    8 core Intel No RFS or RPS 104K tps at 30% CPU No RFS (best RPS config): 290K tps at 63% CPU RFS 303K tps at 61% CPU ! RPC test tps CPU% 50/90/99% usec latency StdDev No RFS or RPS 103K 48% 757/900/3185 4472.35 RPS only: 174K 73% 415/993/2468 491.66 RFS 223K 73% 379/651/1382 315.61

50. Accelerated RFS • RFSΛMultiQueue NICͰ΋࣮ݱ͢ΔͨΊ ͷNICυϥΠό֦ு • Linux kernel͸ϓϩηεͷ࣮ߦதCPUΛ NICυϥΠόʹ௨஌

    • NICυϥΠό͸௨஌Λड͚ͯϑϩʔͷ ΩϡʔׂΓ౰ͯΛߋ৽

51. Receive Side Scalingͷ੍ݶ • 32bitͷϋογϡ஋Λͦͷ··࢖༻ͯ͠ ͍Ε͹ϋογϡিಥ͠ʹ͍͕͘ɺ Indirection Table͕খ͍͞ͷͰগͳ͍Ϗο τ਺Ͱindex஋ΛϚεΫ͍ͯ͠Δ
 ˠϑϩʔ͕ଟ͍࣌ʹϋογϡিಥ͢Δ

    • Accelerated RFSʹ͸ෆ޲͖

52. Flow Steering • ϑϩʔͱΩϡʔͷରԠ৘ใΛهԱ
 4tupleɿΩϡʔ൪߸ͷΑ͏ͳܗࣜͰઃఆ • RSSͷΑ͏ͳ໌֬ͳڞ௨࢓༷͸ແ͍͕ɺ ֤ࣾͷ10GbEʹ࣮૷͞Ε͍ͯΔ • Accelerated

    RFS͸Flow SteeringΛલఏͱ ͍ͯ͠Δ

53. Flow SteeringͰ
    खಈϑΟϧλઃఆ # ethtool --config-nfc ix00 flow-type tcp4 src-ip

    10.0.0.1 dst-ip 10.0.0.2 src-port 10000 dst-port 10001 action 6 Added rule with ID 2045

54. XPS • MultiQueue NIC͸ૹ৴Ωϡʔ΋ෳ਺ ͍࣋ͬͯΔ • XPS͸CPUͱૹ৴ΩϡʔͷׂΓ౰ͯΛܾ ΊΔΠϯλϑΣʔε

55. XPSͷ࢖͍ํ # echo 1 > /sys/class/net/eth0/queues/tx-0/xps_cpus # echo 2 >

    /sys/class/net/eth0/queues/tx-1/xps_cpus # echo 4 > /sys/class/net/eth0/queues/tx-2/xps_cpus # echo 8 > /sys/class/net/eth0/queues/tx-3/xps_cpus

56. 4.σʔλҠಈʹ൐͏ ϨΠςϯγͷ࡟ݮ

57. σʔλҠಈʹ൐͏
    ϨΠςϯγͷ࡟ݮ • ϓϩτίϧॲཧΑΓ΋Ή͠ΖNIC㲗ϝϞ Ϧ㲗CPUΩϟογϡͷؒͰͷσʔλҠ ಈʹ൐͏Φʔόϔουͷํ͕ॏ͍έʔε ͕͋Δ • ಛʹϝϞϦΞΫηε͕௿଎

58. Intel Data Direct I/O Technology • NIC͕DMAͨ͠ύέοτͷσʔλ͸ɺ࠷ॳʹCPU ͕ΞΫηεͨ࣌͠ʹඞͣΩϟογϡώοτϛεΛ ى͜͢
 ɹɹɹɹɹɹɹɹɹˣ

    • CPUͷLLCʢࡾ࣍ΩϟογϡʣʹDMAͯ͠͠·͑ʂ • ৽͍͠XeonͱIntel 10GbEͰαϙʔτ • OSରԠ͸ෆཁʢHW͕ಁաతʹఏڙ͢Δػೳʣ

59. ίϐʔ͕ॏ͍ Process(User) Process(Kernel) HW Intr Handler SW Intr Handler ύέοτड৴

    ϓϩτίϧॲཧ ιέοτ ड৴ॲཧ Ϣʔβ ϓϩάϥϜ VTFS
    CV⒎FS input queue socket queue ύέοτ γεςϜίʔϧ ϓϩηεىচ ιϑτ΢ΣΞׂΓࠐΈεέδϡʔϧ ϋʔυ΢ΣΞׂΓࠐΈ Ϣʔβۭؒ΁ίϐʔ

60. ίϐʔ͕ॏ͍͕
    θϩίϐʔԽ͸ࠔ೉ • NICͷDMAόοϑΝ͸ΩϡʔຖʹઃఆͰ͖Δ͕ϑ ϩʔຖͰ͸ͳ͍
 ˠͦ΋ͦ΋ΩϡʔΛҰͭͷΞϓϦͰઐ༗ग़དྷΔલ ఏͰͳ͍ͱແཧ • όοϑΝ͕ϖʔδαΠζʹΞϥΠϯɾΞϩέʔτ ͞Εͯͳ͍ͱແཧ

    • ύέοτϔομͱϖΠϩʔυ͕෼཭͞Εͯͳ͍ͱ όοϑΝʹύέοτϔομ·Ͱॻ͔Εͯ͠·͏

61. • ʢIntel I/O ATͱ΋ݺ͹ΕΔʣ • NICͷόοϑΝˠΞϓϦέʔγϣϯͷόο ϑΝ΁DMAసૹ • CPUෛՙΛ࡟ݮ •

    νοϓηοτʹ࣮૷ • CONFIG_NET_DMA=y in Linux Intel QuickData Technology

62. 5.ϓϩτίϧελοΫΛ
    ܦ༝͠ͳ͍ωοτϫʔΫIO

63. ϓϩτίϧελοΫΛ
    ܦ༝͠ͳ͍ωοτϫʔΫIO • ϓϩτίϧॲཧΛ͢Δඞཁ΋Socket APIͰ͋Δඞཁ ΋ແ͍ͳΒɺωοτϫʔΫIO͸΋ͬͱ଎͘ग़དྷΔ • ಛఆ༻్޲͚ • ϓϩτίϧॲཧΛඞཁͱ͠ͳ͍ΞϓϦέʔγϣϯ


    ˠsnortɺOpenvSwitchͳͲ • ϓϩτίϧॲཧΛࣗલͰߦͳͬͯͰ΋ੑೳΛ্͛ ͍ͨΞϓϦέʔγϣϯ

64. جຊతͳ࢓૊Έ • ઐ༻NICυϥΠόͱઐ༻ ϥΠϒϥϦΛ༻͍ͯɺ NICͷड৴όοϑΝΛ MMAP • ύέοτΛϙʔϦϯά • ΞϓϦݻ༗ͷύέοτ

    ʹର͢ΔॲཧΛ࣮ߦ NIC RX1 RX2 RX3 Kernel Driver App RX1 RX2 RX3 MMAP 1BD LFUT Polling Do some work

65. RAWιέοτɾBPF
    ͱͷҧ͍ʁ • θϩίϐʔ͕جຊ • ϚϧνΩϡʔͷड৴όοϑΝΛͦͷ··Ϣʔ βϥϯυʹΤΫεϙʔτ͍ͯ͠Δ • ↑ʹΑΓɺϚϧνεϨουੑೳ͕ߴ͍
 ʢRAWιέοτɾBPF͸γϯάϧεϨουʣ

    • ্ड़ͷػೳΛ࣮ݱ͢ΔͨΊNICͷυϥΠόΛվ ଄

66. Intel DPDK • ׂΓࠐΈΛ΍ΊͯϙʔϦϯάΛ࢖༻͠Φʔόϔου࡟ݮ • ड৴όοϑΝʹHugePageΛ࢖͏ࣄʹΑΓTLB missΛ௿ݮ • 64 byte

    packetͷL3ϑΥϫʔσΟϯάੑೳʢIntelࢿྉΑΓʣ • Linux network stackɿXeon E5645 x 2 → 12.2Mpps • DPDKɿXeon E5645 x 1 → 35.2Mpps • DPDK : Next generation Intel Processor x 1 → 80Mpps
 • OpenvSwitchରԠ • ରԠNICɿIntel


67. ྨࣅͷ࣮૷ • PF_RING DNA
 ntopͷ࣮૷ɺLinux޲͚
 libpcapαϙʔτ
 ରԠNICɿIntel • Netmap
 FreeBSD޲͚ͷ࣮૷ɺҰԠLinux൛͋Γ


    libpcap, OpenvSwitchαϙʔτ
 ରԠNICɿIntel, Realtek...

68. ·ͱΊ • ߴ଎ͳωοτϫʔΫIOΛࡹͨ͘Ίʹ༷ʑͳվળ͕ ߦΘΕ͍ͯΔࣄΛ঺հ • ϋʔυ΢ΣΞɾιϑτ΢ΣΞͷ྆໘Ͱ࣮૷ͷݟ௚ ͕͠ཁٻ͞Ε͓ͯΓɺͦͷൣғ͸ωοτϫʔΫʹ ௚઀ؔ܎ͳ͍Α͏ͳॴʹ·ͰٴͿ • औΓ׶͑ͣ໌೔͔Βग़དྷΔ͜ͱɿ


    ·ͣ͸αʔόʹऔΓ෇͚ΔNICΛ
 ʮϚϧνΩϡʔNICʯʮRSSରԠʯʹ͠Α͏