Share
Embed
超高速なパケットI/Oフレームワーク netmap の紹介
netmap: a novel framework for fast packet I/O
Yuuki Tsubouchi (yuuk1)
August 01, 2013
Transcript
25 slides
-
netmap: a novel framework for fast packet I/O Luigi Rizzo,
Universita` di Pisa, Italy In Proceedings of the 2012 USENIX Annual Technical Conference, June 2012. (Best Paper award at Usenix ATC'12) id:y_uuki ྠߨࢿྉ -
༻ޠ NIC ≒ device ≒ network adapter ≒ hardware OS
≒ host stack ≒ network stack -
Introduction (1) ωοτϫʔΫϞχλʔ, τϥϑΟοΫδΣωϨʔλͳͲͷΞ ϓϦέʔγϣϯʹٻΊΒΕΔߴϨʔτͳRawύέοτI/OΛ൚ ༻OSͰαϙʔτ͍ͯ͠ͳ͍ APIͱͯ͠Raw Socket, Berkeley Packet
Filter, AF SOCKET familyͳͲ͕ΘΕΔ ύϑΥʔϚϯε͕ेͰͳ͍ طଘͷੑೳ্ख๏ͰಛघͳϋʔυΣΞͷػೳ(NICͳ Ͳ)ʹґଘ͕ͪ͠ -
Introduction (3) netmap҆શ͔ͭ؆୯ʹ͑Δ netmap clientγεςϜΛյͤͳ͍ σόΠεϨδελΧʔωϧϝϞϦΛclientʹݟͤͳ͍ zero-copyసૹʹదͨ͠ۃʹ୯७ͳσʔλϞσϧΛ ༻͢Δ multiqueue adapterͷαϙʔτ
Πϕϯτͷ௨ʹඪ४ͷγεςϜίʔϧ(select(2), poll(2))Λ༻͢Δ طଘͷΞϓϦέʔγϣϯ͔ΒͷҠߦ͕؆୯ -
Kernel and User API OSNICͷσʔλߏͷίϐʔΛͭ όοϑΝOSઐ༻·ͨσόΠεʹґଘ͠ͳ͍ίϯςφ (mbufs, sk_buffs)ʹϦϯΫ͍ͯ͠Δ ֤ύέοτʹؔ͢ΔେྔͷϝλσʔλΛؚΉ Driver/OS
σόΠευϥΠόͱOSύέοτΛϑϥάϝϯτʹׂ͢Δ ϑϥάϝϯτԽͷͨΊͷΦʔόϔου͕େ͖͍ RawύέοτI/O RawύέοτΛಡΈॻ͖͢Δඪ४APIΧʔωϧɾϢʔβۭؒ ͷؒͷσʔλίϐʔʹ࠷1ճͷϝϞϦίϐʔΛཁ͢Δ 1ύέοτ͋ͨΓ1ճͷγεςϜίʔϧΛཁ͢Δ -
Related Work ύέοτॲཧ্ͷͨΊͷطଘख๏ Socket APIs BPF, AF_PACKETͳͲ ͍ΘΏΔRawιέοτɻύέοτΛෳͯ͠userlandʹΈͤΔ Packet Filter
hooks Netgraph(FreeBSD), Netfilter(FreeBSD) ύέοτͷෳ͕ඞཁͳ͍ (in kernel) applicationύέοτॲཧʹڬ·ΕΔ (firewall)ͳͲ Direct buffer access Kernel mode Click (applicationΛkernelͰ࣮ߦ͢Δ) PF_RING, PACKET_MMAP(userlandʹpacket bufferΛΈͤΔ) NIC DMA engine, NetChannels, PacketShader I/O Engine Hardware solutions (FPGA) -
Netmap ϢʔβۭؒͷΞϓϦέʔγϣϯʹߴͳύέοτΞΫη εΛఏڙ͢Δ ܰྔͳϝλσʔλ: ίϯύΫτɼ͍͍͢ 1ճͷγεςϜίʔϧͰଟ͘ͷύέοτΛॲཧ ઢܗͷݻఆύέοτόοϑΝ σόΠεΛopenͨ͠ͱ͖ʹࣄલ֬อ͞ΕΔ ύέοτ͝ͱʹϝϞϦΛ֬อɾഁغ͢ΔίετΛΔ ΞϓϦέʔγϣϯʹύέοτόοϑΝͷอޢΞΫηε
ΛڐՄ͢Δ͜ͱʹΑΓσʔλίϐʔίετݮ ศརͳϋʔυΣΞػೳͷαϙʔτ(ϚϧνΩϡʔͳͲ) -
netmap netmap modeͰNIC ringshost stack͔Βଓ Λஅ͞Εͯnetmap APIΛհͯ͠ύέοτΛަ͢Δ Ճ͞Εͨ2ͭͷnetmap ringsʹΑΓApplication host
stackͱͤΔ netmap ringsڞ༗ϝϞϦ্ɹɹɹɹɹɹɹɹɹɹɹ ʹ࣮͞ΕΔ OSஅʹؾ͔ͮͣʹɼɹɹɹɹɹɹɹɹɹɹɹɹɹ ௨ৗͲ͓ΓΠϯλϑΣʔεɹɹɹɹɹɹɹɹɹɹɹɹɹ Λ༻ɾཧ͢Δ select(2)/poll(2)ɹɹɹɹɹɹɹɹɹɹɹɹɹɹɹಉ ظʹ༻͞ΕΔ -
Data Structures(1) netmapͷΩʔίϯϙʔωϯτ per-packetΦʔόϔουͷݮ ΠϯλϑΣʔεؒͷߴͳϑΥϫʔσΟϯά NICͱOSελοΫؒͷޮతͳσʔλͷΓͱΓ ϚϧνΩϡʔϦϯάͱϚϧνίΞͷαϙʔτ ֤ΠϯλϑΣʔεΛ3छྨͷΦϒδΣΫτʹؔ࿈͚Δ packet buffers,
netmap rings, netmap_if શͯͷΦϒδΣΫτΧʔωϧಠཱͨ͠ྖҬʹஔ શͯͷϢʔβϓϩηεʹڞ༗͞ΕΔ ୯ҰྖҬͷ༻zero-copyసૹʹศར -
Data Structures(2) packet buffers (pkt_buf) ݻఆͰNICͱϢʔβϓϩηεʹΑΓڞ༗͞ΕΔ netmap modeʹҠߦ͢Δͱશͯͷnetmap ringsʹରԠ͢Δ buffer͕ࣄલʹ֬อ͞ΕΔ
(࠶֬อ͞Εͳ͍) netmap ring: NIC ringɹɹɹɹɹɹɹɹɹɹɹɹɹ ͷσόΠεඇґଘͳෳ ring-size:εϩοτͷݸ cur: ring্ͷݱࡏͷɹɹɹɹɹɹɹɹɹɹɹɹɹɹɹɹɹ read/writeҐஔ avail:ར༻Մೳͳbufferͷݸ buf_ofs, slots netmap_if: read-onlyͳใ -
The netmap API /dev/netmapΛopenͯ͠ɼioctl(fd,NIOCREQ,arg) Λ࣮ߦ͢Δ͜ͱʹΑΓɼnetmap modeʹͳΔ mmap(2)ʹΑΓϓϩηεͷΞυϨεۭ͔ؒΒڞ༗ϝϞϦ ΞΫηεՄೳʹ͢Δ ioctl(2)͕ύέοτͷૹड৴Λαϙʔτ ioctl(fd,
NIOCTXSYNC) OSʹ৽͍͠ύέοτͷૹ৴Λ௨ ioctl(fd, NIOCRXSYNC) ಡΈࠐΈՄೳͳύέοτͷݸΛOSʹฉ͘ non blockingͳͨΊσʔλίϐʔ͕ͳ͘(netmapͱ hardware ringͷಉظҎ֎)ͷෳύέοτΛಉ࣌ʹѻ͑Δ per-packetΦʔόϔουΛݮͰ͖Δ -
Talkinkg to the host stack netmap client2ͭͷnetmap ringʹΑΓOSελο ΫͱΓͱΓ͢Δ ૹ৴ύέοτ·ΔͰཧΠϯλϑΣʔε͔Βདྷ͔ͨͷΑ͏ʹ
OSελοΫʹ͞ΕΔ OSελοΫ͔Βདྷͨύέοτnetmap ringʹܨ͕ΕΔ netmap clientɼOSελοΫʹଓ͞Εͨnetmap ringͱNICʹଓ͞Εͨnetmap ringͱͷؒͰύέοτ ͕ਖ਼͘͠ΓͱΓ͞Ε͔ͨΛ֬ೝ͢Δ -
Implementation FreeBSDͷ࣮ΛؚΉݱߦόʔδϣϯ2000ߦఔ γεςϜίʔϧ(ioctl,select/poll)ͱυϥΠόͷมߋআ͘ ֤σόΠευϥΠόͷύονͦΕͧΕ500ߦఔ σόΠευϥΠόͷมߋΛ࠷খʹ͢ΔͨΊʹػೳͷ΄ͱΜͲ͕֤ υϥΠόͰڞ௨ͨ͠ίʔυͰ࣮͞Ε͍ͯΔ ֤υϥΠόͷมߋ2ͭͷػೳ netmap mode༗ޮ࣌ͷ࠶ॳظԽ υϥΠόͷϩοΫॲཧΛڞ௨ίʔυʹҠߦ
γεςϜίʔϧͷ࣮ߦதɼnetmapͰdevice driver͕ ΄ͱΜͲͷࣄΛ͢Δ ΩϟογϡϩʔΧϦςΟͷ্ɼϦιʔεཧͷ୯७Խ ׂΓࠐΈͰͳ͍ίϯςΩετͰଟ͗͢ΔॲཧΛ࣮ߦ͢Δ৺͕ ͳ͍ -
Performance metrics ύέοτॲཧෳͷαϒγεςϜͱؔ࿈͢Δ CPUύΠϓϥΠϯɼΩϟογϡɼϝϞϦɼI/Oόε ରͷΞϓϦέʔγϣϯCPUόϯυ -> CPUίετ Λܭଌ ΞϓϦέʔγϣϯ͔ΒNIC·ͰͷύέοτҠಈ͕ର Per-byte
costs NICͷbuffer͔ΒͷσʔλҠಈʹফඅ͢ΔCPUαΠΫϧ Per-packet costs ֤ύέοτʹରͯ͠NIC ringͷεϩοτΛߋ৽͕ඞཁ memory allocation, system calls, ... very simple test programs a packet generator, a packet receiver -
Test equipment netmapඇৗʹޮతͰ10GbpsͷଳҬΛ͍Δͨ ΊɼCPUΫϩοΫΛམͱ͢ ૹ৴ଳҬpacket generatorͰܭଌ͢Δ Ωϡʔ,ίΞ/εϨουͷͱύέοτ࣮ߦ࣌ʹܾఆ ύέοτ͋Β͔͡Ί४උ͞ΕΔͨΊɼper-byteίετ ΄΅θϩ CPU
i7-870 4 core 2.93GHz memory 1.33GHz NIC dual port 10Gbps netmap FreeBSD HEAD/amd64 Apr 2012 -
Transmit speed VS clock rate ύέοτ64όΠτͰɹɹɹɹɹɹɹɹɹɹɹɹɹɹɹ ΫϩοΫϨʔτͱίΞΛɹɹɹɹɹɹɹɹɹɹɹɹɹ มԽͤͨ͞ͱ͖ͷૹ৴ੑೳ 1ίΞͷͱ͖900MHz·Ͱɹɹɹɹɹɹɹɹɹɹɹɹɹ ΫϩοΫʹରͯ͠εϧʔϓɹɹɹɹɹɹɹɹɹɹɹɹɹɹ
οτεέʔϧ͢Δ ɹɹɹɹɹɹɹɹɹɹɹɹɹɹ (60-65 cycles/packet) ͜ͷςετͰper-packetॲཧҎԼͷ2͔ͭ͠ͳ͍ netmap ringεϩοτͷதͷݕূ ରԠ͢ΔNIC ringεϩοτͷߋ৽ -
Speed VS packet size ૹ৴ड৴ͱʹύέοτͷมԽʹର͢Δεϧʔϓοτ ड৴ଆ150όΠτҎԼͰsurprisingͳۂઢ 64όΠτͷഒͷͱ͖ɹɹɹɹɹɹɹɹɹɹɹɹɹɹɹɹɹɹ ͚ͩ࠷େϨʔτ ΩϟογϡϥΠϯ͕ؔ࿈ɹɹɹɹɹɹɹɹɹɹɹɹɹɹɹɹɹ ͍ͯ͠Δʁ
(͜ͷ͋ͨΓͷઆ໌ɹɹɹɹɹɹɹɹɹɹɹɹɹɹɹɹɹɹɹɹ ཧղͰ͖͍ͯͳ͍) -
Transmit speed VS batch size ύέοτΛ·ͱΊͯѻ͏͜ͱʹΑΔੑೳ্Λ֬ೝ͢Δ γεςϜίʔϧͷίετNICͷϨδελͷΞΫηεͳͲ͕ ݮ͞Ε͍ͯΔͣ batch size=ύέοτͷݸ
batch size=1ͷͱ͖ɹɹɹɹɹɹɹɹɹɹɹɹɹɹ 2.45 Mpps(408ns/pkt) batch size=8ͷͱ͖ɹɹɹɹɹɹɹɹɹɹɹɹɹ 14.88 Mpps FreeBSDͷඪ४తͳpoll(2)1ճ 250ns 1 callͰෳͷύέοτΛѻ͏͜ͱඞਢ -
Conclusions and feature work netmapϢʔβۭؒͷΞϓϦέʔγϣϯʹରͯ͠Rawύ έοτΛૹड৴͢ΔߴͳνϟϯωϧΛఏڙ͢Δ ಛघͳϋʔυΣΞʹґଘ͠ͳ͍ ҆શ͔͍͍ͭ͢ ࣮ݧͷ݁ՌɼlowϨϕϧͳύέοτI/OΛͬͨ෯͍Ξ ϓϦέʔγϣϯʹରͯ͠େ෯ͳੑೳ্ΛఏڙͰ͖Δ͜ͱ
͕Θ͔ͬͨ Future work netmapͷػೳΛOSͷωοτϫʔΫελοΫʹ࠾༻ ԾڥͰͷޮతͳωοτϫʔΫॲཧͷαϙʔτ