Upgrade to Pro — share decks privately, control downloads, hide ads and more …

XDPの最新動向とLINEにおける取り組み事例 / Recent updates on XDP

XDPの最新動向とLINEにおける取り組み事例 / Recent updates on XDP

A3966f193f4bef226a0d3e3c1f728d7f?s=128

LINE Developers
PRO

November 01, 2019
Tweet

More Decks by LINE Developers

Other Decks in Technology

Transcript

  1. 9%1ͷ࠷৽ಈ޲ͱ -*/&ʹ͓͚ΔऔΓ૊Έࣄྫ :PIFJ ,BOFNBSV -*/&$PSQPSBUJPO ZPIFJLBOFNBSV BUMJOFDPSQDPN 0QFO/FUXPSLJOH$POGFSFODF+BQBOr 

  2. ࣗݾ঺հ ؙۚ ༸ฏ ZVOB[VOP ωοτϫʔΫӡ༻  d  σʔληϯλωοτϫʔΫͷӡ༻ɺϐΞϦϯάௐ੔ ΦϑΟεແઢ-"/ͷߏஙɾӡ༻

    ϓϥΠϕʔτΫϥ΢υ l7FSEBzͷωοτϫʔΫؔ࿈։ൃ d -PBE#BMBODFSBTB4FSWJDF -#BB4 ͷઃܭɾ։ൃɾߏஙɾӡ༻ Open Networking Conference Japan 2019 2019-11-01 2
  3. "HFOEB • 9%1ͷ֓ཁͱϢʔεέʔε • ࠷ۙͷ9%1ʹ·ͭΘΔΞοϓσʔτ • -*/&ʹ͓͚Δ9%1ར༻ͷྺ࢙ͱݱঢ় Open Networking Conference

    Japan 2019 2019-11-01 3
  4. 9%1ͷ֓ཁͱϢʔεέʔε Open Networking Conference Japan 2019 2019-11-01 4

  5. 9%1 F9QSFTT %BUB 1BUI -JOVYLFSOFMͷ಺Ͱಈ࡞͢Δɺߴ଎ύέοτॲཧج൫ lGBTUQBUIz /*$υϥΠόͷϨϕϧͰύέοτॲཧΛద༻ ,FSOFM ೥݄ ͕࠷ॳͷϦϦʔε

    F#1'Λ༻͍ͨߴ଎͔ͭݎ࿚ͳج൫ Χʔωϧ಺ͰͷόΠτίʔυ࣮ߦ؀ڥ F#1' WFSJGJFSΛ༻͍࣮ͨߦલͷ੩తݕূʹΑΓ҆શੑΛ୲อ .BQʹΑΔঢ়ଶ؅ཧ͓ΑͼϢʔβۭؒͱͷ࿈ܞ +*5ʹΑΔߴ଎ॲཧ Open Networking Conference Japan 2019 2019-11-01 5
  6. 9%1ͱF#1'ͰͰ͖Δ͜ͱ Open Networking Conference Japan 2019 Kernel User Device Driver

    Kernel NIC Device Driver XDP skb Network Interface Network Interface IP TCP / UDP / ICMP Socket BPF maps AF_XDP App App App Transmit Redirect Redirect Pass Drop Parse Lookup Modify Push/Pop Hdr Verdict 2019-11-01 App 6
  7. 9%1ͳΒͰ͸ͷಛ௃ -JOVYʹ࣮૷͞Εͨଟ͘ͷػೳ΍࢓૊ΈΛ׆༻Մೳ Χʔωϧͷ'*#ΛϓϩάϥϜ಺͔ΒࢀরՄೳ ௨ৗͷωοτϫʔΫελοΫ΁ॲཧΛҠৡ Ϣʔβۭؒͷϓϩηεͱͷ࿈ܞ͕༰қ ΧʔωϧϞδϡʔϧͱൺֱͯ͠։ൃ͕༰қ ύέοτॲཧઐ༻ͷ$16ׂΓ౰͕ͯෆཁ ࣮ߦதͷϓϩάϥϜΛಈతʹࠩ͠ସ͑Մೳ 2019-11-01 Open

    Networking Conference Japan 2019 7
  8. 9%1ͷϢʔεέʔε -PBE#BMBODFS 4FDVSJUZ෼໺Ͱͷద༻ྫ͕ଟ͍ 'BDFCPPL,BUSBO -*/&'BCSJD-# $MPVEGMBSF-%SPQ 4VSJDBUB $JMJVN $BMJDP ࠷ۙͷద༻ྫ

    "'@9%1΋ؚΉ /"5 %FDBQ 1BDLFU TBNQMFS ! NJYJ <> 43W 1SPYZ z&OE"$z ! *OUFSPQ 5PLZP  <> 0QFO W4XJUDI <> %1%, "'@9%1 1.% <> Open Networking Conference Japan 2019 2019-11-01 8 [1]      https://bcu30.jp/2019/talk/mabuchi-toshiya/ [2] edenden/end.ac https://github.com/edenden/end.ac [3] Open vSwitch with AF_XDP http://docs.openvswitch.org/en/latest/intro/install/afxdp/ [4] AF_XDP Poll Mode Driver https://doc.dpdk.org/guides/nics/af_xdp.html
  9. ࠷ۙͷ9%1ʹ·ͭΘΔΞοϓσʔτ πʔϧνΣΠϯ΍֤छαϙʔτͷ֦ॆ F#1'ͷ֦ு "'@9%1 2019-11-01 Open Networking Conference Japan 2019

    9
  10. πʔϧνΣΠϯ΍࢖༻؀ڥͷ֦ॆ ϥΠϒϥϦ΍πʔϧ͕੔උ͞ΕɺओཁύοέʔδϚωʔδϟͰఏڙ MJCCQG Ϣʔβۭ͔ؒΒF#1'Λѻ͏ͨΊͷϥΠϒϥϦ CQGUPPMΞλον͞Ε͍ͯΔϓϩάϥϜ΍Ϛοϓͷμϯϓ͕Մೳͳπʔϧ ֤छσΟετϦϏϡʔγϣϯͰσϑΥϧτͰ࢖༻Մೳʹ 'FEPSB 6CVOUVͰ͸"'@9%1ؚΊ͙͢ʹ࢖༻Մೳ 3)&-Ͱ΋9%1͕࢖༻Մೳʹ UFDIOPMPHZ

    QSFWJFX 2019-11-01 Open Networking Conference Japan 2019 10
  11. YEQUVUPSJBM ໢ཏతͳνϡʔτϦΞϧ 9%1։ൃʹର͢Δ೉қ౓Λܰݮ “Kernel selftests/bpf and samples/bpf: bad starting point”[1]

    2019-11-01 Open Networking Conference Japan 2019 11 https://github.com/xdp-project/xdp-tutorial [1] https://linuxplumbersconf.org/event/4/contributions/460/attachments/244/426/xdp-distro-view.pdf
  12. F#1'ͷ֦ு #PVOEFE-PPQͷαϙʔτ<> ैདྷɺF#1'Ͱ͸MPPQΛ࢖༻Ͱ͖ͳ͔ͬͨ l3FTUSJDUFE$z F#1' WFSJGJFSͷڧԽʹΑΓɺ༗ݶ௕ͷϧʔϓ͸࢖༻Մೳʹ lάϩʔόϧม਺zͷαϙʔτ<> ैདྷɺάϩʔόϧͳઃఆ৘ใͷѻ͍ʹ೉͕͋ͬͨ ํ๏EFGJOF ͯ͠࠶ίϯύΠϧ

    ํ๏ ௕͞ͷBSSBZ NBQΛ࡞੒࣮ͯ͠ߦ࣌ʹ౎౓MPPLVQ ಺෦తʹ͸௕͞ͷNBQͱͯ͠ѻΘΕΔ͕ɺMPPLVQແ͠ͰࢀরͰ͖ΔΑ͏ʹ ಁաతʹల։͞ΕΔ 2019-11-01 Open Networking Conference Japan 2019 12 [1] Bounded loops in BPF for the 5.3 kernel https://lwn.net/Articles/794934/ [2] BPF support for global data https://lore.kernel.org/bpf/20190409210910.32048-1-daniel@iogearbox.net/
  13. F#1'ͷ֦ு DPOU ϓϩάϥϜඇґଘͷDIBJOJOH 8*1 <> ैདྷΑΓɺUBJMDBMMͰϓϩάϥϜͷDIBJOJOH͸Մೳͩͬͨ ݺͼग़͠ݩϓϩάϥϜͰ໌ࣔతʹUBJM DBMMΛ࢖༻ͤͣͱ΋DIBJOJOHͰ͖ΔΑ͏ʹ ྫ -#ϓϩάϥϜͷલʹɺYEQEVNQΛࠩ͠ࠐΜͰύέοτΩϟϓνϟ

    2019-11-01 Open Networking Conference Japan 2019 13 [1] xdp: Support multiple programs on a single interface through chain calls https://lore.kernel.org/netdev/157046883502.2092443.146052429591277809.stgit@alrua-x1/
  14. App UMEM "'@9%1 9%1Λ༻͍ͯɺύέοτΛϢʔβۭؒʹ౉͢࢓૊Έ F#1'ʹ੍ݶ͞Εͳ͍ɺॊೈͰߴ଎ͳΞϓϦέʔγϣϯΛ࣮ݱ 2019-11-01 Open Networking Conference Japan

    2019 14 Device Driver XDP BPF map (XSKMAP) NIC Network Stack Queue Queue Parse Lookup Modify Verdict Queue Filter Kernel NIC AF_XDP socket (XSK) User Kernel Tx Completion Fill Redirect Rx Lookup
  15. "'@9%1  /*$ͱͷؔ܎ ϋʔυ΢ΣΞΩϡʔͱ94,͸ ର৅Ωϡʔʹύέοτ͕ೖΔΑ͏ߟྀ͕ඞཁ ํ๏ ΩϡʔΛݸʹ͢Δ ํ๏ OUVQMF GJMUFSΛઃఆ͢Δ

    ํ๏ Ωϡʔͷ਺͚ͩ94,Λ࡞੒͢Δ Ωϡʔͷ؅ཧػೳ͸ผ్ݕ౼த<> 2019-11-01 Open Networking Conference Japan 2019 15 NIC Queue Queue Queue Filter Kernel NIC [1] Making Networking Queues a First Class Citizen in the Kernel https://linuxplumbersconf.org/event/4/contributions/462/
  16. "'@9%1  9%1ϓϩάϥϜͰͷॲཧ 94,."1ΛݟͯϦμΠϨΫτઌΛܾఆ ͜ͷϓϩάϥϜ͸ݱ࣌఺Ͱ͸ࣗ෼Ͱ༻ҙ লུ͢Δ࢓૊Έ͕ݕ౼த<> ͲͷύέοτΛϦμΠϨΫτ͢Δ͔ɺ ϓϩάϥϜͰهड़Մೳ ׬શͳΧʔωϧόΠύεͰ͸ͳ͍ 2019-11-01

    Open Networking Conference Japan 2019 16 Device Driver XDP BPF map (XSKMAP) Network Stack Parse Lookup Modify Verdict Redirect Lookup [1] Enable direct receive on AF_XDP sockets https://lore.kernel.org/bpf/1570515415-45593-1-git-send-email-sridhar.samudrala@intel.com/
  17. App UMEM "'@9%1  ϢʔβۭؒͰͷॲཧ ϝϞϦ؅ཧ͸ΞϓϦέʔγϣϯଆͷ੹೚ ड৴༻όοϑΝ͸'JMM3JOHΛ௨ͯ͠Χʔωϧʹ ద੾ʹڙڅ͢Δඞཁ͕͋Δ "'@9%1 TPDLFUࣗମ͸੍ޚ༻νϟϯωϧ

    ύέοτͷૹड৴͸3JOHܦ༝ ;FSPDPQZ΋ར༻Մೳ /*$͔Β6.&.ʹ%." 2019-11-01 Open Networking Conference Japan 2019 17 AF_XDP socket (XSK) User Kernel Tx Completion Fill Rx
  18. -*/&ʹ͓͚Δ9%1ར༻ͷྺ࢙ͱݱঢ় 2019-11-01 Open Networking Conference Japan 2019 18

  19. ར༻ྫ -BZFS ϩʔυόϥϯα Open Networking Conference Japan 2019 L4LB (Software)

    Real Server L7LB (Software) L3 Switch (Hardware) Client TCP/HTTP(S) Proxy IPIP encap ECMP [2] #*%($%:4 ;>?/&'-0!26 @  https://www.slideshare.net/linecorp/ss-116879618 [3] LINE0*-+-$%*,19=!)"". 85 "%3 0*-7<  https://www.slideshare.net/linecorp/lines-infrastructure-platform-how-it-scales-massive-services-and-maintains-low-operational-cost 2019-11-01 19
  20. ར༻ྫ 43W /FUXPSL /PEF 2019-11-01 Open Networking Conference Japan 2019

    20 Push/Pop SRH [1] LINE Data Center Networking with SRv6 https://speakerdeck.com/line_developers/line-data-center-networking-with-srv6 [2]       https://engineering.linecorp.com/ja/blog/intern2019-report-infra/
  21. ͜Ε·Ͱͷঢ়گ ೥݄ࠒ͔Β࣮؀ڥͰ࢖༻த 9%1σʔλϓϨʔϯ༝དྷͷো֐͸ݱ࣌఺Ͱ݅ Ή͠Ζ9%1Ҏ֎ͷ෦෼ ಛʹϔϧενΣοΫ َ͕໳ ໌֬ͳ$%1MBOF෼཭ʹΑΓɺൺֱతͳେ͖ͳมߋ΋εϜʔζʹ ։ൃ؀ڥ΍ӡ༻؀ڥͷݟ௚͠͸ਵ࣮࣌ࢪ ؀ڥߏ੒ɾσϓϩΠํࣜͷมߋ $POUJOVPVT*OUFHSBUJPO

    $* ͷಋೖ ϞχλϦϯάͷ֦ॆ 2019-11-01 Open Networking Conference Japan 2019 21
  22. ࣗಈԽ͞Εͨςετ࣮ߦ 2019-11-01 Open Networking Conference Japan 2019 22 linelinux ver.

    X on VM linelinux ver. Y on VM linelinux ver. X on Baremetal L4LB XDP Program L4LB XDP Program L4LB XDP Program Performance Test Behavior Test (BPF_PROG_TEST_RUN) Issue PR Kernel build system (“linelinux”) GitHub Enterprise Drone CI Run tests
  23. ϞχλϦϯά 1SPNFUIFVTΛ࢖༻ ઐ༻ͷFYQPSUFSΛ࣮૷͠ɺҎԼͷΑ͏ͳϝτϦΫεΛऩू #1' NBQʹॻ͔Εͨ౷ܭ৘ใ /*$ͷ9%1 RVFVFؔ࿈ϝτϦΫε FUIUPPM 4ܦ༝ ਖ਼ৗੑ൑ఆͱΞϥʔτ͸֎ܗ؂ࢹΛॏࢹ

    ϊʔυ୆਺͕ଟ͍ͨΊɺ֤छϝτϦΫεͰࡉ͔͘ΞϥʔτΛઃఆ͢Δͷ͕ࠔ೉ 2019-11-01 Open Networking Conference Japan 2019 23
  24. 2019-11-01 Open Networking Conference Japan 2019 24

  25. ࠷ۙͷࠔΓ͝ͱ #1'NBQͷಡΈॻ͖଎౓͕஗͍ ΤϯτϦͣͭಡΈॻ͖͢Δඞཁ͕͋Δ ΞϓϦέʔγϣϯىಈ࣌ʹઃఆྲྀ͠ࠐΈ଴͕ͪൃੜ --#Ͱ࢖༻͢Δ-36 -FBTU3FDFOUMZ6TFE UBCMFͷσόοά͕ਏ͍ ΤϯτϦ͕ଟ͍ͷͰμϯϓ͕େม ΤϯτϦΛಡΈࠐΉͱ-36UBCMFͷڍಈʹӨڹΛ༩͑ͯ͠·͏ +VNCPGSBNFαϙʔτ

    ݱঢ়͸,#·Ͱ<> 2019-11-01 Open Networking Conference Japan 2019 25 [1] XDP multi buffer design https://github.com/xdp-project/xdp-project/blob/master/areas/core/xdp-multi-buffer01-design.org
  26. ·ͱΊ 9%1͸ϦϦʔεҎޙண࣮ʹ։ൃ͕ਐΊΒΕ͍ͯΔ جຊࢥ૝͸มΘΒͣɺͦͷपลͷվળ΋ଟ͍ ҰํͰ"'@9%1ͷΑ͏ͳେ͖ͳมԽ΋͋Δ ػೳͷෆ଍΋·ͩ·ͩ͋Δ͕ɺվળʹ޲͚׆ൃͳٞ࿦͕ߦͳΘΕ͍ͯΔ ద༻ྫ΋ଟ͘ɺ௕ظؒ҆ఆՔಈ͍ͯ͠Δ ։ൃ΍ӡ༻ͷ͠΍͢͞΋౰ॳ͔Βང͔ʹվળ͞Ε͍ͯΔ Open Networking Conference

    Japan 2019 2019-11-01 26
  27. <13> -*/& %&7&-01&3 %":  ݄೔ ! άϥϯυχοίʔ౦ژ ୆৔ 2019-11-01

    Open Networking Conference Japan 2019 27 https://linedevday.linecorp.com/jp/2019/