͘͞ΒΠϯλʔωοτ גࣜձࣾ (C) Copyright 1996-2019 SAKURA internet Inc ͘͞ΒΠϯλʔωοτ ݚڀॴ Transtracer: ෼ࢄγεςϜʹ͓͚ΔTCP/UDP௨৴ͷ ऴ୺఺ͷ؂ࢹʹΑΔϓϩηεؒґଘؔ܎ͷࣗಈ௥੻ ৘ใॲཧֶձ ୈ12ճΠϯλʔωοτͱӡ༻ٕज़γϯϙδ΢Ϝ(IOTS)2019 2019.12.6 ௶಺༎थ*1, ݹ઒խେ*2, দຊ྄հ*1 *1) ͘͞ΒΠϯλʔωοτ, *2) ͸ͯͳ

2 1. എܠͱ໨త 2. ෼ࢄγεςϜ಺ͷґଘؔ܎ͷ௥੻ٕज़ 3. ఏҊख๏ 4. ࣮ݧͱߟ࡯ 5. ·ͱΊͱࠓޙͷల๬ ໨࣍

1. എܠͱ໨త

4 WebαʔϏε಺ͷґଘؔ܎ͷෳࡶԽͷഎܠ ௕ظؒͷαʔϏεఏڙத ͷػೳ௥Ճ Ϣʔβʔ͔ΒͷΞΫηε ૿Ճ ୯ҰͷαʔϏεࣄۀऀ͕ ෳ਺ͷαʔϏεΛఏڙ (SNS,ECαΠτͳͲ) ༻్ಛԽͷϛυϧ΢ΣΞͷ௥Ճ ৽چγεςϜͷࠞ߹ঢ়ଶ εέʔϧΞ΢τʹΑΔϗετ਺ ͷ૿Ճ ෳ਺ͷαʔϏεͷҰ෦Λڞ༗ (Ϣʔβʔೝূج൫ͳͲ) ϚΠΫϩαʔϏεԽ

5 ɾෳࡶͳґଘؔ܎ʹΑΓɼγεςϜʹมߋΛՃ͑ͨ࣌ʹ໰୊͕͋ͬͨ ৔߹ɼ༧૝ΑΓେ͖ͳো֐ʹͭͳ͕Δ͜ͱ΋͋Δ ɾো֐ൃੜΛڪΕͯɼௐࠪʹ࣌ؒΛཁ͠ɼ݁Ռతʹมߋ଎౓͕௿Լ ɾґଘؔ܎͸γεςϜͷՔಇதʹมߋ͞Ε͍ͯͨ͘ΊɼखಈͰυΩϡ ϝϯτΛߋ৽͠ଓ͚Δख͕ؒେ͖͍ ґଘؔ܎ͷෳࡶԽʹΑΓੜ·ΕΔ໰୊ҙࣝ มߋͷӨڹൣғΛ༧ଌ͢ΔͨΊʹ γεςϜ؅ཧऀʹͱͬͯະ஌ͷϓϩηεͷґଘؔ܎Λ ࣗಈͰ௥੻͢Δඞཁ͕͋Δ

6 ɾLinuxͷύέοτϑΟϧλϩάΛར༻ͯ͠L4઀ଓͷঢ়ଶΛऔಘ ɾLinuxΧʔωϧͷ௨৴ػߏΛར༻͢ΔݶΓɼOSϓϩηεʹରͯ͠ಁ աతʹ௥੻Մೳ ɾ஗ԆΦʔόϔου: ύέοτϑΟϧλॲཧͷΦʔόϔου ɾِӄੑ: ϦιʔεফඅΛ཈͑ΔͨΊʹɼύέοτΛແ࡞ҝʹαϯϓϦ ϯά͢ΔͨΊɼݕ஌࿙ΕͷՄೳੑ͕͋Δ طଘͷࣗಈ௥੻ٕज़ [1] [1] J. K. Clawson, Service Dependency Analysis via TCP/UDP Port Tracing, Master’s thesis, Brigham Young University-Provo 2015.

7 ݚڀ໨త: ະ஌ͷϓϩηεΛ௿ෛՙͰ௥੻ Linux OS Kernel Process Process TCP/UDP Flows … . . . User ιέοτ Λ؂ࢹ TCP/UDP઀ଓͷऴ୺఺Ͱ͋ΔιέοτΛ؂ࢹ͠઀ଓΛࣗಈ௥੻ 1. ஗ԆΦʔόϔουͷղܾ ɾιέοτͷ؂ࢹΛϓϩηε ͷ௨৴ͱಠཱͤ͞Δ 2. ِӄੑͷղܾ ɾ͢΂ͯͷιέοτΛ؂ࢹ ɾύέοτ͸ඵؒ਺ສ୯Ґ͕ ͋Γ͑Δ ɾιέοτ͸࠶ར༻͞Ε͏Δ ͨΊɼιέοτ਺ͷ΄͏͕ খ͘͞ͳΓ΍͍͢

8 γεςϜ؅ཧऀʹͱͬͯ఍߅ͳ͘ಋೖՄೳ ɾΦʔόϔου͕େ͖͍ͱɼ؅ཧऀ͸Өڹͷେ͖͍ίϯϙʔωϯτΛ ௥੻ର৅͔Β֎͢͜ͱ͕͋Δ ↪ ؅ཧऀͷهԱ΍υΩϡϝϯςʔγϣϯʹґଘ͢Δ ɾِӄੑ͕େ͖͍ͱɼ؅ཧऀ͸ಘΒΕͨґଘؔ܎Λ৴༻͠ͳ͘ͳΔ ↪ ؅ཧऀ͕खಈͰґଘؔ܎Λௐࠪ͢Δ ௿͍Φʔόʔϔουͱ௿ِ͍ӄੑʹΑΓ γεςϜ؅ཧऀ͕ར༻͢Δͱ͖ͷ৺ཧత఍߅Λ௿ݮՄೳ

2. ෼ࢄγεςϜ಺ͷ ґଘؔ܎ͷ௥੻ٕज़

10 ɾWebαʔϏεͷ෼ࢄγεςϜ಺ͷґଘؔ܎Λ௥੻Մೳ ɾWebαʔϏεͰ͸ɼΞΫηε਺͕େ͖͘ɼԠ౴଎౓͕1ඵҎԼͰ͋ Δ͜ͱ͕ಛ௃ ɾ઀ଓΛ࠶ར༻͠ɼ઀ଓ͝ͱͷෛՙΛ௿ݮ͍ͤͯ͞Δ ɾσʔληϯλʔ಺௨৴ͰHTTP/2ͷར༻͕૿େ ɾίωΫγϣϯϓʔϦϯάʹΑΓσʔλϕʔε΁ͷ઀ଓΛ࠶ར༻ ɾτϥϯεϙʔτ૚Ͱ͸ɼ௕໋ͳ઀ଓͱͳΔ܏޲͕͋Δ ຊݚڀͷ௥੻ର৅ͷલఏ

11 ෼ࢄγεςϜ಺ͷґଘؔ܎ͷ௥੻ٕज़ͷ෼ྨ ϦΫΤετϕʔε Ξϓϩʔν ΞϓϦέʔγϣϯ૚ͷϦΫΤετ͕ Ͳͷ௨৴ܦ࿏ΛͨͲΔ͔Λ௥੻͢Δख๏ ίωΫγϣϯϕʔε Ξϓϩʔν ऴ୺ϗετ্Ͱτϥϯεϙʔτ઀ଓͷঢ়ଶΛ औಘ͢Δ͜ͱʹΑΓґଘؔ܎Λ௥੻͢Δख๏ ύέοτϕʔε Ξϓϩʔν ϗετ্΍εΠον্ͰύέοτϔομΛ؍ଌ ͢Δ͜ͱʹΑΓɼ ґଘؔ܎Λ௥੻͢Δख๏

12 ɾΞϓϦέʔγϣϯ૚ͷ֤ϦΫΤετʹࣝผࢠΛׂΓৼΓɼޙଓͷϦ ΫΤετʹຒΊࠐΜ্ͩͰɼޙଓͷϓϩηε΁఻ൖͤ͞Δ ɾࣝผࢠΛཔΓʹɼϦΫΤετ͕γεςϜ಺ͷͲͷϓϩηεΛܦ༝͠ ͯॲཧ͞Ε͔ͨΛ௥੻ ϦΫΤετϕʔεΞϓϩʔν M. Y. Chen, et al., Pinpoint: Problem Determination in Large, Dynamic Internet Services, IEEE/IFIP International Conference on DSN, pp. 595–604 2002. P. Barham, et al., Magpie: Online Modelling and Performance-aware Systems, HotOS, pp. 85–90 2003. R. Fonseca, et al., X-Trace: A Pervasive Network Tracing Framework, USENIX Conference on NSDI, pp. 20–20 2007. B. H. Sigelman, et al., Dapper, a Large-Scale Distributed Systems Tracing Infrastructure, Technical report, Google 2010.

13 ɾLinuxͷύέοτϑΟϧλ(iptables)Λར༻ͯ͠ɼτϥϯεϙʔτ૚ ઀ଓΛݕग़ ɾiptablesͰ઀ଓ։࢝ϩάΛऩू͠ɼϩάίϨΫλͰதԝͷσʔλ ϕʔεʹอଘ ɾLinuxΧʔωϧͷL4௨৴ػߏΛར༻͢ΔݶΓɼґଘΛ௥੻Մೳ ίωΫγϣϯϕʔεΞϓϩʔν [Clawson 15] [Clawson 15] J. K. Clawson, Service Dependency Analysis via TCP/UDP Port Tracing, Master’s thesis, Brigham Young University-Provo 2015.

14 ɾωοτϫʔΫ૚ͷύέοτΛ΋ͱʹґଘΛ௥੻͢Δ ɾطଘͷτϥώοΫ͔ΒύέοτΛऩू͠ɼύέοτϔομ্ͷIPΞ υϨεͱϙʔτɼύέοτͷૹड৴ͷ࣌ࠁͳͲͷ৘ใΛղੳ͢Δ ɾ֤ܦ࿏ͰτϥώοΫྲྀྔʹ͕࣌ؒࠩ͋Δ͜ͱʹண໨͠ɼ֤ܦ࿏Ͱτ ϥώοΫੑ࣭ʹ૬͕ؔ͋ΔύλʔϯΛൃݟ͠ɼґଘΛਪఆ͢Δ ύέοτϕʔεΞϓϩʔν P. Bahl, et.al.: Towards Highly Reliable Enterprise Network Services via Inference of Multi-Level Dependencies, ACM SIGCOMM Review, Vol. 37, No. 4, pp.13–24 2007. X. Chen, et.al.: Automating Network Application Dependency Discovery: Experiences, Limitations, and New Solutions, USENIX Symposium on OSDI, pp.117–130 2008. P. Lucian, etl.al.: Macroscope: End-Point Approach to Networked Application Dependency Discovery, CoNEXT, pp.229–240 2009. A. Natarajan, et.al.: NSDMiner: Automated Discovery of Network Service Dependencies, IEEE INFOCOM, pp. 2507–2515 2012. A. Zand, et.al.: Rippler: Delay Injection for Service Dependency Detection, IEEE INFOCOM, pp. 2157–2165 2014.

15 ɾϦΫΤετϕʔεɾίωΫγϣϯϕʔε ɾ஗ԆΦʔόϔου: ௨৴ܦ࿏தʹ௥ՃͷॲཧΛڬΈࠐΉΦʔόϔου ɾίωΫγϣϯϕʔεɾύέοτϕʔε ɾِӄੑ: ύέοτͷαϯϓϦϯάʹΑΓྲྀྔͷখ͍͞઀ଓΛݕ஌Ͱ ͖ͳ͍ ɾ௥੻୯Ґ: OSϓϩηεͰ͸ͳ͘IPΞυϨε୯ҐͱͳΔ ɾϦΫΤετϕʔε ɾܭଌ४උίετ: طଘͷΞϓϦέʔγϣϯ಺ʹ௥੻ॲཧΛ௥Ճ͢Δ खؒ ֤Ξϓϩʔνͷ՝୊

3. ఏҊख๏

17 ఏҊख๏: ιέοτϕʔεΞϓϩʔν Linux OS Kernel Process Process TCP/UDP Flows … . . . User ιέοτ Λ؂ࢹ ɾιέοτͷ؂ࢹ͸ϓϩηεͷ ௨৴ͱ͸ಠཱ͍ͯ͠ΔͨΊɼ ஗ԆΦʔόϔουΛ௿ݮՄೳ ɾ઀ଓ࠶ར༻࣌ʹ͸͢΂ͯͷι έοτΛ௿ෛՙͰ؂ࢹՄೳ ɾιέοτʹΑΓɼϓϩηεͱ ઀ଓͷඥ෇͚͕Մೳ ɾϓϩηε͕ιέοτΛར༻͢ ΔݶΓಁաతʹ௥੻Մೳ

18 TranstracerͷγεςϜߏ੒ Host 1 Host 2 Host N CMDB Tracer Tracer Tracer Systems Administrator ɾϗετ্ʹTracerϓϩηεΛ഑ஔ ɾ֤Tracerϓϩηε͸औಘͨ͠઀ଓ৘ ใΛCMDB(઀ଓ৘ใ؅ཧDB)ʹอଘ ɾγεςϜ؅ཧऀ͸CMDBʹΞΫηε ͠ɼෳ਺ͷϗετʹ·͕ͨΓґଘؔ ܎Λऔಘ

19 ઀ଓ৘ใͷऔಘ Host Kernel Process Process TCP/UDP Flows … Tracer Polling ɾTracerϓϩηε͕LinuxΧʔωϧʹ໰͍߹Θ ͤɼTCP/UDPιέοτ৘ใΛϙʔϦϯάऔಘ ɾ઀ଓΛऴ୺͢ΔOSϓϩηε৘ใ΋͋Θͤͯ औಘ ɾιέοτ৘ใ: /proc/net/tcp΍Netlink sock_diag ɾϓϩηε৘ใ: /proc//{stat,fd} . . . ॲཧʹհೖ͠ͳ͍ͨΊ ௿Φʔόʔϔου

20 Transtracerͷར༻ྫ $ ttctl --dbhost 10.0.0.20 --ipv4 10.0.0.10 10.0.0.10:80 (’nginx’, pgid=4656) ᵋ<-- 10.0.0.11:many (’wrk’, pgid=5982) 10.0.0.10:80 (’nginx’, pgid=4656) ᵋ--> 10.0.0.12:8080 (’python’, pgid=6111) 10.0.0.10:many (’fluentd’, pgid=2127) ᵋ--> 10.0.0.13:24224 (’fluentd’, pgid=2001) 10.0.0.10 nginx 10.0.0.11 wrk 10.0.0.12 python 10.0.0.13 fluentd :80 fluentd :8080 :24224

21 ϓϩηεͷσʔλߏ଄ ᶃ LinuxͷϓϩηεάϧʔϓΛ ϊʔυͷ࠷খ୯Ґͱ͢Δ ᵓᴷnginx,627,627 ᴹ ᵓᴷnginx,628,627 ᴹ ᵋᴷnginx,629,627 $ pstree -apg | grep nginx ᶄ (ipv4, pgid, pname)Ͱϓϩηε ʹҰҙ੍໿Λ͔͚Δ ϓϩηε͸࠶ىಈ͢ΔͱID͕มԽ ͯ͠͠·͏ͨΊɼ໰͍߹Θͤ࣌ʹ ͸pgid͕ҟͳΔ΋ͷΛॏෳഉআ

22 ઀ଓ؅ཧͷͨΊͷσʔλߏ଄ ᶅ ActiveͱPassiveʹϊʔυΛ෼ྨ ᶆ Active => PassiveͷϑϩʔΛอଘ Active Passive Process Passive Active Port N Port M Active ಉҰϓϩηε͕Activeʹ΋ Passiveʹ΋ͳΓ͑Δ ಉҰϓϩηε͕ෳ਺ͷϙʔτ ΛϦοεϯ͢Δ͜ͱ͕͋Δ

23 ɾIPΞυϨεͱϙʔτ൪߸͕௨৴ܦ࿏தʹมߋ͞ΕΔͱϓϩηεؒͷඥ ෇͚͕Ͱ͖ͳ͘ͳΔ ɾϩʔυόϥϯαʹΑΔDNATͳͲ ɾNATͷηογϣϯΛಡΈऔΓɼมߋલޙͷΞυϨεͱϙʔτͷඥ෇ ͚Λߦ͏͜ͱ͸Մೳ ɾLinuxͷINETιέοτΛར༻͠ͳ͍ΞϓϦέʔγϣϯͷ઀ଓΛ௥੻Ͱ ͖ͳ͍ ɾDPDK΍netmapͳͲͷΧʔωϧΛόΠύεͯ͠௨৴͢Δٕज़ ఏҊख๏ͷ੍໿

4. ࣮ݧͱߟ࡯

25 1. ΞϓϦέʔγϣϯʹ༩͑ΔԠ౴஗ԆΦʔόϔουͷධՁ 2. ઀ଓ৘ใऔಘͷͨΊͷCPUར༻཰ΦʔόϔουͷධՁ ධՁ࣮ݧ ఏҊख๏ͱಛ௃͕ࣅ͍ͯΔίωΫγϣϯϕʔεΞϓϩʔνͷઌߦख๏[1] ͱ֤ΦʔόϔουΛൺֱ [1] J. K. Clawson, Service Dependency Analysis via TCP/UDP Port Tracing, Master’s thesis, Brigham Young University-Provo 2015.

26 ࣮ݧ؀ڥͷߏ੒ CMDB Client Server HTTP benchmarker HTTP server Tracer

27 ࣮ݧ؀ڥͷৄࡉ ߲໨ ࢓༷ Client CPU Memory Benchmarker Intel Xeon CPU E5-2650 v3 2.30GHz 2core 1 GB wrk 4.1.0-4 Server CPU Memory HTTP Server Intel Xeon CPU E5-2650 v3 2.30GHz 4core 1GB nginx 1.17.3 CMDB CPU Memory Database Intel Xeon CPU E5-2650 v3 2.30GHz 1core 1 GB PostgreSQL 11.3 ɾΠϯελϯε͸͢΂ͯ͘͞ΒͷΫϥ΢υ্ʹߏங ɾLinux Kernel 4.15 (Ubuntu Server 18.04.3 LTS)

28 1. Normal: ௥੻ॲཧ͕ͳ͍ঢ়ଶ 2. Transtracer: ఏҊख๏ (https://github.com/yuuki/transtracer v0.1.0) ɾϙʔϦϯάִؒ͸1ඵ 3. iptables NEWϑΟϧλํࣜ: ৽ن઀ଓͷΈϩάΛग़ྗ 4. iptables ESTBϑΟϧλํࣜ: ઀ଓཱ֬தʹ΍ΓͱΓ͞ΕΔύέοτ ͷϩάΛαϯϓϦϯάͤͣʹग़ྗ ɾઌߦख๏ͷແ࡞ҝͳαϯϓϦϯάͰ͸ɼ௕໋ͳ઀ଓͰ͋ͬͯ΋ྲྀ ྔ͕খ͍͞઀ଓΛݟಀ͢Մೳੑ͕͋Δ ࣮ݧʹ༻͍Δ֤࣮૷

29 Ԡ౴஗ԆΦʔόʔϔου 50 100 150 200 250 300 350 400 450 500 5000 10000 15000 20000 Average Latency (ms) Connections Normal 93.1 191.6 279.3 353.8 Transtracer 94.7 188.3 291.8 401.2 ESTB filter 115.0 236.0 359.0 462.5 NEW filter 113.1 214.4 310.0 449.3 ɾNormalʹରͯ͠transtracer ͕1.7~13.4%ͷΦʔόϔο υ૿ ɾiptables࣮૷ͷESTBϑΟϧ λํࣜʹରͯ͠ɼ transtracer͕13-20%ͷ Φʔόϔουݮগ

30 CPUར༻཰Φʔόʔϔου 0 10 20 30 40 50 60 70 80 90 100 5000 10000 15000 20000 0 50 100 150 200 250 300 350 400 450 500 CPU usage (%) Reading sockets time(ms) Connections ttracerd’s CPU usage 13.2 23.0 34.2 44.4 ESTB filter’s CPU usage 72.2 75.9 78.8 78.6 Reading sockets time 102.3 199.1 317.8 408.6 ɾ20,000઀ଓʹ͓͍ͯɼ TranstracerͷCPUར༻཰ 44.4%ɼESTBϑΟϧλํ ࣜͷCPUར༻཰͸78.6% ɾ43.5%ͷCPUར༻཰ͷ௿ ݮ

31 ɾ2ສ઀ଓ࣌ʹϙʔϦϯάؒ ִΛ૿Ճͤ͞ΔͱCPUར༻ ཰͕Ͳͷఔ౓௿ݮ͞ΕΔ͔ ɾ5ඵҎ಺ͷ୹໋ͳ઀ଓΛݕ ग़Ͱ͖ͳ͘ͳΔՄೳੑ͕͋ Δ͔ΘΓʹɼ8.6%·Ͱ௿ݮ Մೳ ϙʔϦϯάִؒͱCPUར༻཰ͷؔ܎ 0 5 10 15 20 25 30 35 40 45 50 55 1 2 3 4 5 CPU usage (%) Polling interval CPU usage 44.4 21.6 13.0 10.8 8.6

32 ɾԠ౴஗ԆΦʔόϔου ɾઌߦख๏ͷ2ͭͷϑΟϧλํࣜͱൺֱ͠ɼఏҊख๏͕༗ར ɾCPUར༻཰Φʔόϔου ɾNEWϑΟϧλํ͕ࣜɼॳճͷ઀ଓཱ֬࣌ͷΈCPUΛར༻͢Δͨ Ίɼ઀ଓͷ࠶ར༻؀ڥͰ͸༗ར ɾͨͩ͠ɼఏҊख๏͸ϙʔϦϯάִؒͷௐ੔ʹΑΓCPUར༻཰ͷ௿ ݮ͸Մೳ ઌߦख๏ͱͷൺֱߟ࡯

33 ɾ઀ଓΛ࠶ར༻͢Δ؀ڥͰ͸ɼ୹໋ͳ઀ଓΛݕ஌Ͱ͖ͳ͘ͱ΋࣮༻ ্໰୊ʹͳΒͳ͍ ɾϙʔϦϯάִؒΛCPUར༻཰͕े෼খ͘͞ͳΔ·ͰௐઅՄೳ ɾϦΞϧλΠϜੑͷ؍఺ͰɼϙʔϦϯάִ͕ؒ਺ඵ୯ҐͰ͋Ε͹ɼ ਓؒͷ൑அ଎౓ʹ͸Өڹ͠ͳ͍ ࣮؀ڥͰͷར༻Λ૝ఆͨ͠ߟ࡯

5. ·ͱΊͱࠓޙͷల๬

35 ɾγεςϜมߋ࣌ͷϦεΫΛ༧ଌ͢Δ͜ͱΛ໨తʹɼϓϩηεؒґଘ ؔ܎Λ௥੻͢ΔΞʔΩςΫνϟ Transtracer ΛఏҊͨ͠ ɾιέοτ؂ࢹʹΑΓɼΦʔόʔϔουΛ௿ݮͤͭͭ͞ɼϓϩηεؒ ͷґଘؔ܎Λ໢ཏతʹݕग़Մೳ ɾ઀ଓΛ࠶ར༻͢Δ͜ͱͷଟ͍WebαʔϏεͷ෼ࢄγεςϜʹ͓͍ ͯɼ࣮༻ՄೳͳϨϕϧͷΦʔόʔϔουͱݕग़ਫ਼౓ΛಘΒΕͨ ·ͱΊ

36 ɾ͢΂ͯͷ઀ଓΛ௥੻͢ΔલఏͰɼ͍͔ʹΦʔόʔϔουΛ௿ݮͤ͞ ͍͔ͯ͘Λݚڀ։ൃ͍ͯ͘͠ ɾLinuxΧʔωϧ಺ͷॲཧΛ֦ுՄೳͳeBPFΛར༻͠ɼconnectͱ acceptΠϕϯτͷΈΛετϦʔϛϯάͰ௥੻͢Δ ɾUDP௨৴ͷ௥੻ͷରԠ ɾKubernetesͳͲͷීٴ͍ͯ͠ΔΞϓϦέʔγϣϯ࣮ߦج൫্Ͱಈ࡞ ͤ͞Δ ɾΤοδίϯϐϡʔςΟϯάΛݟਾ͑ͯCMDBͷ෼ࢄ഑ஔ ࠓޙͷల๬

ิ଍ࢿྉ

38 ɾιέοτͷϙʔϦϯά ɾ਺ඵҎԼͷ୹໋ͳ઀ଓΛݟಀ͢Մೳੑ͕͋Δ ɾWebαʔϏεͷΑ͏ͳ઀ଓ࠶ར༻؀ڥͰ͋Ε͹໰୊ͳ͍ ɾύέοτͷαϯϓϦϯά ɾ௕໋ͳ઀ଓͰ͋ͬͯ΋ɼݟಀ͢Մೳੑ͕͋Δ ɾಉҰϗετ্Ͱྲྀྔͷେ͖͍઀ଓͱྲྀྔͷখ͍͞઀ଓ͕ࠞ߹ͯ͠ ͍ΔͱޙऀΛݟಀ͠΍͍͢ ɾγεςϜ؅ཧऀ͕Ͳͷ઀ଓΛݟಀ͔͢Λ༧ଌͮ͠Β͍ ِӄੑͷൺֱ: ϙʔϦϯάͱαϯϓϦϯά

39 ɾϙʔϦϯά͕ݟಀ͢୹໋ͳ ઀ଓΛݕग़͢ΔͨΊʹετ ϦʔϛϯάΛ૊Έ߹ΘͤΔ ɾeBPFʹΑΓɼconnect(2)ͱ accept(2)ΠϕϯτΛऔಘ ͠ɼϑϩʔ৘ใΛऔಘ͢Δ ɾUDPͷ৔߹͸send_msg(2), recv_msg(2)Πϕϯτ ετϦʔϛϯάʹΑΔ઀ଓͷݕग़ Linux Host Kernel Process Process TCP/UDP Flows … . . . User Streaming Tracer

40 ɾiovisor/bcc಺ͷtcpacceptͰɼඇӬଓԽ؀ڥͰͷෛՙ࣮ݧ ɾwrk (HTTP KeepAlive off)Ͱಉ࣌઀ଓ1000ͰnginxʹϕϯνϚʔΫ ɾCPUར༻཰͸45~50%/coreఔ౓ ɾԠ౴஗ԆͷΦʔόϔου͸༗ҙͳѱԽ͸ݟΒΕͳ͔ͬͨ ɾeBPFͷΠϕϯτΛ͢΂ͯϢʔβʔϥϯυʹίϐʔ͍ͯ͠Δ͜ͱ͕ CPUෛՙ͕ߴ͍ཁҼ͔΋͠Εͳ͍ eBPFͷετϦʔϛϯάෛՙͷ༧උ࣮ݧ ৄࡉ͸࣮ݧϊʔτ΁ https://www.notion.so/yuuk1/iovisor-bcc-tcpconnect-tcpaccept-af2d1fdce35c49fb945b548db373213d

41 ɾ͢΂ͯͷ઀ଓ৘ใΛऩू͢ΔͱɼCMDBʹ֨ೲ͢Δσʔλྔ͕େ͖͘ ͳΔͨΊɼ৑௕ͳ৘ใΛ࡟ݮ͢Δ ɾΤϑΣϝϥϧϙʔτ: Χʔωϧ͔ΒׂΓ౰ͯΒΕΔϥϯμϜͳૹ৴ݩ ϙʔτ ɾಛఆͷLISTENϙʔτ΁ෳ਺ͷΤϑΣϝϥϧϙʔτ͔Β઀ଓ͞ΕΔ ɾ͜ΕΒͷ઀ଓΛू໿͠ɼ୯Ұͷ઀ଓͱΈͳ͢ ΤϑΣϝϥϧϙʔτͷू໿ Host Port Process Port Port Host Port Process ୯Ұͷ઀ଓ ͱͯ͠ू໿ LISTEN ΤϑΣϝϥϧ