Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Transtracer: 分散システムにおけるTCP/UDP通信の終端点の監視によるプロセス間依存関係の自動追跡

Transtracer: 分散システムにおけるTCP/UDP通信の終端点の監視によるプロセス間依存関係の自動追跡

情報処理学会 第12回インターネットと運用技術シンポジウム(IOTS)2019
論文: https://yuuk.io/papers/transtracer_iots2019.pdf
OSS: https://github.com/yuuki/transtracer

Yuuki Tsubouchi (yuuk1)

December 06, 2019
Tweet

More Decks by Yuuki Tsubouchi (yuuk1)

Other Decks in Research

Transcript

  1. ͘͞ΒΠϯλʔωοτ
    גࣜձࣾ
    (C) Copyright 1996-2019 SAKURA internet Inc
    ͘͞ΒΠϯλʔωοτ
    ݚڀॴ
    Transtracer: ෼ࢄγεςϜʹ͓͚ΔTCP/UDP௨৴ͷ
    ऴ୺఺ͷ؂ࢹʹΑΔϓϩηεؒґଘؔ܎ͷࣗಈ௥੻
    ৘ใॲཧֶձ ୈ12ճΠϯλʔωοτͱӡ༻ٕज़γϯϙδ΢Ϝ(IOTS)2019
    2019.12.6
    ௶಺༎थ*1, ݹ઒խେ*2, দຊ྄հ*1
    *1) ͘͞ΒΠϯλʔωοτ, *2) ͸ͯͳ

    View full-size slide

  2. 2
    1. എܠͱ໨త
    2. ෼ࢄγεςϜ಺ͷґଘؔ܎ͷ௥੻ٕज़
    3. ఏҊख๏
    4. ࣮ݧͱߟ࡯
    5. ·ͱΊͱࠓޙͷల๬
    ໨࣍

    View full-size slide

  3. 1.
    എܠͱ໨త

    View full-size slide

  4. 4
    WebαʔϏε಺ͷґଘؔ܎ͷෳࡶԽͷഎܠ
    ௕ظؒͷαʔϏεఏڙத
    ͷػೳ௥Ճ
    Ϣʔβʔ͔ΒͷΞΫηε
    ૿Ճ
    ୯ҰͷαʔϏεࣄۀऀ͕
    ෳ਺ͷαʔϏεΛఏڙ
    (SNS,ECαΠτͳͲ)
    ༻్ಛԽͷϛυϧ΢ΣΞͷ௥Ճ
    ৽چγεςϜͷࠞ߹ঢ়ଶ
    εέʔϧΞ΢τʹΑΔϗετ਺
    ͷ૿Ճ
    ෳ਺ͷαʔϏεͷҰ෦Λڞ༗
    (Ϣʔβʔೝূج൫ͳͲ)
    ϚΠΫϩαʔϏεԽ

    View full-size slide

  5. 5
    ɾෳࡶͳґଘؔ܎ʹΑΓɼγεςϜʹมߋΛՃ͑ͨ࣌ʹ໰୊͕͋ͬͨ
    ৔߹ɼ༧૝ΑΓେ͖ͳো֐ʹͭͳ͕Δ͜ͱ΋͋Δ
    ɾো֐ൃੜΛڪΕͯɼௐࠪʹ࣌ؒΛཁ͠ɼ݁Ռతʹมߋ଎౓͕௿Լ
    ɾґଘؔ܎͸γεςϜͷՔಇதʹมߋ͞Ε͍ͯͨ͘ΊɼखಈͰυΩϡ
    ϝϯτΛߋ৽͠ଓ͚Δख͕ؒେ͖͍
    ґଘؔ܎ͷෳࡶԽʹΑΓੜ·ΕΔ໰୊ҙࣝ
    มߋͷӨڹൣғΛ༧ଌ͢ΔͨΊʹ
    γεςϜ؅ཧऀʹͱͬͯະ஌ͷϓϩηεͷґଘؔ܎Λ
    ࣗಈͰ௥੻͢Δඞཁ͕͋Δ

    View full-size slide

  6. 6
    ɾLinuxͷύέοτϑΟϧλϩάΛར༻ͯ͠L4઀ଓͷঢ়ଶΛऔಘ
    ɾLinuxΧʔωϧͷ௨৴ػߏΛར༻͢ΔݶΓɼOSϓϩηεʹରͯ͠ಁ
    աతʹ௥੻Մೳ
    ɾ஗ԆΦʔόϔου: ύέοτϑΟϧλॲཧͷΦʔόϔου
    ɾِӄੑ: ϦιʔεফඅΛ཈͑ΔͨΊʹɼύέοτΛແ࡞ҝʹαϯϓϦ
    ϯά͢ΔͨΊɼݕ஌࿙ΕͷՄೳੑ͕͋Δ
    طଘͷࣗಈ௥੻ٕज़ [1]
    [1] J. K. Clawson, Service Dependency Analysis via TCP/UDP Port Tracing, Master’s thesis, Brigham Young University-Provo 2015.

    View full-size slide

  7. 7
    ݚڀ໨త: ະ஌ͷϓϩηεΛ௿ෛՙͰ௥੻
    Linux OS
    Kernel
    Process Process
    TCP/UDP
    Flows

    .
    .
    .
    User
    ιέοτ
    Λ؂ࢹ
    TCP/UDP઀ଓͷऴ୺఺Ͱ͋ΔιέοτΛ؂ࢹ͠઀ଓΛࣗಈ௥੻
    1. ஗ԆΦʔόϔουͷղܾ
    ɾιέοτͷ؂ࢹΛϓϩηε
    ͷ௨৴ͱಠཱͤ͞Δ
    2. ِӄੑͷղܾ
    ɾ͢΂ͯͷιέοτΛ؂ࢹ
    ɾύέοτ͸ඵؒ਺ສ୯Ґ͕
    ͋Γ͑Δ
    ɾιέοτ͸࠶ར༻͞Ε͏Δ
    ͨΊɼιέοτ਺ͷ΄͏͕
    খ͘͞ͳΓ΍͍͢

    View full-size slide

  8. 8
    γεςϜ؅ཧऀʹͱͬͯ఍߅ͳ͘ಋೖՄೳ
    ɾΦʔόϔου͕େ͖͍ͱɼ؅ཧऀ͸Өڹͷେ͖͍ίϯϙʔωϯτΛ
    ௥੻ର৅͔Β֎͢͜ͱ͕͋Δ
    ↪ ؅ཧऀͷهԱ΍υΩϡϝϯςʔγϣϯʹґଘ͢Δ
    ɾِӄੑ͕େ͖͍ͱɼ؅ཧऀ͸ಘΒΕͨґଘؔ܎Λ৴༻͠ͳ͘ͳΔ
    ↪ ؅ཧऀ͕खಈͰґଘؔ܎Λௐࠪ͢Δ
    ௿͍Φʔόʔϔουͱ௿ِ͍ӄੑʹΑΓ
    γεςϜ؅ཧऀ͕ར༻͢Δͱ͖ͷ৺ཧత఍߅Λ௿ݮՄೳ

    View full-size slide

  9. 2.
    ෼ࢄγεςϜ಺ͷ
    ґଘؔ܎ͷ௥੻ٕज़

    View full-size slide

  10. 10
    ɾWebαʔϏεͷ෼ࢄγεςϜ಺ͷґଘؔ܎Λ௥੻Մೳ
    ɾWebαʔϏεͰ͸ɼΞΫηε਺͕େ͖͘ɼԠ౴଎౓͕1ඵҎԼͰ͋
    Δ͜ͱ͕ಛ௃
    ɾ઀ଓΛ࠶ར༻͠ɼ઀ଓ͝ͱͷෛՙΛ௿ݮ͍ͤͯ͞Δ
    ɾσʔληϯλʔ಺௨৴ͰHTTP/2ͷར༻͕૿େ
    ɾίωΫγϣϯϓʔϦϯάʹΑΓσʔλϕʔε΁ͷ઀ଓΛ࠶ར༻
    ɾτϥϯεϙʔτ૚Ͱ͸ɼ௕໋ͳ઀ଓͱͳΔ܏޲͕͋Δ
    ຊݚڀͷ௥੻ର৅ͷલఏ

    View full-size slide

  11. 11
    ෼ࢄγεςϜ಺ͷґଘؔ܎ͷ௥੻ٕज़ͷ෼ྨ
    ϦΫΤετϕʔε
    Ξϓϩʔν
    ΞϓϦέʔγϣϯ૚ͷϦΫΤετ͕
    Ͳͷ௨৴ܦ࿏ΛͨͲΔ͔Λ௥੻͢Δख๏
    ίωΫγϣϯϕʔε
    Ξϓϩʔν
    ऴ୺ϗετ্Ͱτϥϯεϙʔτ઀ଓͷঢ়ଶΛ
    औಘ͢Δ͜ͱʹΑΓґଘؔ܎Λ௥੻͢Δख๏
    ύέοτϕʔε
    Ξϓϩʔν
    ϗετ্΍εΠον্ͰύέοτϔομΛ؍ଌ
    ͢Δ͜ͱʹΑΓɼ ґଘؔ܎Λ௥੻͢Δख๏

    View full-size slide

  12. 12
    ɾΞϓϦέʔγϣϯ૚ͷ֤ϦΫΤετʹࣝผࢠΛׂΓৼΓɼޙଓͷϦ
    ΫΤετʹຒΊࠐΜ্ͩͰɼޙଓͷϓϩηε΁఻ൖͤ͞Δ
    ɾࣝผࢠΛཔΓʹɼϦΫΤετ͕γεςϜ಺ͷͲͷϓϩηεΛܦ༝͠
    ͯॲཧ͞Ε͔ͨΛ௥੻
    ϦΫΤετϕʔεΞϓϩʔν
    M. Y. Chen, et al., Pinpoint: Problem Determination in Large, Dynamic Internet Services, IEEE/IFIP International Conference on DSN, pp. 595–604 2002.
    P. Barham, et al., Magpie: Online Modelling and Performance-aware Systems, HotOS, pp. 85–90 2003.
    R. Fonseca, et al., X-Trace: A Pervasive Network Tracing Framework, USENIX Conference on NSDI, pp. 20–20 2007.
    B. H. Sigelman, et al., Dapper, a Large-Scale Distributed Systems Tracing Infrastructure, Technical report, Google 2010.

    View full-size slide

  13. 13
    ɾLinuxͷύέοτϑΟϧλ(iptables)Λར༻ͯ͠ɼτϥϯεϙʔτ૚
    ઀ଓΛݕग़
    ɾiptablesͰ઀ଓ։࢝ϩάΛऩू͠ɼϩάίϨΫλͰதԝͷσʔλ
    ϕʔεʹอଘ
    ɾLinuxΧʔωϧͷL4௨৴ػߏΛར༻͢ΔݶΓɼґଘΛ௥੻Մೳ
    ίωΫγϣϯϕʔεΞϓϩʔν [Clawson 15]
    [Clawson 15] J. K. Clawson, Service Dependency Analysis via TCP/UDP Port Tracing, Master’s thesis, Brigham Young University-Provo 2015.

    View full-size slide

  14. 14
    ɾωοτϫʔΫ૚ͷύέοτΛ΋ͱʹґଘΛ௥੻͢Δ
    ɾطଘͷτϥώοΫ͔ΒύέοτΛऩू͠ɼύέοτϔομ্ͷIPΞ
    υϨεͱϙʔτɼύέοτͷૹड৴ͷ࣌ࠁͳͲͷ৘ใΛղੳ͢Δ
    ɾ֤ܦ࿏ͰτϥώοΫྲྀྔʹ͕࣌ؒࠩ͋Δ͜ͱʹண໨͠ɼ֤ܦ࿏Ͱτ
    ϥώοΫੑ࣭ʹ૬͕ؔ͋ΔύλʔϯΛൃݟ͠ɼґଘΛਪఆ͢Δ
    ύέοτϕʔεΞϓϩʔν
    P. Bahl, et.al.: Towards Highly Reliable Enterprise Network Services via Inference of Multi-Level Dependencies, ACM SIGCOMM Review, Vol. 37, No. 4,
    pp.13–24 2007.
    X. Chen, et.al.: Automating Network Application Dependency Discovery: Experiences, Limitations, and New Solutions, USENIX Symposium on OSDI,
    pp.117–130 2008.
    P. Lucian, etl.al.: Macroscope: End-Point Approach to Networked Application Dependency Discovery, CoNEXT, pp.229–240 2009.
    A. Natarajan, et.al.: NSDMiner: Automated Discovery of Network Service Dependencies, IEEE INFOCOM, pp. 2507–2515 2012.
    A. Zand, et.al.: Rippler: Delay Injection for Service Dependency Detection, IEEE INFOCOM, pp. 2157–2165 2014.

    View full-size slide

  15. 15
    ɾϦΫΤετϕʔεɾίωΫγϣϯϕʔε
    ɾ஗ԆΦʔόϔου: ௨৴ܦ࿏தʹ௥ՃͷॲཧΛڬΈࠐΉΦʔόϔου
    ɾίωΫγϣϯϕʔεɾύέοτϕʔε
    ɾِӄੑ: ύέοτͷαϯϓϦϯάʹΑΓྲྀྔͷখ͍͞઀ଓΛݕ஌Ͱ
    ͖ͳ͍
    ɾ௥੻୯Ґ: OSϓϩηεͰ͸ͳ͘IPΞυϨε୯ҐͱͳΔ
    ɾϦΫΤετϕʔε
    ɾܭଌ४උίετ: طଘͷΞϓϦέʔγϣϯ಺ʹ௥੻ॲཧΛ௥Ճ͢Δ
    खؒ
    ֤Ξϓϩʔνͷ՝୊

    View full-size slide

  16. 3.
    ఏҊख๏

    View full-size slide

  17. 17
    ఏҊख๏: ιέοτϕʔεΞϓϩʔν
    Linux OS
    Kernel
    Process Process
    TCP/UDP
    Flows

    .
    .
    .
    User
    ιέοτ
    Λ؂ࢹ
    ɾιέοτͷ؂ࢹ͸ϓϩηεͷ
    ௨৴ͱ͸ಠཱ͍ͯ͠ΔͨΊɼ
    ஗ԆΦʔόϔουΛ௿ݮՄೳ
    ɾ઀ଓ࠶ར༻࣌ʹ͸͢΂ͯͷι
    έοτΛ௿ෛՙͰ؂ࢹՄೳ
    ɾιέοτʹΑΓɼϓϩηεͱ
    ઀ଓͷඥ෇͚͕Մೳ
    ɾϓϩηε͕ιέοτΛར༻͢
    ΔݶΓಁաతʹ௥੻Մೳ

    View full-size slide

  18. 18
    TranstracerͷγεςϜߏ੒
    Host 1
    Host 2
    Host N
    CMDB
    Tracer
    Tracer
    Tracer
    Systems
    Administrator
    ɾϗετ্ʹTracerϓϩηεΛ഑ஔ
    ɾ֤Tracerϓϩηε͸औಘͨ͠઀ଓ৘
    ใΛCMDB(઀ଓ৘ใ؅ཧDB)ʹอଘ
    ɾγεςϜ؅ཧऀ͸CMDBʹΞΫηε
    ͠ɼෳ਺ͷϗετʹ·͕ͨΓґଘؔ
    ܎Λऔಘ

    View full-size slide

  19. 19
    ઀ଓ৘ใͷऔಘ
    Host
    Kernel
    Process Process
    TCP/UDP
    Flows

    Tracer
    Polling
    ɾTracerϓϩηε͕LinuxΧʔωϧʹ໰͍߹Θ
    ͤɼTCP/UDPιέοτ৘ใΛϙʔϦϯάऔಘ
    ɾ઀ଓΛऴ୺͢ΔOSϓϩηε৘ใ΋͋Θͤͯ
    औಘ
    ɾιέοτ৘ใ: /proc/net/tcp΍Netlink sock_diag
    ɾϓϩηε৘ใ: /proc//{stat,fd}
    .
    .
    .
    ॲཧʹհೖ͠ͳ͍ͨΊ
    ௿Φʔόʔϔου

    View full-size slide

  20. 20
    Transtracerͷར༻ྫ
    $ ttctl --dbhost 10.0.0.20 --ipv4 10.0.0.10
    10.0.0.10:80 (’nginx’, pgid=4656)
    ᵋ<-- 10.0.0.11:many (’wrk’, pgid=5982)
    10.0.0.10:80 (’nginx’, pgid=4656)
    ᵋ--> 10.0.0.12:8080 (’python’, pgid=6111)
    10.0.0.10:many (’fluentd’, pgid=2127)
    ᵋ--> 10.0.0.13:24224 (’fluentd’, pgid=2001)
    10.0.0.10
    nginx
    10.0.0.11
    wrk
    10.0.0.12
    python
    10.0.0.13
    fluentd
    :80
    fluentd
    :8080
    :24224

    View full-size slide

  21. 21
    ϓϩηεͷσʔλߏ଄
    ᶃ LinuxͷϓϩηεάϧʔϓΛ
    ϊʔυͷ࠷খ୯Ґͱ͢Δ
    ᵓᴷnginx,627,627
    ᴹ ᵓᴷnginx,628,627
    ᴹ ᵋᴷnginx,629,627
    $ pstree -apg | grep nginx
    ᶄ (ipv4, pgid, pname)Ͱϓϩηε
    ʹҰҙ੍໿Λ͔͚Δ
    ϓϩηε͸࠶ىಈ͢ΔͱID͕มԽ
    ͯ͠͠·͏ͨΊɼ໰͍߹Θͤ࣌ʹ
    ͸pgid͕ҟͳΔ΋ͷΛॏෳഉআ

    View full-size slide

  22. 22
    ઀ଓ؅ཧͷͨΊͷσʔλߏ଄
    ᶅ ActiveͱPassiveʹϊʔυΛ෼ྨ
    ᶆ Active => PassiveͷϑϩʔΛอଘ
    Active Passive
    Process
    Passive
    Active
    Port N
    Port M
    Active
    ಉҰϓϩηε͕Activeʹ΋
    Passiveʹ΋ͳΓ͑Δ
    ಉҰϓϩηε͕ෳ਺ͷϙʔτ
    ΛϦοεϯ͢Δ͜ͱ͕͋Δ

    View full-size slide

  23. 23
    ɾIPΞυϨεͱϙʔτ൪߸͕௨৴ܦ࿏தʹมߋ͞ΕΔͱϓϩηεؒͷඥ
    ෇͚͕Ͱ͖ͳ͘ͳΔ
    ɾϩʔυόϥϯαʹΑΔDNATͳͲ
    ɾNATͷηογϣϯΛಡΈऔΓɼมߋલޙͷΞυϨεͱϙʔτͷඥ෇
    ͚Λߦ͏͜ͱ͸Մೳ
    ɾLinuxͷINETιέοτΛར༻͠ͳ͍ΞϓϦέʔγϣϯͷ઀ଓΛ௥੻Ͱ
    ͖ͳ͍
    ɾDPDK΍netmapͳͲͷΧʔωϧΛόΠύεͯ͠௨৴͢Δٕज़
    ఏҊख๏ͷ੍໿

    View full-size slide

  24. 4.
    ࣮ݧͱߟ࡯

    View full-size slide

  25. 25
    1. ΞϓϦέʔγϣϯʹ༩͑ΔԠ౴஗ԆΦʔόϔουͷධՁ
    2. ઀ଓ৘ใऔಘͷͨΊͷCPUར༻཰ΦʔόϔουͷධՁ
    ධՁ࣮ݧ
    ఏҊख๏ͱಛ௃͕ࣅ͍ͯΔίωΫγϣϯϕʔεΞϓϩʔνͷઌߦख๏[1]
    ͱ֤ΦʔόϔουΛൺֱ
    [1] J. K. Clawson, Service Dependency Analysis via TCP/UDP Port Tracing, Master’s thesis, Brigham Young University-Provo 2015.

    View full-size slide

  26. 26
    ࣮ݧ؀ڥͷߏ੒
    CMDB
    Client Server
    HTTP benchmarker HTTP server
    Tracer

    View full-size slide

  27. 27
    ࣮ݧ؀ڥͷৄࡉ
    ߲໨ ࢓༷
    Client CPU
    Memory
    Benchmarker
    Intel Xeon CPU E5-2650 v3 2.30GHz 2core
    1 GB
    wrk 4.1.0-4
    Server CPU
    Memory
    HTTP Server
    Intel Xeon CPU E5-2650 v3 2.30GHz 4core
    1GB
    nginx 1.17.3
    CMDB CPU
    Memory
    Database
    Intel Xeon CPU E5-2650 v3 2.30GHz 1core
    1 GB
    PostgreSQL 11.3
    ɾΠϯελϯε͸͢΂ͯ͘͞ΒͷΫϥ΢υ্ʹߏங
    ɾLinux Kernel 4.15 (Ubuntu Server 18.04.3 LTS)

    View full-size slide

  28. 28
    1. Normal: ௥੻ॲཧ͕ͳ͍ঢ়ଶ
    2. Transtracer: ఏҊख๏ (https://github.com/yuuki/transtracer v0.1.0)
    ɾϙʔϦϯάִؒ͸1ඵ
    3. iptables NEWϑΟϧλํࣜ: ৽ن઀ଓͷΈϩάΛग़ྗ
    4. iptables ESTBϑΟϧλํࣜ: ઀ଓཱ֬தʹ΍ΓͱΓ͞ΕΔύέοτ
    ͷϩάΛαϯϓϦϯάͤͣʹग़ྗ
    ɾઌߦख๏ͷແ࡞ҝͳαϯϓϦϯάͰ͸ɼ௕໋ͳ઀ଓͰ͋ͬͯ΋ྲྀ
    ྔ͕খ͍͞઀ଓΛݟಀ͢Մೳੑ͕͋Δ
    ࣮ݧʹ༻͍Δ֤࣮૷

    View full-size slide

  29. 29
    Ԡ౴஗ԆΦʔόʔϔου
    50
    100
    150
    200
    250
    300
    350
    400
    450
    500
    5000 10000 15000 20000
    Average Latency (ms)
    Connections
    Normal
    93.1
    191.6
    279.3
    353.8
    Transtracer
    94.7
    188.3
    291.8
    401.2
    ESTB filter
    115.0
    236.0
    359.0
    462.5
    NEW filter
    113.1
    214.4
    310.0
    449.3
    ɾNormalʹରͯ͠transtracer
    ͕1.7~13.4%ͷΦʔόϔο
    υ૿
    ɾiptables࣮૷ͷESTBϑΟϧ
    λํࣜʹରͯ͠ɼ
    transtracer͕13-20%ͷ
    Φʔόϔουݮগ

    View full-size slide

  30. 30
    CPUར༻཰Φʔόʔϔου
    0
    10
    20
    30
    40
    50
    60
    70
    80
    90
    100
    5000 10000 15000 20000
    0
    50
    100
    150
    200
    250
    300
    350
    400
    450
    500
    CPU usage (%)
    Reading sockets time(ms)
    Connections
    ttracerd’s CPU usage
    13.2
    23.0
    34.2
    44.4
    ESTB filter’s CPU usage
    72.2
    75.9
    78.8 78.6
    Reading sockets time
    102.3
    199.1
    317.8
    408.6
    ɾ20,000઀ଓʹ͓͍ͯɼ
    TranstracerͷCPUར༻཰
    44.4%ɼESTBϑΟϧλํ
    ࣜͷCPUར༻཰͸78.6%
    ɾ43.5%ͷCPUར༻཰ͷ௿
    ݮ

    View full-size slide

  31. 31
    ɾ2ສ઀ଓ࣌ʹϙʔϦϯάؒ
    ִΛ૿Ճͤ͞ΔͱCPUར༻
    ཰͕Ͳͷఔ౓௿ݮ͞ΕΔ͔
    ɾ5ඵҎ಺ͷ୹໋ͳ઀ଓΛݕ
    ग़Ͱ͖ͳ͘ͳΔՄೳੑ͕͋
    Δ͔ΘΓʹɼ8.6%·Ͱ௿ݮ
    Մೳ
    ϙʔϦϯάִؒͱCPUར༻཰ͷؔ܎
    0
    5
    10
    15
    20
    25
    30
    35
    40
    45
    50
    55
    1 2 3 4 5
    CPU usage (%)
    Polling interval
    CPU usage
    44.4
    21.6
    13.0
    10.8
    8.6

    View full-size slide

  32. 32
    ɾԠ౴஗ԆΦʔόϔου
    ɾઌߦख๏ͷ2ͭͷϑΟϧλํࣜͱൺֱ͠ɼఏҊख๏͕༗ར
    ɾCPUར༻཰Φʔόϔου
    ɾNEWϑΟϧλํ͕ࣜɼॳճͷ઀ଓཱ֬࣌ͷΈCPUΛར༻͢Δͨ
    Ίɼ઀ଓͷ࠶ར༻؀ڥͰ͸༗ར
    ɾͨͩ͠ɼఏҊख๏͸ϙʔϦϯάִؒͷௐ੔ʹΑΓCPUར༻཰ͷ௿
    ݮ͸Մೳ
    ઌߦख๏ͱͷൺֱߟ࡯

    View full-size slide

  33. 33
    ɾ઀ଓΛ࠶ར༻͢Δ؀ڥͰ͸ɼ୹໋ͳ઀ଓΛݕ஌Ͱ͖ͳ͘ͱ΋࣮༻
    ্໰୊ʹͳΒͳ͍
    ɾϙʔϦϯάִؒΛCPUར༻཰͕े෼খ͘͞ͳΔ·ͰௐઅՄೳ
    ɾϦΞϧλΠϜੑͷ؍఺ͰɼϙʔϦϯάִ͕ؒ਺ඵ୯ҐͰ͋Ε͹ɼ
    ਓؒͷ൑அ଎౓ʹ͸Өڹ͠ͳ͍
    ࣮؀ڥͰͷར༻Λ૝ఆͨ͠ߟ࡯

    View full-size slide

  34. 5.
    ·ͱΊͱࠓޙͷల๬

    View full-size slide

  35. 35
    ɾγεςϜมߋ࣌ͷϦεΫΛ༧ଌ͢Δ͜ͱΛ໨తʹɼϓϩηεؒґଘ
    ؔ܎Λ௥੻͢ΔΞʔΩςΫνϟ Transtracer ΛఏҊͨ͠
    ɾιέοτ؂ࢹʹΑΓɼΦʔόʔϔουΛ௿ݮͤͭͭ͞ɼϓϩηεؒ
    ͷґଘؔ܎Λ໢ཏతʹݕग़Մೳ
    ɾ઀ଓΛ࠶ར༻͢Δ͜ͱͷଟ͍WebαʔϏεͷ෼ࢄγεςϜʹ͓͍
    ͯɼ࣮༻ՄೳͳϨϕϧͷΦʔόʔϔουͱݕग़ਫ਼౓ΛಘΒΕͨ
    ·ͱΊ

    View full-size slide

  36. 36
    ɾ͢΂ͯͷ઀ଓΛ௥੻͢ΔલఏͰɼ͍͔ʹΦʔόʔϔουΛ௿ݮͤ͞
    ͍͔ͯ͘Λݚڀ։ൃ͍ͯ͘͠
    ɾLinuxΧʔωϧ಺ͷॲཧΛ֦ுՄೳͳeBPFΛར༻͠ɼconnectͱ
    acceptΠϕϯτͷΈΛετϦʔϛϯάͰ௥੻͢Δ
    ɾUDP௨৴ͷ௥੻ͷରԠ
    ɾKubernetesͳͲͷීٴ͍ͯ͠ΔΞϓϦέʔγϣϯ࣮ߦج൫্Ͱಈ࡞
    ͤ͞Δ
    ɾΤοδίϯϐϡʔςΟϯάΛݟਾ͑ͯCMDBͷ෼ࢄ഑ஔ
    ࠓޙͷల๬

    View full-size slide

  37. ิ଍ࢿྉ

    View full-size slide

  38. 38
    ɾιέοτͷϙʔϦϯά
    ɾ਺ඵҎԼͷ୹໋ͳ઀ଓΛݟಀ͢Մೳੑ͕͋Δ
    ɾWebαʔϏεͷΑ͏ͳ઀ଓ࠶ར༻؀ڥͰ͋Ε͹໰୊ͳ͍
    ɾύέοτͷαϯϓϦϯά
    ɾ௕໋ͳ઀ଓͰ͋ͬͯ΋ɼݟಀ͢Մೳੑ͕͋Δ
    ɾಉҰϗετ্Ͱྲྀྔͷେ͖͍઀ଓͱྲྀྔͷখ͍͞઀ଓ͕ࠞ߹ͯ͠
    ͍ΔͱޙऀΛݟಀ͠΍͍͢
    ɾγεςϜ؅ཧऀ͕Ͳͷ઀ଓΛݟಀ͔͢Λ༧ଌͮ͠Β͍
    ِӄੑͷൺֱ: ϙʔϦϯάͱαϯϓϦϯά

    View full-size slide

  39. 39
    ɾϙʔϦϯά͕ݟಀ͢୹໋ͳ
    ઀ଓΛݕग़͢ΔͨΊʹετ
    ϦʔϛϯάΛ૊Έ߹ΘͤΔ
    ɾeBPFʹΑΓɼconnect(2)ͱ
    accept(2)ΠϕϯτΛऔಘ
    ͠ɼϑϩʔ৘ใΛऔಘ͢Δ
    ɾUDPͷ৔߹͸send_msg(2),
    recv_msg(2)Πϕϯτ
    ετϦʔϛϯάʹΑΔ઀ଓͷݕग़
    Linux Host
    Kernel
    Process Process
    TCP/UDP Flows

    .
    .
    .
    User
    Streaming
    Tracer

    View full-size slide

  40. 40
    ɾiovisor/bcc಺ͷtcpacceptͰɼඇӬଓԽ؀ڥͰͷෛՙ࣮ݧ
    ɾwrk (HTTP KeepAlive off)Ͱಉ࣌઀ଓ1000ͰnginxʹϕϯνϚʔΫ
    ɾCPUར༻཰͸45~50%/coreఔ౓
    ɾԠ౴஗ԆͷΦʔόϔου͸༗ҙͳѱԽ͸ݟΒΕͳ͔ͬͨ
    ɾeBPFͷΠϕϯτΛ͢΂ͯϢʔβʔϥϯυʹίϐʔ͍ͯ͠Δ͜ͱ͕
    CPUෛՙ͕ߴ͍ཁҼ͔΋͠Εͳ͍
    eBPFͷετϦʔϛϯάෛՙͷ༧උ࣮ݧ
    ৄࡉ͸࣮ݧϊʔτ΁ https://www.notion.so/yuuk1/iovisor-bcc-tcpconnect-tcpaccept-af2d1fdce35c49fb945b548db373213d

    View full-size slide

  41. 41
    ɾ͢΂ͯͷ઀ଓ৘ใΛऩू͢ΔͱɼCMDBʹ֨ೲ͢Δσʔλྔ͕େ͖͘
    ͳΔͨΊɼ৑௕ͳ৘ใΛ࡟ݮ͢Δ
    ɾΤϑΣϝϥϧϙʔτ: Χʔωϧ͔ΒׂΓ౰ͯΒΕΔϥϯμϜͳૹ৴ݩ
    ϙʔτ
    ɾಛఆͷLISTENϙʔτ΁ෳ਺ͷΤϑΣϝϥϧϙʔτ͔Β઀ଓ͞ΕΔ
    ɾ͜ΕΒͷ઀ଓΛू໿͠ɼ୯Ұͷ઀ଓͱΈͳ͢
    ΤϑΣϝϥϧϙʔτͷू໿
    Host
    Port
    Process Port
    Port
    Host
    Port Process
    ୯Ұͷ઀ଓ
    ͱͯ͠ू໿
    LISTEN
    ΤϑΣϝϥϧ

    View full-size slide