Transtracer: 分散システムにおけるTCP/UDP通信の終端点の監視によるプロセス間依存関係の自動追跡

情報処理学会 第12回インターネットと運用技術シンポジウム(IOTS)2019
論文: https://yuuk.io/papers/transtracer_iots2019.pdf
OSS: https://github.com/yuuki/transtracer

Transcript

1. ͘͞ΒΠϯλʔωοτ
   גࣜձࣾ
   (C) Copyright 1996-2019 SAKURA internet Inc
   ͘͞ΒΠϯλʔωοτ
   ݚڀॴ
   Transtracer: ෼ࢄγεςϜʹ͓͚ΔTCP/UDP௨৴ͷ
   ऴ୺఺ͷ؂ࢹʹΑΔϓϩηεؒґଘؔ܎ͷࣗಈ௥੻
   ৘ใॲཧֶձ ୈ12ճΠϯλʔωοτͱӡ༻ٕज़γϯϙδ΢Ϝ(IOTS)2019
   2019.12.6
   ௶಺༎थ*1, ݹ઒խେ*2, দຊ྄հ*1
   *1) ͘͞ΒΠϯλʔωοτ, *2) ͸ͯͳ
   

   View Slide

2. 2
   1. എܠͱ໨త
   2. ෼ࢄγεςϜ಺ͷґଘؔ܎ͷ௥੻ٕज़
   3. ఏҊख๏
   4. ࣮ݧͱߟ࡯
   5. ·ͱΊͱࠓޙͷల๬
   ໨࣍
   

   View Slide

3. 1.
   എܠͱ໨త
   

   View Slide

4. 4
   WebαʔϏε಺ͷґଘؔ܎ͷෳࡶԽͷഎܠ
   ௕ظؒͷαʔϏεఏڙத
   ͷػೳ௥Ճ
   Ϣʔβʔ͔ΒͷΞΫηε
   ૿Ճ
   ୯ҰͷαʔϏεࣄۀऀ͕
   ෳ਺ͷαʔϏεΛఏڙ
   (SNS,ECαΠτͳͲ)
   ༻్ಛԽͷϛυϧ΢ΣΞͷ௥Ճ
   ৽چγεςϜͷࠞ߹ঢ়ଶ
   εέʔϧΞ΢τʹΑΔϗετ਺
   ͷ૿Ճ
   ෳ਺ͷαʔϏεͷҰ෦Λڞ༗
   (Ϣʔβʔೝূج൫ͳͲ)
   ϚΠΫϩαʔϏεԽ
   

   View Slide

5. 5
   ɾෳࡶͳґଘؔ܎ʹΑΓɼγεςϜʹมߋΛՃ͑ͨ࣌ʹ໰୊͕͋ͬͨ
   ৔߹ɼ༧૝ΑΓେ͖ͳো֐ʹͭͳ͕Δ͜ͱ΋͋Δ
   ɾো֐ൃੜΛڪΕͯɼௐࠪʹ࣌ؒΛཁ͠ɼ݁Ռతʹมߋ଎౓͕௿Լ
   ɾґଘؔ܎͸γεςϜͷՔಇதʹมߋ͞Ε͍ͯͨ͘ΊɼखಈͰυΩϡ
   ϝϯτΛߋ৽͠ଓ͚Δख͕ؒେ͖͍
   ґଘؔ܎ͷෳࡶԽʹΑΓੜ·ΕΔ໰୊ҙࣝ
   มߋͷӨڹൣғΛ༧ଌ͢ΔͨΊʹ
   γεςϜ؅ཧऀʹͱͬͯະ஌ͷϓϩηεͷґଘؔ܎Λ
   ࣗಈͰ௥੻͢Δඞཁ͕͋Δ
   

   View Slide

6. 6
   ɾLinuxͷύέοτϑΟϧλϩάΛར༻ͯ͠L4઀ଓͷঢ়ଶΛऔಘ
   ɾLinuxΧʔωϧͷ௨৴ػߏΛར༻͢ΔݶΓɼOSϓϩηεʹରͯ͠ಁ
   աతʹ௥੻Մೳ
   ɾ஗ԆΦʔόϔου: ύέοτϑΟϧλॲཧͷΦʔόϔου
   ɾِӄੑ: ϦιʔεফඅΛ཈͑ΔͨΊʹɼύέοτΛແ࡞ҝʹαϯϓϦ
   ϯά͢ΔͨΊɼݕ஌࿙ΕͷՄೳੑ͕͋Δ
   طଘͷࣗಈ௥੻ٕज़ [1]
   [1] J. K. Clawson, Service Dependency Analysis via TCP/UDP Port Tracing, Master’s thesis, Brigham Young University-Provo 2015.
   

   View Slide

7. 7
   ݚڀ໨త: ະ஌ͷϓϩηεΛ௿ෛՙͰ௥੻
   Linux OS
   Kernel
   Process Process
   TCP/UDP
   Flows
   …
   .
   .
   .
   User
   ιέοτ
   Λ؂ࢹ
   TCP/UDP઀ଓͷऴ୺఺Ͱ͋ΔιέοτΛ؂ࢹ͠઀ଓΛࣗಈ௥੻
   1. ஗ԆΦʔόϔουͷղܾ
   ɾιέοτͷ؂ࢹΛϓϩηε
   ͷ௨৴ͱಠཱͤ͞Δ
   2. ِӄੑͷղܾ
   ɾ͢΂ͯͷιέοτΛ؂ࢹ
   ɾύέοτ͸ඵؒ਺ສ୯Ґ͕
   ͋Γ͑Δ
   ɾιέοτ͸࠶ར༻͞Ε͏Δ
   ͨΊɼιέοτ਺ͷ΄͏͕
   খ͘͞ͳΓ΍͍͢
   

   View Slide

8. 8
   γεςϜ؅ཧऀʹͱͬͯ఍߅ͳ͘ಋೖՄೳ
   ɾΦʔόϔου͕େ͖͍ͱɼ؅ཧऀ͸Өڹͷେ͖͍ίϯϙʔωϯτΛ
   ௥੻ର৅͔Β֎͢͜ͱ͕͋Δ
   ↪ ؅ཧऀͷهԱ΍υΩϡϝϯςʔγϣϯʹґଘ͢Δ
   ɾِӄੑ͕େ͖͍ͱɼ؅ཧऀ͸ಘΒΕͨґଘؔ܎Λ৴༻͠ͳ͘ͳΔ
   ↪ ؅ཧऀ͕खಈͰґଘؔ܎Λௐࠪ͢Δ
   ௿͍Φʔόʔϔουͱ௿ِ͍ӄੑʹΑΓ
   γεςϜ؅ཧऀ͕ར༻͢Δͱ͖ͷ৺ཧత఍߅Λ௿ݮՄೳ
   

   View Slide

9. 2.
   ෼ࢄγεςϜ಺ͷ
   ґଘؔ܎ͷ௥੻ٕज़
   

   View Slide

10. 10
    ɾWebαʔϏεͷ෼ࢄγεςϜ಺ͷґଘؔ܎Λ௥੻Մೳ
    ɾWebαʔϏεͰ͸ɼΞΫηε਺͕େ͖͘ɼԠ౴଎౓͕1ඵҎԼͰ͋
    Δ͜ͱ͕ಛ௃
    ɾ઀ଓΛ࠶ར༻͠ɼ઀ଓ͝ͱͷෛՙΛ௿ݮ͍ͤͯ͞Δ
    ɾσʔληϯλʔ಺௨৴ͰHTTP/2ͷར༻͕૿େ
    ɾίωΫγϣϯϓʔϦϯάʹΑΓσʔλϕʔε΁ͷ઀ଓΛ࠶ར༻
    ɾτϥϯεϙʔτ૚Ͱ͸ɼ௕໋ͳ઀ଓͱͳΔ܏޲͕͋Δ
    ຊݚڀͷ௥੻ର৅ͷલఏ
    

    View Slide

11. 11
    ෼ࢄγεςϜ಺ͷґଘؔ܎ͷ௥੻ٕज़ͷ෼ྨ
    ϦΫΤετϕʔε
    Ξϓϩʔν
    ΞϓϦέʔγϣϯ૚ͷϦΫΤετ͕
    Ͳͷ௨৴ܦ࿏ΛͨͲΔ͔Λ௥੻͢Δख๏
    ίωΫγϣϯϕʔε
    Ξϓϩʔν
    ऴ୺ϗετ্Ͱτϥϯεϙʔτ઀ଓͷঢ়ଶΛ
    औಘ͢Δ͜ͱʹΑΓґଘؔ܎Λ௥੻͢Δख๏
    ύέοτϕʔε
    Ξϓϩʔν
    ϗετ্΍εΠον্ͰύέοτϔομΛ؍ଌ
    ͢Δ͜ͱʹΑΓɼ ґଘؔ܎Λ௥੻͢Δख๏
    

    View Slide

12. 12
    ɾΞϓϦέʔγϣϯ૚ͷ֤ϦΫΤετʹࣝผࢠΛׂΓৼΓɼޙଓͷϦ
    ΫΤετʹຒΊࠐΜ্ͩͰɼޙଓͷϓϩηε΁఻ൖͤ͞Δ
    ɾࣝผࢠΛཔΓʹɼϦΫΤετ͕γεςϜ಺ͷͲͷϓϩηεΛܦ༝͠
    ͯॲཧ͞Ε͔ͨΛ௥੻
    ϦΫΤετϕʔεΞϓϩʔν
    M. Y. Chen, et al., Pinpoint: Problem Determination in Large, Dynamic Internet Services, IEEE/IFIP International Conference on DSN, pp. 595–604 2002.
    P. Barham, et al., Magpie: Online Modelling and Performance-aware Systems, HotOS, pp. 85–90 2003.
    R. Fonseca, et al., X-Trace: A Pervasive Network Tracing Framework, USENIX Conference on NSDI, pp. 20–20 2007.
    B. H. Sigelman, et al., Dapper, a Large-Scale Distributed Systems Tracing Infrastructure, Technical report, Google 2010.
    

    View Slide

13. 13
    ɾLinuxͷύέοτϑΟϧλ(iptables)Λར༻ͯ͠ɼτϥϯεϙʔτ૚
    ઀ଓΛݕग़
    ɾiptablesͰ઀ଓ։࢝ϩάΛऩू͠ɼϩάίϨΫλͰதԝͷσʔλ
    ϕʔεʹอଘ
    ɾLinuxΧʔωϧͷL4௨৴ػߏΛར༻͢ΔݶΓɼґଘΛ௥੻Մೳ
    ίωΫγϣϯϕʔεΞϓϩʔν [Clawson 15]
    [Clawson 15] J. K. Clawson, Service Dependency Analysis via TCP/UDP Port Tracing, Master’s thesis, Brigham Young University-Provo 2015.
    

    View Slide

14. 14
    ɾωοτϫʔΫ૚ͷύέοτΛ΋ͱʹґଘΛ௥੻͢Δ
    ɾطଘͷτϥώοΫ͔ΒύέοτΛऩू͠ɼύέοτϔομ্ͷIPΞ
    υϨεͱϙʔτɼύέοτͷૹड৴ͷ࣌ࠁͳͲͷ৘ใΛղੳ͢Δ
    ɾ֤ܦ࿏ͰτϥώοΫྲྀྔʹ͕࣌ؒࠩ͋Δ͜ͱʹண໨͠ɼ֤ܦ࿏Ͱτ
    ϥώοΫੑ࣭ʹ૬͕ؔ͋ΔύλʔϯΛൃݟ͠ɼґଘΛਪఆ͢Δ
    ύέοτϕʔεΞϓϩʔν
    P. Bahl, et.al.: Towards Highly Reliable Enterprise Network Services via Inference of Multi-Level Dependencies, ACM SIGCOMM Review, Vol. 37, No. 4,
    pp.13–24 2007.
    X. Chen, et.al.: Automating Network Application Dependency Discovery: Experiences, Limitations, and New Solutions, USENIX Symposium on OSDI,
    pp.117–130 2008.
    P. Lucian, etl.al.: Macroscope: End-Point Approach to Networked Application Dependency Discovery, CoNEXT, pp.229–240 2009.
    A. Natarajan, et.al.: NSDMiner: Automated Discovery of Network Service Dependencies, IEEE INFOCOM, pp. 2507–2515 2012.
    A. Zand, et.al.: Rippler: Delay Injection for Service Dependency Detection, IEEE INFOCOM, pp. 2157–2165 2014.
    

    View Slide

15. 15
    ɾϦΫΤετϕʔεɾίωΫγϣϯϕʔε
    ɾ஗ԆΦʔόϔου: ௨৴ܦ࿏தʹ௥ՃͷॲཧΛڬΈࠐΉΦʔόϔου
    ɾίωΫγϣϯϕʔεɾύέοτϕʔε
    ɾِӄੑ: ύέοτͷαϯϓϦϯάʹΑΓྲྀྔͷখ͍͞઀ଓΛݕ஌Ͱ
    ͖ͳ͍
    ɾ௥੻୯Ґ: OSϓϩηεͰ͸ͳ͘IPΞυϨε୯ҐͱͳΔ
    ɾϦΫΤετϕʔε
    ɾܭଌ४උίετ: طଘͷΞϓϦέʔγϣϯ಺ʹ௥੻ॲཧΛ௥Ճ͢Δ
    खؒ
    ֤Ξϓϩʔνͷ՝୊
    

    View Slide

16. 3.
    ఏҊख๏
    

    View Slide

17. 17
    ఏҊख๏: ιέοτϕʔεΞϓϩʔν
    Linux OS
    Kernel
    Process Process
    TCP/UDP
    Flows
    …
    .
    .
    .
    User
    ιέοτ
    Λ؂ࢹ
    ɾιέοτͷ؂ࢹ͸ϓϩηεͷ
    ௨৴ͱ͸ಠཱ͍ͯ͠ΔͨΊɼ
    ஗ԆΦʔόϔουΛ௿ݮՄೳ
    ɾ઀ଓ࠶ར༻࣌ʹ͸͢΂ͯͷι
    έοτΛ௿ෛՙͰ؂ࢹՄೳ
    ɾιέοτʹΑΓɼϓϩηεͱ
    ઀ଓͷඥ෇͚͕Մೳ
    ɾϓϩηε͕ιέοτΛར༻͢
    ΔݶΓಁաతʹ௥੻Մೳ
    

    View Slide

18. 18
    TranstracerͷγεςϜߏ੒
    Host 1
    Host 2
    Host N
    CMDB
    Tracer
    Tracer
    Tracer
    Systems
    Administrator
    ɾϗετ্ʹTracerϓϩηεΛ഑ஔ
    ɾ֤Tracerϓϩηε͸औಘͨ͠઀ଓ৘
    ใΛCMDB(઀ଓ৘ใ؅ཧDB)ʹอଘ
    ɾγεςϜ؅ཧऀ͸CMDBʹΞΫηε
    ͠ɼෳ਺ͷϗετʹ·͕ͨΓґଘؔ
    ܎Λऔಘ
    

    View Slide

19. 19
    ઀ଓ৘ใͷऔಘ
    Host
    Kernel
    Process Process
    TCP/UDP
    Flows
    …
    Tracer
    Polling
    ɾTracerϓϩηε͕LinuxΧʔωϧʹ໰͍߹Θ
    ͤɼTCP/UDPιέοτ৘ใΛϙʔϦϯάऔಘ
    ɾ઀ଓΛऴ୺͢ΔOSϓϩηε৘ใ΋͋Θͤͯ
    औಘ
    ɾιέοτ৘ใ: /proc/net/tcp΍Netlink sock_diag
    ɾϓϩηε৘ใ: /proc//{stat,fd}
    .
    .
    .
    ॲཧʹհೖ͠ͳ͍ͨΊ
    ௿Φʔόʔϔου
    

    View Slide

20. 20
    Transtracerͷར༻ྫ
    $ ttctl --dbhost 10.0.0.20 --ipv4 10.0.0.10
    10.0.0.10:80 (’nginx’, pgid=4656)
    ᵋ<-- 10.0.0.11:many (’wrk’, pgid=5982)
    10.0.0.10:80 (’nginx’, pgid=4656)
    ᵋ--> 10.0.0.12:8080 (’python’, pgid=6111)
    10.0.0.10:many (’fluentd’, pgid=2127)
    ᵋ--> 10.0.0.13:24224 (’fluentd’, pgid=2001)
    10.0.0.10
    nginx
    10.0.0.11
    wrk
    10.0.0.12
    python
    10.0.0.13
    fluentd
    :80
    fluentd
    :8080
    :24224
    

    View Slide

21. 21
    ϓϩηεͷσʔλߏ଄
    ᶃ LinuxͷϓϩηεάϧʔϓΛ
    ϊʔυͷ࠷খ୯Ґͱ͢Δ
    ᵓᴷnginx,627,627
    ᴹ ᵓᴷnginx,628,627
    ᴹ ᵋᴷnginx,629,627
    $ pstree -apg | grep nginx
    ᶄ (ipv4, pgid, pname)Ͱϓϩηε
    ʹҰҙ੍໿Λ͔͚Δ
    ϓϩηε͸࠶ىಈ͢ΔͱID͕มԽ
    ͯ͠͠·͏ͨΊɼ໰͍߹Θͤ࣌ʹ
    ͸pgid͕ҟͳΔ΋ͷΛॏෳഉআ
    

    View Slide

22. 22
    ઀ଓ؅ཧͷͨΊͷσʔλߏ଄
    ᶅ ActiveͱPassiveʹϊʔυΛ෼ྨ
    ᶆ Active => PassiveͷϑϩʔΛอଘ
    Active Passive
    Process
    Passive
    Active
    Port N
    Port M
    Active
    ಉҰϓϩηε͕Activeʹ΋
    Passiveʹ΋ͳΓ͑Δ
    ಉҰϓϩηε͕ෳ਺ͷϙʔτ
    ΛϦοεϯ͢Δ͜ͱ͕͋Δ
    

    View Slide

23. 23
    ɾIPΞυϨεͱϙʔτ൪߸͕௨৴ܦ࿏தʹมߋ͞ΕΔͱϓϩηεؒͷඥ
    ෇͚͕Ͱ͖ͳ͘ͳΔ
    ɾϩʔυόϥϯαʹΑΔDNATͳͲ
    ɾNATͷηογϣϯΛಡΈऔΓɼมߋલޙͷΞυϨεͱϙʔτͷඥ෇
    ͚Λߦ͏͜ͱ͸Մೳ
    ɾLinuxͷINETιέοτΛར༻͠ͳ͍ΞϓϦέʔγϣϯͷ઀ଓΛ௥੻Ͱ
    ͖ͳ͍
    ɾDPDK΍netmapͳͲͷΧʔωϧΛόΠύεͯ͠௨৴͢Δٕज़
    ఏҊख๏ͷ੍໿
    

    View Slide

24. 4.
    ࣮ݧͱߟ࡯
    

    View Slide

25. 25
    1. ΞϓϦέʔγϣϯʹ༩͑ΔԠ౴஗ԆΦʔόϔουͷධՁ
    2. ઀ଓ৘ใऔಘͷͨΊͷCPUར༻཰ΦʔόϔουͷධՁ
    ධՁ࣮ݧ
    ఏҊख๏ͱಛ௃͕ࣅ͍ͯΔίωΫγϣϯϕʔεΞϓϩʔνͷઌߦख๏[1]
    ͱ֤ΦʔόϔουΛൺֱ
    [1] J. K. Clawson, Service Dependency Analysis via TCP/UDP Port Tracing, Master’s thesis, Brigham Young University-Provo 2015.
    

    View Slide

26. 26
    ࣮ݧ؀ڥͷߏ੒
    CMDB
    Client Server
    HTTP benchmarker HTTP server
    Tracer
    

    View Slide

27. 27
    ࣮ݧ؀ڥͷৄࡉ
    ߲໨ ࢓༷
    Client CPU
    Memory
    Benchmarker
    Intel Xeon CPU E5-2650 v3 2.30GHz 2core
    1 GB
    wrk 4.1.0-4
    Server CPU
    Memory
    HTTP Server
    Intel Xeon CPU E5-2650 v3 2.30GHz 4core
    1GB
    nginx 1.17.3
    CMDB CPU
    Memory
    Database
    Intel Xeon CPU E5-2650 v3 2.30GHz 1core
    1 GB
    PostgreSQL 11.3
    ɾΠϯελϯε͸͢΂ͯ͘͞ΒͷΫϥ΢υ্ʹߏங
    ɾLinux Kernel 4.15 (Ubuntu Server 18.04.3 LTS)
    

    View Slide

28. 28
    1. Normal: ௥੻ॲཧ͕ͳ͍ঢ়ଶ
    2. Transtracer: ఏҊख๏ (https://github.com/yuuki/transtracer v0.1.0)
    ɾϙʔϦϯάִؒ͸1ඵ
    3. iptables NEWϑΟϧλํࣜ: ৽ن઀ଓͷΈϩάΛग़ྗ
    4. iptables ESTBϑΟϧλํࣜ: ઀ଓཱ֬தʹ΍ΓͱΓ͞ΕΔύέοτ
    ͷϩάΛαϯϓϦϯάͤͣʹग़ྗ
    ɾઌߦख๏ͷແ࡞ҝͳαϯϓϦϯάͰ͸ɼ௕໋ͳ઀ଓͰ͋ͬͯ΋ྲྀ
    ྔ͕খ͍͞઀ଓΛݟಀ͢Մೳੑ͕͋Δ
    ࣮ݧʹ༻͍Δ֤࣮૷
    

    View Slide

29. 29
    Ԡ౴஗ԆΦʔόʔϔου
    50
    100
    150
    200
    250
    300
    350
    400
    450
    500
    5000 10000 15000 20000
    Average Latency (ms)
    Connections
    Normal
    93.1
    191.6
    279.3
    353.8
    Transtracer
    94.7
    188.3
    291.8
    401.2
    ESTB filter
    115.0
    236.0
    359.0
    462.5
    NEW filter
    113.1
    214.4
    310.0
    449.3
    ɾNormalʹରͯ͠transtracer
    ͕1.7~13.4%ͷΦʔόϔο
    υ૿
    ɾiptables࣮૷ͷESTBϑΟϧ
    λํࣜʹରͯ͠ɼ
    transtracer͕13-20%ͷ
    Φʔόϔουݮগ
    

    View Slide

30. 30
    CPUར༻཰Φʔόʔϔου
    0
    10
    20
    30
    40
    50
    60
    70
    80
    90
    100
    5000 10000 15000 20000
    0
    50
    100
    150
    200
    250
    300
    350
    400
    450
    500
    CPU usage (%)
    Reading sockets time(ms)
    Connections
    ttracerd’s CPU usage
    13.2
    23.0
    34.2
    44.4
    ESTB filter’s CPU usage
    72.2
    75.9
    78.8 78.6
    Reading sockets time
    102.3
    199.1
    317.8
    408.6
    ɾ20,000઀ଓʹ͓͍ͯɼ
    TranstracerͷCPUར༻཰
    44.4%ɼESTBϑΟϧλํ
    ࣜͷCPUར༻཰͸78.6%
    ɾ43.5%ͷCPUར༻཰ͷ௿
    ݮ
    

    View Slide

31. 31
    ɾ2ສ઀ଓ࣌ʹϙʔϦϯάؒ
    ִΛ૿Ճͤ͞ΔͱCPUར༻
    ཰͕Ͳͷఔ౓௿ݮ͞ΕΔ͔
    ɾ5ඵҎ಺ͷ୹໋ͳ઀ଓΛݕ
    ग़Ͱ͖ͳ͘ͳΔՄೳੑ͕͋
    Δ͔ΘΓʹɼ8.6%·Ͱ௿ݮ
    Մೳ
    ϙʔϦϯάִؒͱCPUར༻཰ͷؔ܎
    0
    5
    10
    15
    20
    25
    30
    35
    40
    45
    50
    55
    1 2 3 4 5
    CPU usage (%)
    Polling interval
    CPU usage
    44.4
    21.6
    13.0
    10.8
    8.6
    

    View Slide

32. 32
    ɾԠ౴஗ԆΦʔόϔου
    ɾઌߦख๏ͷ2ͭͷϑΟϧλํࣜͱൺֱ͠ɼఏҊख๏͕༗ར
    ɾCPUར༻཰Φʔόϔου
    ɾNEWϑΟϧλํ͕ࣜɼॳճͷ઀ଓཱ֬࣌ͷΈCPUΛར༻͢Δͨ
    Ίɼ઀ଓͷ࠶ར༻؀ڥͰ͸༗ར
    ɾͨͩ͠ɼఏҊख๏͸ϙʔϦϯάִؒͷௐ੔ʹΑΓCPUར༻཰ͷ௿
    ݮ͸Մೳ
    ઌߦख๏ͱͷൺֱߟ࡯
    

    View Slide

33. 33
    ɾ઀ଓΛ࠶ར༻͢Δ؀ڥͰ͸ɼ୹໋ͳ઀ଓΛݕ஌Ͱ͖ͳ͘ͱ΋࣮༻
    ্໰୊ʹͳΒͳ͍
    ɾϙʔϦϯάִؒΛCPUར༻཰͕े෼খ͘͞ͳΔ·ͰௐઅՄೳ
    ɾϦΞϧλΠϜੑͷ؍఺ͰɼϙʔϦϯάִ͕ؒ਺ඵ୯ҐͰ͋Ε͹ɼ
    ਓؒͷ൑அ଎౓ʹ͸Өڹ͠ͳ͍
    ࣮؀ڥͰͷར༻Λ૝ఆͨ͠ߟ࡯
    

    View Slide

34. 5.
    ·ͱΊͱࠓޙͷల๬
    

    View Slide

35. 35
    ɾγεςϜมߋ࣌ͷϦεΫΛ༧ଌ͢Δ͜ͱΛ໨తʹɼϓϩηεؒґଘ
    ؔ܎Λ௥੻͢ΔΞʔΩςΫνϟ Transtracer ΛఏҊͨ͠
    ɾιέοτ؂ࢹʹΑΓɼΦʔόʔϔουΛ௿ݮͤͭͭ͞ɼϓϩηεؒ
    ͷґଘؔ܎Λ໢ཏతʹݕग़Մೳ
    ɾ઀ଓΛ࠶ར༻͢Δ͜ͱͷଟ͍WebαʔϏεͷ෼ࢄγεςϜʹ͓͍
    ͯɼ࣮༻ՄೳͳϨϕϧͷΦʔόʔϔουͱݕग़ਫ਼౓ΛಘΒΕͨ
    ·ͱΊ
    

    View Slide

36. 36
    ɾ͢΂ͯͷ઀ଓΛ௥੻͢ΔલఏͰɼ͍͔ʹΦʔόʔϔουΛ௿ݮͤ͞
    ͍͔ͯ͘Λݚڀ։ൃ͍ͯ͘͠
    ɾLinuxΧʔωϧ಺ͷॲཧΛ֦ுՄೳͳeBPFΛར༻͠ɼconnectͱ
    acceptΠϕϯτͷΈΛετϦʔϛϯάͰ௥੻͢Δ
    ɾUDP௨৴ͷ௥੻ͷରԠ
    ɾKubernetesͳͲͷීٴ͍ͯ͠ΔΞϓϦέʔγϣϯ࣮ߦج൫্Ͱಈ࡞
    ͤ͞Δ
    ɾΤοδίϯϐϡʔςΟϯάΛݟਾ͑ͯCMDBͷ෼ࢄ഑ஔ
    ࠓޙͷల๬
    

    View Slide

37. ิ଍ࢿྉ
    

    View Slide

38. 38
    ɾιέοτͷϙʔϦϯά
    ɾ਺ඵҎԼͷ୹໋ͳ઀ଓΛݟಀ͢Մೳੑ͕͋Δ
    ɾWebαʔϏεͷΑ͏ͳ઀ଓ࠶ར༻؀ڥͰ͋Ε͹໰୊ͳ͍
    ɾύέοτͷαϯϓϦϯά
    ɾ௕໋ͳ઀ଓͰ͋ͬͯ΋ɼݟಀ͢Մೳੑ͕͋Δ
    ɾಉҰϗετ্Ͱྲྀྔͷେ͖͍઀ଓͱྲྀྔͷখ͍͞઀ଓ͕ࠞ߹ͯ͠
    ͍ΔͱޙऀΛݟಀ͠΍͍͢
    ɾγεςϜ؅ཧऀ͕Ͳͷ઀ଓΛݟಀ͔͢Λ༧ଌͮ͠Β͍
    ِӄੑͷൺֱ: ϙʔϦϯάͱαϯϓϦϯά
    

    View Slide

39. 39
    ɾϙʔϦϯά͕ݟಀ͢୹໋ͳ
    ઀ଓΛݕग़͢ΔͨΊʹετ
    ϦʔϛϯάΛ૊Έ߹ΘͤΔ
    ɾeBPFʹΑΓɼconnect(2)ͱ
    accept(2)ΠϕϯτΛऔಘ
    ͠ɼϑϩʔ৘ใΛऔಘ͢Δ
    ɾUDPͷ৔߹͸send_msg(2),
    recv_msg(2)Πϕϯτ
    ετϦʔϛϯάʹΑΔ઀ଓͷݕग़
    Linux Host
    Kernel
    Process Process
    TCP/UDP Flows
    …
    .
    .
    .
    User
    Streaming
    Tracer
    

    View Slide

40. 40
    ɾiovisor/bcc಺ͷtcpacceptͰɼඇӬଓԽ؀ڥͰͷෛՙ࣮ݧ
    ɾwrk (HTTP KeepAlive off)Ͱಉ࣌઀ଓ1000ͰnginxʹϕϯνϚʔΫ
    ɾCPUར༻཰͸45~50%/coreఔ౓
    ɾԠ౴஗ԆͷΦʔόϔου͸༗ҙͳѱԽ͸ݟΒΕͳ͔ͬͨ
    ɾeBPFͷΠϕϯτΛ͢΂ͯϢʔβʔϥϯυʹίϐʔ͍ͯ͠Δ͜ͱ͕
    CPUෛՙ͕ߴ͍ཁҼ͔΋͠Εͳ͍
    eBPFͷετϦʔϛϯάෛՙͷ༧උ࣮ݧ
    ৄࡉ͸࣮ݧϊʔτ΁ https://www.notion.so/yuuk1/iovisor-bcc-tcpconnect-tcpaccept-af2d1fdce35c49fb945b548db373213d
    

    View Slide

41. 41
    ɾ͢΂ͯͷ઀ଓ৘ใΛऩू͢ΔͱɼCMDBʹ֨ೲ͢Δσʔλྔ͕େ͖͘
    ͳΔͨΊɼ৑௕ͳ৘ใΛ࡟ݮ͢Δ
    ɾΤϑΣϝϥϧϙʔτ: Χʔωϧ͔ΒׂΓ౰ͯΒΕΔϥϯμϜͳૹ৴ݩ
    ϙʔτ
    ɾಛఆͷLISTENϙʔτ΁ෳ਺ͷΤϑΣϝϥϧϙʔτ͔Β઀ଓ͞ΕΔ
    ɾ͜ΕΒͷ઀ଓΛू໿͠ɼ୯Ұͷ઀ଓͱΈͳ͢
    ΤϑΣϝϥϧϙʔτͷू໿
    Host
    Port
    Process Port
    Port
    Host
    Port Process
    ୯Ұͷ઀ଓ
    ͱͯ͠ू໿
    LISTEN
    ΤϑΣϝϥϧ
    

    View Slide