分散システム内の関係性に着目したObservabilityツール / Observability tool focused on relationship in distributed systems

ゆううきが開発しているlstfやtranstracerなどのツールを最近のObservabilityの流れから紹介した話です。

Kyoto.なんか #5, https://kyoto-nanka.connpass.com/event/141982/, 2019年8月24日.

Transcript

1. ෼ࢄγεςϜ಺ͷؔ܎ੑʹண໨ͨ͠ Observabilityπʔϧ id:y_uuki / @yuuk1t Kyoto.ͳΜ͔ #5, 2019.08.24
2. ࣗݾ঺հ 2 https://yuuk.io/ @yuuk1t id:y_uuki ݩ͸ͯͳΠϯλʔϯ (2011೥) ݩ͸ͯͳΤϯδχΞ ͘͞ΒΠϯλʔωοτݚڀॴ
4. 4 Line OpenChat Ώ͏͏͖ϥϘ
5. 5 ෼ࢄγεςϜ΍ͬͯ·͔͢
6. 6 WebαʔϏεͷ෼ࢄγεςϜ
7. External DNS Server Application flow DNS flow RDB server Application server Web server Internal DNS server Full text search server KVS server Message queue server Batch server Application server ͜͜10೥͘Β͍ͷ෼ࢄγεςϜ
8. Log collector agent Main network process Monitoring agent Proxy User Authentication ୯Ұϗετ্ͷ༷ࢠ DNS forwarder
9. ෳ਺ͷγεςϜ͕݁߹͢ΔΞʔΩςΫνϟ ݁߹ͷํࣜͷҰ͕ͭϚΠΫϩαʔϏε Message Queue Reverse Proxy
10. ෼ࢄγεςϜͷґଘؔ܎͕ະ஌ 10 • ෼ࢄγεςϜͰ͸ɼ୭͕୭ʹґଘ͍ͯ͠Δ͔ɼਓؒͷهԱʹཔΓ͕ͪ • ෳࡶԽ͗ͯ͢͠هԱͨ͠ΓɼυΩϡϝϯτ͖͠Εͳ͘ͳ͍ͬͯΔ
11. ґଘؔ܎͕ະ஌Ͱ͋Δ͜ͱͷ໰୊ҙࣝ 11 • ͋ΔίϯϙʔωϯτΛมߋ͢Δͱ͖ʹɼมߋͷӨڹൣғ͕෼͔Βͳ͍ • ௐࠪʹ࣌ؒΛ͔͚Δ͔ɼͦ΋ͦ΋ఘΊΔ • ো֐ൃੜ࣌ʹɼ૬ؔؔ܎΍ҼՌؔ܎͕Θ͔Βͳ͍ • ֤छϝτϦοΫ΍ϩάΛ͕Μ͹͖ͬͯͭ͋ΘͤΔ
12. 12 ෼ࢄγεςϜ಺ͷཁૉಉ࢜ͷ ؔ܎ੑʹண໨ͨ͠ Observability
13. Observability 13 • Մ؍ଌੑͱ͔ɼ؍ଌՄೳੑͱ͔ • γεςϜ͕࣮؀ڥͰͲͷΑ͏ʹಈ࡞͍ͯ͠Δ͔Λ֎෦͔Β஌Δ͜ͱ͕ Ͱ͖Δೳྗ • ݱࡏͰ͸ɼLog, Trace, MetricsΛσʔλιʔεͱͯ͠ར༻ͯ͠ ObservabilityΛୡ੒͍ͯ͠Δ -PHHJOH 5SBDJOH .FUSJDT
14. ؔ܎ੑʹண໨ͨ͠Observability 14 • ෼ࢄτϨʔγϯάʹΑΓɼϚΠΫϩαʔϏεؒͷؔ܎ੑ΍ϦΫΤετ ͷॱংɼ֤ܦ࿏ͰͷԠ౴଎౓ͳͲΛՄࢹԽͰ͖Δ • ΞϓϦέʔγϣϯʹ௨৴ϩάΛు͔ͤͯऩू͢Δ • αʔϏεϝογϡͰαΠυΧʔϓϩΩγ͕ϩάΛ೺ѲͰ͖Δ
15. HTTPϕʔεͷϚΠΫϩαʔϏεؒҎ֎ͷؔ܎ੑ͸ʁ 15 • HTTPҎ֎ͷ༷ʑͳϓϩτίϧʹϓϩΩγͰରԠ͢Δͷ΋େม • ࣗ෼Ͱ։ൃ͍ͯ͠ΔΘ͚Ͱ͸ͳ͍ϛυϧ΢ΣΞͷίʔυʹϩΪϯάͷ ͨΊͷίʔυΛຒΊࠐΉͷ͸େม
16. 16 ൚༻తͰܰྔͳτϨʔγϯάΛ ߟ͑Δ
17. Lightweight Traceability 17 LinuxΧʔωϧͷTCP/UDP૚Ͱ൚༻తʹτϨʔγϯά • TCP/UDP઀ଓཱ֬ͨ͠ͱ͖ͷΠϕϯτ͚ͩ௥੻͢Ε͹Α͍ • Πϕϯτʹ͸ѼઌIPΞυϨεͱϙʔτɼૹ৴ઌIPΞυϨεͱϙʔ τͷ૊͕͋Δ
18. TCP/UDPͷ઀ଓΠϕϯτͷऩू 18 Host Kernel Process Process Transport … Tracer Polling ɾTracerϓϩηε͕LinuxΧʔωϧʹ໰͍߹Θ ͤɼTCP/UDPιέοτ৘ใΛϙʔϦϯάऔಘ ɾ઀ଓΛऴ୺͢ΔOSϓϩηε৘ใ΋͋Θͤͯ औಘ ɾιέοτ৘ใ: /proc/net/tcp΍Netlink sock_diag ɾϓϩηε৘ใ: /proc/<pid>/{stat,fd} . . . ॲཧʹհೖ͠ͳ͍ͨΊ ௿Φʔόʔϔου
19. 19 ܧଓతʹτϨʔγϯά͢Δલʹ ·ͣ͸ॠؒతʹ৘ใΛදࣔ͢Δ CLIπʔϧΛͭͬͨ͘
20. github.com/yuuki/lstf 20
21. lstf 21 $ lstf -n Local Address:Port <--> Peer Address:Port Connections 10.0.1.9:many --> 10.0.1.10:3306 22 10.0.1.9:many --> 10.0.1.11:3306 14 10.0.2.10:22 <-- 192.168.10.10:many 1 10.0.1.9:80 <-- 10.0.2.13:many 120 10.0.1.9:80 <-- 10.0.2.14:many 202
22. Ͳͷϓϩηεͱ௨৴͍ͯ͠Δ͔΋Θ͔Δ 22 $ lstf -n —process Local Address:Port <--> Peer Address:Port Connections Process 10.0.1.9:many --> 10.0.1.10:3306 22 {“mysqld”,pgid=6342} 10.0.1.9:many --> 10.0.1.11:3306 14 {“mysqld”,pgid=9398} 10.0.2.10:22 <-- 192.168.10.10:many 1 {“sshd”, pgid=27027} 10.0.1.9:80 <-- 10.0.2.13:many 120 {“unicorn”, pgid=3790} 10.0.1.9:80 <-- 10.0.2.14:many 202 {“unicorn”, pgid=3790}
23. 23 Demo ISUCON4༧બ benchmarker web app db
24. 24 ࣮૷ํ๏
25. ࣮૷ͷϙΠϯτ 25 1. TCP઀ଓΠϕϯτΛͲ͏΍ͬͯऔಘ͢Δͷ͔ʁ 2. TCP઀ଓͷํ޲ΛͲ͏΍ͬͯೝࣝ͢Δͷ͔ʁ 3. TCP઀ଓΠϕϯτͷू໿ͱ͸ͳʹ͔ʁ 4. TCP઀ଓΠϕϯτͱϓϩηεΛͲ͏΍ͬͯඥ෇͚Δͷ͔ʁ
26. 1. TCP઀ଓΠϕϯτΛͲ͏΍ͬͯऔಘ͢Δͷ͔ʁ 26 • /proc/net/tcp͔Βऔಘ • procfsͱ͍͏ϑΝΠϧγεςϜܗࣜͰΧʔωϧ͔Β৘ใΛͱΔ • https://github.com/shirou/gopsutil Λར༻ • Netlink API͔Βऔಘ • ιέοτܗࣜͰΧʔωϧ͔Β৘ใΛͱΔ ߴ଎ • Socket Monitoring Interface • github.com/elastic/gosigar/sys/linuxΛར༻
27. Procfs vs Netlink 27 • ໿40,000઀ଓ͋ΔWebαʔό্ʹͯɼlstfίϚϯυͷ࣮ߦ࣌ؒΛ໊લ ղܾ࣌ؒΛؚ·ͣʹൺֱ • EC2ͷc4.2xlargeɺDebian 8.10ɺLinuxΧʔωϧ3.16 • 500ms(procfs) => 300ms(netlink) ΁ • Netlink࣮૷ͷ΄͏͕1.6ഒ͸΍͍ https://memo.yuuk.io/entry/2018/06/18/003157
28. 2. TCP઀ଓͷํ޲ΛͲ͏΍ͬͯࣝผ͢Δͷ͔ʁ 28 Host Y Port N Process B CONNECT Host X Port M Process A LISTEN ɾ઀ଓΛཁٻ͢ΔϗετY͸ɼ઀ଓΛड͚෇͚ΔϗετXʹґଘ͢Δ ɾϗετY͔ΒΈͯѼઌϙʔτ͕LISTENϙʔτMͰ͋Ε͹ɼHost Y ͔Β઀ଓΛཁٻ͍ͯ͠Δ͜ͱ͕Θ͔Δ ɾLISTENϙʔτ͸ɼϗετXͷOSʹ໰͍߹Θͤͯऔಘ͢Δ
29. 3. TCP઀ଓΠϕϯτͷू໿ͱ͸ͳʹ͔ʁ 29 ɾΤϑΣϝϥϧϙʔτͷ৘ใ͸ґଘ೺Ѳʹ͸༨෼ͳͷͰू໿ ɾΤϑΣϝϥϧϙʔτ: Χʔωϧ͔ΒׂΓ౰ͯΒΕΔϥϯμϜͳૹ৴ݩ ϙʔτ ɾಛఆͷLISTENϙʔτ΁ෳ਺ͷΤϑΣϝϥϧϙʔτ͔Β઀ଓ͞ΕΔ ɾ͜ΕΒͷ઀ଓΛू໿͠ɼ1ݸͷ઀ଓͱΈͳ͢ Host Port Process Port Port Host Port Process 1ݸͷ઀ଓ
 ͱͯ͠ू໿ LISTEN ΤϑΣϝϥϧ
30. 4. TCP઀ଓΠϕϯτͱϓϩηεͷඥ෇͚ 30 • procfsͱ͔netlink͔Βͷ઀ଓΠϕϯτʹϓϩηεͷ৘ใ͕ͳ͍ • ͔ΘΓʹ઀ଓΠϕϯτʹ͸ιέοτͷinode৘ใ͸͋Δ • /proc/<pid>/fd ҎԼ͔Βinode৘ใ͸ͱΕΔ • ϓϩηεϦετͱɼ઀ଓΠϕϯτϦετͷ2ͭͷinodeΛΩʔʹ݁߹ • Nested Loop݁߹ https://memo.yuuk.io/entry/2019/linux-process-and-connection
31. 31 skb->sk->socket->file->f_owner->pid
32. 32 Ԡ༻
33. ؔ܎ੑʹج͍ͮͨpingϞχλϦϯά 33 ICMP/TCP/HTTP ݱࡏ ͜͏͍ͨ͠ ࣮ࡍͷ௨৴ܦ࿏Λ ؂ࢹ͍ͨ͠ ICMP/TCP/HTTP
34. lstfͰಈతʹ௨৴ઌΛࣝผ 34 • lstfͷΑ͏ͳػߏͳΒ௨৴ઌΛಈతʹऔಘͰ͖Δ • ෦෼ωοτϫʔΫো֐ൃੜ࣌ʹdeadmanͱ૊Έ߹ΘͤΔ • github.com/upa/deadman ͸pingͷTUI؂ࢹπʔϧ • deadman͸؂ࢹઌΛ੩తʹઃఆ͢ΔͷͰɼlstfͰಈతੜ੒͢Δ • ؂ࢹΤʔδΣϯτʹ૊ΈࠐΜͰৗ࣌؂ࢹ
35. 35 Transtracer(WIP) github.com/yuuki/transtracer
36. γεςϜߏ੒ 36 Host 1 Host 2 Host N Postgres Tracer Tracer Tracer Systems Administrator ɾϗετ্ʹTracerΤʔδΣϯτΛ഑ஔ ɾ֤TracerΤʔδΣϯτ͸औಘͨ͠઀ଓ৘ ใΛPostgreSQLʹอଘ ɾγεςϜ؅ཧऀ͸PostgresʹΞΫηε ͠ɼෳ਺ͷϗετʹ·͕ͨΓґଘؔ܎Λ औಘ
37. 37 ख๏ͷ੍໿
38. ੍໿ 38 • TCP/UDP૚ͷ৘ใͳͷͰɼHTTPͳͲͷL7ϓϩτίϧͷ৘ใ͕ Θ͔Βͳ͍ • ϦΫΤετύε΍ϦΫΤετ୯ҐͷԠ౴଎౓ͳͲ • ผͷπʔϧͱ૊Έ߹ΘͤΔ͜ͱʹͳΔ • ϑΥϫʔυϓϩΩγ΍NATͷΑ͏ͳதܧऀΛڬΜͩͱ͖ͷґଘ ؔ܎Λ௥੻Ͱ͖ͳ͘ͳΔ • NAT: NATͷઌΛޓ͍ʹೝࣝͰ͖ͳ͍ • ϓϩΩγ: ࣮ࡍͷґଘΑΓ΋ଟ͘ͷґଘ͕͋ΔΑ͏ʹΈ͑Δ
39. 39 ·ͱΊ
40. ·ͱΊ 40 • ໰୊ҙࣝ: ෼ࢄγεςϜͷґଘؔ܎͕ෳࡶԽͯ͠มߋ଎౓͕௿Լ • ໨త: ؔ܎ੑʹண໨ͨ͠Obserbabilityͷ֬อ • ՝୊: ϚΠΫϩαʔϏεؒҎ֎ͷ௨৴ͷґଘΛ௥੻͢Δπʔϧ͸ ·ͩͳ͍ • ղܾ: LinuxͷTCP/UDP૚ͷ઀ଓΠϕϯτΛ࢖ͬͯɼܰྔτϨʔ γϯά
41. ࠓޙͷ՝୊ 41 • ϙʔϦϯάͱΠϕϯτ௨஌Λ૊Έ߹Θͤͯਫ਼౓޲্ • eBPFͰconnect(2)ͱaccept(2)ͰΠϕϯτ௨஌ • ίϯςφରԠ