Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
簡略化したネットワーク依存関係グラフの可視化基盤 / Simplified network d...
Search
Yuuki Tsubouchi (yuuk1)
November 17, 2018
Research
2
22k
簡略化したネットワーク依存関係グラフの可視化基盤 / Simplified network dependency graph visualization
第3回Web System Architecture研究会@東京
Yuuki Tsubouchi (yuuk1)
November 17, 2018
Tweet
Share
More Decks by Yuuki Tsubouchi (yuuk1)
See All by Yuuki Tsubouchi (yuuk1)
博士論文公聴会: Scaling Telemetry Workloads in Cloud Applications: Techniques for Instrumentation, Storage, and Mining / PhD Defence
yuukit
1
25
博士学位論文予備審査 / Scaling Telemetry Workloads in Cloud Applications: Techniques for Instrumentation, Storage, and Mining
yuukit
1
1.7k
MetricSifter:クラウドアプリケーションにおける故障箇所特定の効率化のための多変量時系列データの特徴量削減 / FIT 2024
yuukit
2
190
工学としてのSRE再訪 / Revisiting SRE as Engineering
yuukit
19
13k
Cloudless Computingの論文紹介
yuukit
2
490
#SRE論文紹介 Detection is Better Than Cure: A Cloud Incidents Perspective V. Ganatra et. al., ESEC/FSE’23
yuukit
3
1.8k
エンジニアのためのSRE論文への招待 / Introduction to SRE Papers for Engineers
yuukit
2
11k
博士課程での研究まとめ 2023年1月版 / Summary of my research in the PhD course
yuukit
1
280
AI時代に向けたクラウドにおける信頼性エンジニアリングの未来構想 / DICOMO2022 6A-1
yuukit
7
3k
Other Decks in Research
See All in Research
Zipf 白色化:タイプとトークンの区別がもたらす良質な埋め込み空間と損失関数
eumesy
PRO
8
1.4k
The Economics of Platforms 輪読会 第1章
tomonatu8
0
150
VisFocus: Prompt-Guided Vision Encoders for OCR-Free Dense Document Understanding
sansan_randd
1
460
Building Height Estimation Using Shadow Length in Satellite Imagery
satai
3
200
文書画像のデータ化における VLM活用 / Use of VLM in document image data conversion
sansan_randd
2
530
【NLPコロキウム】Stepwise Alignment for Constrained Language Model Policy Optimization (NeurIPS 2024)
akifumi_wachi
3
540
Composed image retrieval for remote sensing
satai
3
240
AWS 音声基盤モデル トーク解析AI MiiTelの音声処理について
ken57
0
140
Global Evidence Summit (GES) 参加報告
daimoriwaki
0
240
Poster: Feasibility of Runtime-Neutral Wasm Instrumentation for Edge-Cloud Workload Handover
chikuwait
0
350
o1 pro mode の調査レポート
smorce
0
120
Prithvi-EO-2.0: A Versatile Multi-Temporal Foundation Model for Earth Observation Applications
satai
3
150
Featured
See All Featured
The Success of Rails: Ensuring Growth for the Next 100 Years
eileencodes
44
7k
Producing Creativity
orderedlist
PRO
344
40k
Keith and Marios Guide to Fast Websites
keithpitt
411
22k
Testing 201, or: Great Expectations
jmmastey
42
7.2k
GraphQLの誤解/rethinking-graphql
sonatard
68
10k
Building a Scalable Design System with Sketch
lauravandoore
461
33k
Improving Core Web Vitals using Speculation Rules API
sergeychernyshev
10
510
Facilitating Awesome Meetings
lara
52
6.2k
CSS Pre-Processors: Stylus, Less & Sass
bermonpainter
356
29k
4 Signs Your Business is Dying
shpigford
182
22k
ピンチをチャンスに:未来をつくるプロダクトロードマップ #pmconf2020
aki_iinuma
114
50k
Rails Girls Zürich Keynote
gr2m
94
13k
Transcript
؆ུԽͨ͠ ωοτϫʔΫґଘؔάϥϑ ͷՄࢹԽج൫ id:y_uuki, id:masayoshi, ͯͳ 2018/11/17 ୈ3ճ Web System
Architectureݚڀձ@౦ژ
࣍ 1. എܠͱత 2. ఏҊख๏ 3. ·ͱΊ
1. എܠͱత
എܠ • ωοτϫʔΫґଘؔͷෳࡶԽ • 10୯ҐͷαʔϏεͷظӡ༻ͷؒʹॏͳΔ૿ங • ϚΠΫϩαʔϏεͳͲͷۙͷτϨϯυมԽ • γεςϜཧऀ͕هԱͰ͖ͳ͍ݸʑͷ௨৴ͷґଘؔ •
Өڹൣғ͕Θ͔Βͳ͍ͨΊɺมߋͷͨͼʹґଘؔΛ ௐࠪ͢Δඞཁ͕ൃੜ
ઌߦख๏ • ϑΝΠΞΥʔϧͷϩΪϯάػߏύέοτΩϟϓνϟ • TCP/UDP௨৴ϩάΛूܭαʔόʹసૹ͠ґଘՄࢹԽ • ωοτϫʔΫεΠον͔Βͷ౷ܭใऔಘ (sFlow/NetFlowͳͲ) • αΠυΧʔϓϩΩγΞϓϦέʔγϣϯϩά
(ࢄτϨʔγϯά)
֤ઌߦख๏ͷ՝ • ϑΝΠΞΥʔϧϩΪϯά(iptables) • a) ύέοτϨΠςϯγ͕૿Ճ • ύέοτΩϟϓνϟ(tcpdump) • b)
ύέοτશମΛऩू͢ΔͨΊɺऩू·ͨղੳͷͨΊͷෛՙͷ૿Ճ • sFlow/NetFlow • c) αϯϓϦϯά͞ΕΔͨΊґଘཏ͕Լ • ΞϓϦέʔγϣϯ(L7)ͰͷϦΫΤετ୯ҐͷϩΪϯά • d) ΞϓϦέʔγϣϯͷมߋ͕ඞཁ
ઌߦख๏શମͷ՝ • e) ऩूσʔλ͕ґଘؔΛΔ͚ͩʹ͕ͯ͠ଟ͘ • ύέοτɺTCPଓɺϦΫΤετ୯Ґͷใ • ྫ͑ΤϑΣϝϥϧϙʔτͷଓใͳͲ • f)
ґଘؔάϥϑߏͷෳࡶԽ • ෛՙࢄͳͲʹΑΓϗετ͕૿Ճͨ͠߹ʹϊʔυͱΤο δͷݸ͕૿͑Δ
2. ఏҊख๏
ཁ݅ • 1)ΞϓϦέʔγϣϯੑೳʹӨڹΛ༩͑ͳ͍ଓใͷऩूՄೳ (՝ a),b),c)) • ݱ࣮తͳਫ਼Ͱଓใͷऩूཱ྆ • 2)ΞϓϦέʔγϣϯͷมߋෆཁͰऩूՄೳ (՝d)
• 3)ूԽ͞Εͨϊʔυ୯ҐͰՄࢹԽՄೳ (՝e) • 4)ϗετ୯ҐͰͳ͘ϗετͷෳάϧʔϓ୯ҐͰՄࢹԽՄೳ ՝ f) • 5)ҙͷ࣌ؒൣғͰͷґଘؔΛग़ྗՄೳ
ఏҊख๏ • ΞϓϦέʔγϣϯͷมߋෆཁͳωοτϫʔΫ௨৴ใͷऩू (ཁ݅2) • OSʹ͍߹ΘͤऔಘՄೳͳTCPଓใΛར༻ • ΞϓϦέʔγϣϯੑೳʹӨڹΛ༩͑ͳ͍TCPଓใͷऩू (ཁ݅1) •
OS͕ͭෛՙͰऔಘՄೳͳTCPଓͷ౷ܭใΛར༻ • ूԽ͞ΕͨΤοδͱϊʔυ୯ҐͰՄࢹԽՄೳ • Τοδ: ಉҰϗετ͔ͭಉҰϦοεϯϙʔτͷTCPଓू (ཁ݅3) • ϊʔυ: αʔόཧπʔϧͷAPIͱ࿈ܞՄೳ (ཁ݅4) • ίωΫγϣϯཧDBʹूԽ͞ΕͨTCPଓΛ͖࣌ؒͰཧՄೳ (ཁ݅5)
ϗετϑϩʔू • Τοδ: ಉҰϗετ͔ͭಉҰϦοεϯϙʔτͷTCPଓू • Τοδͷҹͷ͖: TCPͷΞΫςΟϒΦʔϓϯͱύογϒΦʔϓϯͰఆ port 80 port
3306 port 10000 port 10001 port 10002 port 15000 port 15001 port 15002 ϗετϑϩʔ ϗετϑϩʔ Host Host
ϗετϑϩʔूͷ࣮ • 1. Netlink APIʹΑΓTCPଓใΛऔಘ͢Δ • 2. LISTENεςʔτͷଓͷϩʔΧϧϙʔτͷΈநग़ • 3.
1.ͱ2.Λಥ͖߹Θͤɼଓઌϙʔτ͕ϦοεϯϙʔτͰ͋Ε ΞΫςΟϒΦʔϓϯɼͦΕҎ֎ͷଓύογϒΦʔϓϯͱ ఆ͢Δɽ
send agent agent CMDB Host Host agent Host analyzer sysadmin
get get ఏҊख๏ͷશମϑϩʔ֓ཁ
send agent agent Postgres Host Host agent Host CLI sysadmin
get get ఏҊख๏ͷશମϑϩʔ֓ཁ Mackerel get
CMDBͷεΩʔϚ • nodesςʔϒϧ: ϊʔυใ • IDͱIPΞυϨεͱϙʔτ൪߸ • flowsςʔϒϧ: ϗετϑϩʔใ •
IDͱ͖ͱૹ৴ݩͱૹ৴ઌͷͦΕͧΕͷϊʔυID
Mackerel APIʹΑΔෳϊʔυͷάϧʔϓԽ • ͋Β͔͡Ί֤ϗετΛMackerelʹొ͠ɺෳ͞Εͨϗετ܈Λ1 ͭͷϩʔϧʹ·ͱΊ͓ͯ͘ • MackerelΛར༻ͨ͠ෳϊʔυͷάϧʔϓԽखॱ • analyzer͕CMDB͔ΒϊʔυΛऔಘ •
Mackerel APIʹΑΓϗετใҰཡΛऔಘ֘͠ϊʔυͷIPΞυ ϨεͰϑΟϧλ • ಉҰϩʔϧͷϊʔυΛάϧʔϓԽ
github.com/yuuki/mftracer $ mftctl --level 2 --dest-service blog --dest-roles redis --dest-roles
memcached blog:redis ᵋ<-- 10.0.0.22:many (connections:30) ᵋ<-- 10.0.0.23:many (connections:30) ᵋ<-- 10.0.0.24:many (connections:30) ᵋ<-- 10.0.0.30:many (connections:1) ᵋ<-- 10.0.0.31:many (connections:1) ᵋ<-- 10.0.0.25:many (connections:30) blog:memcached ᵋ<-- 10.0.0.23:many (connections:30) ᵋ<-- 10.0.0.25:many (connections:30)
্࣮ͷ՝ • ωοτϫʔΫτϙϩδͷ॥ʹର͢Δߟྀ • Ϋϥυࣄۀऀ͕ఏڙ͢ΔϚωʔδυαʔϏεΛར༻͍ͯ͠Δ ߹ɼIPΞυϨε͔Β࣮ମΛͨͲΔ͜ͱͷࠔ • ͋ΔಛఆͷϊʔυΛࢦఆ͠ɼࢦఆͨ͠ϊʔυ”͔Β”ଓ͢Δ ϊʔυҰཡΛऔಘ͢Δ࣮ (ٯՄೳ)
• ࣌ؒൣғΛࢦఆͨ͠ґଘؔͷऔಘ • ݱࡏ͔Βաڈͯ͢ͷґଘؔΛऔಘ͢ΔΑ͏ʹͳ͍ͬͯΔ
3. ·ͱΊ
·ͱΊ • γεςϜཧऀ͕ݸʑͷωοτϫʔΫґଘؔΛهԱͰ͖ͳ͍ • ΞϓϦέʔγϣϯΛมߋͤͣʹɼΞϓϦέʔγϣϯʹӨڹΛ༩ ͑Δ͜ͱͳ͘ɼదͳநͰใΛऔಘՄೳͳґଘؔղੳ γεςϜ
ࠓޙͷ՝ • αʔϕΠ • ֤ωοτϫʔΫͰઌߦݚڀ͕͋ΔͨΊɺຊݚڀͷཱͪҐஔΛ໌֬Խ • ධՁ • ϑΝΠΞΥʔϧϩΪϯάͱύέοτΩϟϓνϟͷϨΠςϯγ૿େͷධՁ •
ͯ͢ͷଓใΛऔಘͰ͖ΔΘ͚Ͱͳ͍ͨΊɼଓใͷऔಘΛ֬ ೝ͠ɼ࣮ӡ༻ʹ͓͍ͯɼेͳਫ਼Ͱ͋Δ͜ͱΛ֬ೝ • ల • ಉ͡Α͏ͳ௨৴Λ͍ͯ͠ΔϗετΛΫϥελϦϯάਪఆ • ίϯςφܕԾԽڥͰͷґଘؔͷղੳͷൃల