Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
Observability: Mackerel による観測と Mackerel の観測 / ...
Search
astj
PRO
September 17, 2019
Technology
2
2.1k
Observability: Mackerel による観測と Mackerel の観測 / NoOps Meetup Tokyo #8
https://noops.connpass.com/event/145092/
astj
PRO
September 17, 2019
Tweet
Share
More Decks by astj
See All by astj
ISUCON14公式反省会LT: 社内ISUCONの話
astj
PRO
0
330
Mackerel の時系列データベースにおける Redis Cluster の利用と Amazon ElastiCache への移行について / AWS Purpose-Built Databases Week
astj
PRO
3
5.8k
Mackerel のコンテナ監視の近況 / Mackerel Meetup #13
astj
PRO
1
2.3k
サービス開発と健全なプロダクトメンテナンスを開発チームで持続的に取り組む / Developers Boost KANSAI
astj
PRO
1
1.4k
Mackerel and Stripe / Qiita x Stripe Meetup
astj
PRO
0
5.5k
Mackerel をオンプレミスから AWS に移してからの1年半を振り返る / Hatena Engineer Seminar #11
astj
PRO
0
1.1k
AWS で実現した Mackerel 時系列データ1分粒度長期保存の裏側 / Mackerel Meetup #11 Tokyo
astj
PRO
6
30k
稼働中の Web サービスの perl のバージョンを上げていく
astj
PRO
0
7.4k
Perl 6 で Web Application Framework をつくる
astj
PRO
0
4.6k
Other Decks in Technology
See All in Technology
タイミーのデータモデリング事例と今後のチャレンジ
ttccddtoki
6
2.3k
ドメイン特化なCLIPモデルとデータセットの紹介
tattaka
2
570
改めてAWS WAFを振り返る~業務で使うためのポイント~
masakiokuda
2
230
United airlines®️ USA Contact Numbers: Complete 2025 Support Guide
oliversmith12
0
140
Tokyo_reInforce_2025_recap_iam_access_analyzer
hiashisan
0
170
Lazy application authentication with Tailscale
bluehatbrit
0
150
Tech-Verse 2025 Global CTO Session
lycorptech_jp
PRO
0
1.6k
生成AIで小説を書くためにプロンプトの制約や原則について学ぶ / prompt-engineering-for-ai-fiction
nwiizo
6
4k
論文紹介:LLMDet (CVPR2025 Highlight)
tattaka
0
300
生成AI時代の開発組織・技術・プロセス 〜 ログラスの挑戦と考察 〜
itohiro73
1
410
AI専用のリンターを作る #yumemi_patch
bengo4com
5
3.9k
Delta airlines®️ USA Contact Numbers: Complete 2025 Support Guide
airtravelguide
0
330
Featured
See All Featured
Chrome DevTools: State of the Union 2024 - Debugging React & Beyond
addyosmani
7
730
The Art of Programming - Codeland 2020
erikaheidi
54
13k
The Art of Delivering Value - GDevCon NA Keynote
reverentgeek
15
1.5k
Making the Leap to Tech Lead
cromwellryan
134
9.4k
The Invisible Side of Design
smashingmag
301
51k
Thoughts on Productivity
jonyablonski
69
4.7k
Visualizing Your Data: Incorporating Mongo into Loggly Infrastructure
mongodb
46
9.6k
YesSQL, Process and Tooling at Scale
rocio
173
14k
What's in a price? How to price your products and services
michaelherold
246
12k
Git: the NoSQL Database
bkeepers
PRO
430
65k
Designing for humans not robots
tammielis
253
25k
Docker and Python
trallard
44
3.5k
Transcript
Observability: Mackerel ʹΑΔ؍ଌ ͱ Mackerel ͷ؍ଌ 2019/09/17 NoOps Meetup Tokyo
#8 Asato WAKISAKA (Hatena)
Asato WAKISAKA GitHub: @astj / Twitter @ast_j ͯͳ ΞϓϦέʔγϣϯΤϯδχΞ Mackerel
։ൃνʔϜ (2016/08~) Mackerel ςοΫϦʔυ (2018/05~)
कඋൣғ: ϑϩϯτΤϯυ͔ΒSREing·Ͱʢࣗশʣ Scala, Golang, TypeScript, …
Mackerel
None
• Mackerel ͬͯΔਓ • ͋Γ͕ͱ͏͍͟͝·͢ʂ • Mackerel ৮ͬͨ͜ͱ͕͋Δਓ • ͋Γ͕ͱ͏͍͟͝·͢ʂʂ
• Mackerel ͬͯΔਓ • ͋Γ͕ͱ͏͍͟͝·͢ʂʂʂʂ Ξϯέʔτ
None
ࢹ
Monitoring
Observability
?
Observability ͱ Monitoring
• Observability: ʮՄ؍ଌੑʯ • γεςϜͷ෦ঢ়ଶ͕औಘͰ͖Δঢ়ଶ • 3ͭͷओཁͳߏཁૉ • Metrics, Logging,
Tracing • Mackerel ͜ͷ͏ͪ Metrics த৺
• ʮՄ؍ଌੑʯ≠ʮ؍ଌʯ • ʮՄ؍ଌੑʯγεςϜ͕࣋ͭੑ࣭ • ʮ؍ଌʯγεςϜ֎෦ͷߦಈ
Metrics Observability Log Observability Trace Observability Metrics Observation Log Observation
Trace Observation
Metrics Observability Log Observability Trace Observability Metrics Observation Log Observation
Trace Observation • Server Metrics • Middleware Metrics • Application Metrics • Business KPI • Etc… • Alerting • Dashboard Visualization • Analysis • Etc…
Metrics Observability Log Observability Trace Observability Metrics Observation Log Observation
Trace Observation ʮϝτϦοΫࢹʯ͜ͷลʁ • Server Metrics • Middleware Metrics • Application Metrics • Business KPI • Etc… • Alerting • Dashboard Visualization • Analysis • Etc…
• ͱ͍͑ɺͦͷઌͰਓ͕ؒ؍ଌ͍ͯ͠Δͣ • ݹయతʹΞϥʔτରԠͳͲ • ʮάϥϑΛݟΔʯ؍ଌ
Metrics Observability Log Observability Trace Observability Metrics Observation Log Observation
Trace Observation wਓؒ • ͋Ε • ͜Ε ˞͜Εۃ
• લஈʹʮՄ؍ଌੑʯͦͷͷ • ޙஈͷʮ؍ଌʯΛ;ͨͭʹ͚ͯଊ͑Δ • ʮ؍ଌͷ࣮ʯ • ʮʢਓؒͷʣ؍ଌʯ
Observability Observability Observability Observation Observation Observation Observe impl. Observe impl.
Observe Impl.
Observability Observability Observability Observation Observation Observation Observe impl. Observe impl.
Observe Impl. ٕज़ͱͯ͠ͷ Observability ͕ ͜ͷลΓΛࢦ͢͜ͱ͋Ε
Observability Observability Observability Observation Observation Observation Observe impl. Observe impl.
Observe Impl. ιϦϡʔγϣϯͷͰ Observability ͕ݴٴ͞ΕΔ࣌ ͜͜·Ͱؚ·ΕΔ͜ͱଟ͍
……Ͱɺ͍͍Ͱ͔͢Ͷʁ
• ٞͷ༨͍Ζ͍Ζ͋Γͦ͏ • ͔͠͠ࠓ͜ͷͷਂ͍͠·ͤΜ • ߹্ࠨΛՄ؍ଌੑͱͯ͠͠·͢
Observability Observability Observability Observation Observation Observation Observe impl. Observe impl.
Observe Impl. • ϝτϦοΫࢹ: • ϝτϦοΫΛऩू͢Δ • ऩूͨ͠ϝτϦοΫΛूੵ͢Δ • άϥϑμογϡϘʔυͰՄࢹԽ͢Δ • ΞϥʔτΛൃใ͢Δ
Observability Observability Observability Observation Observation Observation Observe impl. Observe impl.
Observe Impl. ϝτϦοΫऩू ϝτϦοΫूੵ • άϥϑ • Ξϥʔτ • μογϡϘʔυ ˞Πϝʔδ • ਓؒ • ࣗಈԽରԠ
Observability Observability Observability Observation Observation Observation Observe impl. Observe impl.
Observe Impl. • ؍ଌ͢ΔͨΊʹ؍ଌՄೳʹ͢Δඞཁ͕͋Δ • Մ؍ଌੑࢹͷલఏͱͳΔ • ʢͷͰɺࢹͷจ຺ͰՄ؍ଌੑʹݴٴ͞ΕΔʣ
None
Mackerel ͷ֓೦
ϗετ
• ࢹɾཧͷىʹͳΔ֓೦ • యܕతʹαʔόʔ • ʮαʔόʔͷΑ͏ͳͷʯϗετ • ίϯςφͷू߹ (e.g. ECS
Task / k8s pod) • ΫϥυϦιʔε (e.g. Lambda Function) ϗετ
Service:Blog Role:App Role:DB Host A Host B Host C Host
D Service:Bookmark Role:App Host E Host F … αʔϏεɾϩʔϧ
• ϗετΛʮׂʯͰྨ • μογϡϘʔυ্Ͱάϧʔϐϯά͞ΕΔ • αʔϏεϩʔϧ୯ҐͰࢹϧʔϧΛཧ αʔϏεɾϩʔϧ
None
None
None
෦ঢ়ଶͷऩू
• ࣌ܥྻͰࢀরՄೳ ϝτϦοΫ
• ୯७ͳ൱ͱϝοηʔδΛه • ฏৗ࣌ͷཤྺهʹΒͳ͍ νΣοΫࢹ
• ϩάʹର͢Δ(ϝτϦοΫ|νΣοΫ)ࢹͰදݱ • ϩάΛࠪͯ͠ύλʔϯϚον͢Δ • LTSV ܗࣜͷΞΫηεϩάΛूܭ͢Δ ϩάΛࢹ͢Δ
None
ऩूͷλʔήοτ • ΤʔδΣϯτܥ • αʔόʔ • ίϯςφ • αʔϏεϝτϦοΫ •
URL ֎ܗࢹ • ΠϯςάϨʔγϣϯ
• αʔόʔ্Ͱಈ͘ΤʔδΣϯτ • ΤʔδΣϯτ͔Β Mackerel ʹ POST ͢Δ • γεςϜϝτϦοΫࣗಈͰऩू
• CPU, memory, interface… • ϓϥάΠϯͰऔಘใΛ֦ுͰ͖Δ αʔόʔ͔Βͷऩू
Server Mackerel mackerel-agent Fluentd Golang app OS stats monitor-agent Plugins
HTTP gostats POST ྫ
• ECS 1 Task k8s 1 pod Λ1ϗετ •
αΠυΧʔίϯςφͱͯ͠ΤʔδΣϯτΛಋೖ • γεςϜϝτϦοΫΛࣗಈͰऩू • ϓϥάΠϯͰ֦ுՄೳ ίϯςφ͔Βͷऩू
Task Task Task Mackerel Agent Container Fluentd container Golang app
container monitor-agent Plugins HTTP gostats POST Orchestration Metadata ྫ
• ಛఆͷϗετʹඥ͔ͳ͍ϝτϦοΫ • Ϗδωε KPI ɺαʔϏεશମͷϨΠςϯγ • API Ͱࣗ༝ʹߘ •
e.g. fluentd αʔϏεϝτϦοΫ Service:Bookmark Role:App Host
• Mackerel ଆ͔Β URL ʹଓ • ଓՄ൱ϨεϙϯελΠϜͳͲΛऩू URL ֎ܗࢹ
• ΫϥυࣄۀऀͷϞχλϦϯά API ͔Βऔಘ • ΤʔδΣϯτͳ͠ͰऩूͰ͖Δ • ऩूϝτϦοΫΫϥυࣄۀऀґଘ AWS /
Azure ΠϯςάϨʔγϣϯ Mackerel User Cloud Services Crawl
ऩू͢Δ༰
• γεςϜϝτϦοΫࣗಈͰऩू • ϛυϧΣΞެࣜϓϥάΠϯͰऩू • ΞϓϦέʔγϣϯͷϝτϦοΫʁʁ
• γεςϜϝτϦοΫࣗಈͰऩू • ϛυϧΣΞެࣜϓϥάΠϯͰऩू • ΞϓϦέʔγϣϯͷϝτϦοΫʁʁ
• ݴޠϑϨʔϜϫʔΫ͚ͷϓϥάΠϯ͋Δ • JVM Go GC ౷ܭ؍ଌͰ͖Δ •
Γͳ͍ϝτϦοΫͷ؍ଌखஈ͕ඞཁ
Observability Observability Observability Observation Observation Observation Observe impl. Observe impl.
Observe Impl. • ؍ଌ͢ΔͨΊʹ؍ଌՄೳʹ͢Δඞཁ͕͋Δ • Մ؍ଌੑࢹͷલఏͱͳΔ • ʢͷͰɺࢹͷจ຺ͰՄ؍ଌੑʹݴٴ͢Δʣ ࠶ܝ • ؍ଌ͢ΔͨΊʹ؍ଌՄೳʹ͢Δඞཁ͕͋Δ
؍ଌ͍ͨ͠ϝτϦοΫԿΒ͔ͷखஈͰ؍ଌՄೳͰ͔͢ʁ ؍ଌՄೳʹ͢Δ ΤʔδΣϯτʢͷϓϥάΠϯʣͰ؍ଌɾऩू͢Δ NO YES
؍ଌ͍ͨ͠ϝτϦοΫԿΒ͔ͷखஈͰ؍ଌՄೳͰ͔͢ʁ ؍ଌՄೳʹ͢Δ ΤʔδΣϯτʢͷϓϥάΠϯʣͰ؍ଌɾऩू͢Δ NO YES
• ͲͷΑ͏ʹ؍ଌ͞ΕΔ͔Ͱ࣮͕ܾ·Δ ؍ଌՄೳʹ͢Δ
؍ଌ͍ͨ͠ϝτϦοΫԿΒ͔ͷखஈͰ؍ଌՄೳͰ͔͢ʁ ؍ଌՄೳʹ͢Δ ΤʔδΣϯτʢͷϓϥάΠϯʣͰ؍ଌɾऩू͢Δ NO YES
• ϓϥάΠϯΛࣗ࡞ͯ͠ҙͷखஈͰ؍ଌ • HTTP Ͱ JSON ܗࣜΛಡΈࠐΜͰ؍ଌ ؍ଌखஈ
• ϓϥάΠϯ = ࣮ߦϑΝΠϧ • ॴఆͷग़ྗͰϝτϦοΫ໊ͱΛग़ྗ͢Δ͚ͩ • ࣮ߦڥͰ“ݟ͑Δ”ͳΒԿͰऔΕΔ ࣗ࡞ϓϥάΠϯͰͷϝτϦοΫऩू $
mackerel-plugin-uptime uptime.seconds 7749392 1568662368
• ࣮ࣗମ Bash Ͱ Go ͰԿͰ • ग़ྗϑΥʔϚοτ͕ૉͳͷͰ࣮Λબͳ͍ • ؍ଌରʹ͋Θ࣮ͤͯ͢Δඞཁ͕͋Δ
• ؍ଌରଆͷࣗ༝ߴ͘ͳΔ ࣗ࡞ϓϥάΠϯͰͷϝτϦοΫऩू
HTTP ܦ༝Ͱͷऩू • mackerel-plugin-json • ؍ଌ͍ͨ͠Λ JSON Ͱ HTTP ৴͢Δ
• ϓϥάΠϯ͔ΒͦͷΤϯυϙΠϯτΛಡΈࠐΉ
ఆ • Q: Prometheus Exporter Έ͍ͨͳʁ • => ํੑ yesʢͬͱγϯϓϧʣ
• Q: Prometheus Exporter ͑ͳ͍ͷʁ • => ……!
HTTP ܦ༝Ͱͷऩू $ mackerel-plugin-json \ > -url=http://localhost/stats active3 1568666460 waiting
2 1568666460 $ curl -fsSL http://localhost/stats {“active”:3,”waiting”:2}
HTTP ܦ༝Ͱͷऩू
• ͲͪΒ͕ྑ͍͔߹ʹΑΔ • طʹHTTPΛΔͳΒHTTP͕ศར • طଘͷ؍ଌखஈ͕͋Ε࠶ར༻͍ͨ͠
ऩूͨ͠ϝτϦοΫͷར༻
• Mackerel Ͱͷओͳ͍ಓ • ࢹʢΞϥʔτʣ • ՄࢹԽʢάϥϑɾμογϡϘʔυʣ
ࢹʢΞϥʔτʣ
;ͨͭͷΞϥʔτ • ଈ࣌ରԠΛଅ͢Ξϥʔτ • ӡ༻ऀͱͯ͠θϩʹ͍ͨ͠ʂ • Information ͱͯ͠ͷΞϥʔτ
ଈ࣌ରԠͷΞϥʔτ (on-call) • γεςϜͷՔಇ͕ଛͳΘΕ͍ͯΔ • ରԠΛඞཁͱ͢Δ • “Δ·Ͱ์ஔ”͕ڐ༰ͳΒଈ࣌ରԠͰͳ͍
Information ͱͯ͠ͷΞϥʔτ • ٸ͕ͳ͍͕ԿΒ͔ͷΞΫγϣϯΛٻΊΔ • ྫ: Խϗετ͕1μϯ • γεςϜ͕ଓߦՄೳͳൣғͷʮҟৗʯ •
ಉ͡ࢦඪ͕͞ΒʹѱԽ͢Εଈ࣌ରԠ
• ࣮ࡍʹγεςϜ͕Քಇ͍ͯ͠Δ͔ Ͳ͏͔Λࢹ͢Δ • e.g. ֎ܗࢹ, ΤϥʔϨʔτ • ˱ CPU
100%Ͱಈ͍ͯΕྑ͍ ଈ࣌ରԠͷΞϥʔτ
• ͱ͍͑߹ʹΑΓ͚Γ • ͷՔಇͷࢹͰͳͯ͘ɺ ͔ͦ͜ΒՔಇঢ়گ͕ԋ៷ՄೳͳΒ༗༻
Information ͱͯ͠ͷΞϥʔτ • γεςϜՔಇҎ֎ͷࢦඪऔΓಘΔ • ଈ࣌ରԠͱಉ͡௨ܦ࿏ʹࡌͤͳ͍
Ξϥʔτશൠ • ͳΜͰ͔ΜͰΞϥʔτʹ͠ͳ͍ • ຊʹඞཁͳΞϥʔτ͕ຒΕΔ • Information ͩΒͩΒ૿͕͑ͪ
ՄࢹԽ
• ෦ͷঢ়ଶΛ࣌ܥྻ͖ͭͰه͢Δ • खಈͰऩू͠ͳͯ͘؍ଌͰ͖Δ • ݱࡏͷঢ়ଶΛѲ͢Δखஈ • ఆৗ࣌ɾো࣌Θͣ • աڈͷঢ়ଶΛৼΓฦΔखஈ
άϥϑɾμογϡϘʔυ
؍ଌΛҭͯΔ
ඞཁͳ؍ଌྔ͕࠷ॳ͔Βἧ͏ͱݶΒͳ͍
• ։ൃɾݕূதʹؾͮ͘ • ࢹ༻ҙ͠Α͏ͱͯ͠ෆʹؾͮ͘ • োى͖ͯΈͯෆʹؾͮ͘ ͨͱ͑…
ඞཁͳ؍ଌ࠷ॳ͔Βἧ͏ͱݶΒͳ͍
• ։ൃɾݕূதʹؾͮ͘ • ෛՙࢼݧͰؾͮ͘ͱඒ͍͠ʂ • োى͖ͯΈͯෆʹؾͮ͘ ͨͱ͑…
αΠΫϧΛճ͍ͯ͘͠
• ࠷ॳ͔ΒඋͰ͖Εྑ͍ • ಛʹඞਢͷ߲͍͑ͯ͘ • γεςϜͷՔಇͷࢹʂʂ • ܧଓతʹมߋɾվળ͍ͯ͘͠
None
Mackerel Ͱ Mackerel Λ؍ଌ͢Δ
Mackerel ͷٕज़ελοΫ • AWS ্ʹߏங • EC2 ӡ༻ͱ ECS Fargate
͕ࠞࡏ • ϝΠϯͷ JVM ͳͲ͕ EC2 • গͣͭ͠ίϯςφϫʔΫϩʔυʹҠߦத
ઢ
ίϯςφศར
• Πϯελϯεͷੈ͕ཁΒͳ͍ • Πϯϑϥ·ͰؔΘΔߏมߋ͕खܰ • “chef ߋ৽ͯ͠Πϯελϯε࡞Γͯ͠ೖΕ ͑ͯ…” • ʮخ͘͠ͳ͍͜ͱʯ͡ΌΜ……
ઢ͜͜·Ͱ
Mackerel ͷٕज़ελοΫ ࣌ܥྻσʔλϕʔε: ϚωʔδυαʔϏεϕʔε
ઢ
αʔόϨεΞʔΩςΫνϟศར
• ૄ݁߹ɾϦτϥΠՄೳͳઃܭ • ϝοηʔδΩϡʔΛڬΉ • Lambda ͷॲཧ༰Λႈʹઃܭ • ύϥϝʔλมߋͰεέʔϧ͢Δ •
ʢ FaaS ಛ༗ͷӡ༻՝θϩͰͳ͍ʣ
ઢ͜͜·Ͱ
Mackerel ͷٕज़ελοΫ ʢڵຯ͕͋ΕաڈͷొஃࢿྉΛ͝ཡԼ͍͞ʂʣ
JVM ΞϓϦέʔγϣϯͷ؍ଌ • ϝτϦΫεɾࢹ: mackerel • jstat ͱ jolokia ʹΑΔJVM෦ߏͷ؍ଌ
• αʔόʔ͝ͱͷϨΠςϯγࢹ • ίωΫγϣϯϓʔϧࢹ • etc…
JVM ΞϓϦέʔγϣϯͷ؍ଌ • APM: NewRelic APM • ΑΓਫ਼៛ͳ෦ੳ͕ඞཁͳͱ͖ʹ • ϩάղੳ:
fluentd => Kibana • ΞΫηεϩάɺΞϓϦέʔγϣϯϩά
։ൃڥͷ؍ଌ • ։ൃڥʢ݉εςʔδϯάڥʣ • ຊ൪ͱಉ༷ͷߏʢنখ͍͞ʣ • ಉ༷ͷ؍ଌπʔϧߏ • => ϦϦʔεલ͔Β։ൃڥͰ؍ଌ
ఆ • Q: ։ൃڥͷࢹͷίετʁ • => զʑMackerelΛແྉͰ͑Δʢଇʣ
• த֩ʹ Mackerel • ීஈͷʮࢹʯʮཧʯͷى͜͜ • ଞͷπʔϧͱΈ߹Θͤͯ؍ଌΛิڧ
None
͝ਗ਼ௌ͋Γ͕ͱ͏͍͟͝·ͨ͠