Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
Observability: Mackerel による観測と Mackerel の観測 / ...
Search
astj
PRO
September 17, 2019
Technology
2
2.1k
Observability: Mackerel による観測と Mackerel の観測 / NoOps Meetup Tokyo #8
https://noops.connpass.com/event/145092/
astj
PRO
September 17, 2019
Tweet
Share
More Decks by astj
See All by astj
ISUCON14公式反省会LT: 社内ISUCONの話
astj
PRO
0
370
Mackerel の時系列データベースにおける Redis Cluster の利用と Amazon ElastiCache への移行について / AWS Purpose-Built Databases Week
astj
PRO
3
5.9k
Mackerel のコンテナ監視の近況 / Mackerel Meetup #13
astj
PRO
1
2.4k
サービス開発と健全なプロダクトメンテナンスを開発チームで持続的に取り組む / Developers Boost KANSAI
astj
PRO
1
1.4k
Mackerel and Stripe / Qiita x Stripe Meetup
astj
PRO
0
5.5k
Mackerel をオンプレミスから AWS に移してからの1年半を振り返る / Hatena Engineer Seminar #11
astj
PRO
0
1.2k
AWS で実現した Mackerel 時系列データ1分粒度長期保存の裏側 / Mackerel Meetup #11 Tokyo
astj
PRO
6
30k
稼働中の Web サービスの perl のバージョンを上げていく
astj
PRO
0
7.4k
Perl 6 で Web Application Framework をつくる
astj
PRO
0
4.6k
Other Decks in Technology
See All in Technology
💡Ruby 川辺で灯すPicoRubyからの光
bash0c7
0
110
「全員プロダクトマネージャー」を実現する、Cursorによる仕様検討の自動運転
applism118
21
11k
【実演版】カンファレンス登壇者・スタッフにこそ知ってほしいマイクの使い方 / 大吉祥寺.pm 2025
arthur1
1
840
EncryptedSharedPreferences が deprecated になっちゃった!どうしよう! / Oh no! EncryptedSharedPreferences has been deprecated! What should I do?
yanzm
0
320
要件定義・デザインフェーズでもAIを活用して、コミュニケーションの密度を高める
kazukihayase
0
110
COVESA VSSによる車両データモデルの標準化とAWS IoT FleetWiseの活用
osawa
1
280
DDD集約とサービスコンテキスト境界との関係性
pandayumi
3
280
Automating Web Accessibility Testing with AI Agents
maminami373
0
1.3k
AI開発ツールCreateがAnythingになったよ
tendasato
0
130
大「個人開発サービス」時代に僕たちはどう生きるか
sotarok
20
10k
react-callを使ってダイヤログをいろんなとこで再利用しよう!
shinaps
1
240
新アイテムをどう使っていくか?みんなであーだこーだ言ってみよう / 20250911-rpi-jam-tokyo
akkiesoft
0
260
Featured
See All Featured
The Power of CSS Pseudo Elements
geoffreycrofte
77
6k
GitHub's CSS Performance
jonrohan
1032
460k
Save Time (by Creating Custom Rails Generators)
garrettdimon
PRO
32
1.6k
実際に使うSQLの書き方 徹底解説 / pgcon21j-tutorial
soudai
PRO
188
55k
Fireside Chat
paigeccino
39
3.6k
What's in a price? How to price your products and services
michaelherold
246
12k
How to Think Like a Performance Engineer
csswizardry
26
1.9k
Automating Front-end Workflow
addyosmani
1370
200k
The Myth of the Modular Monolith - Day 2 Keynote - Rails World 2024
eileencodes
26
3k
4 Signs Your Business is Dying
shpigford
184
22k
Side Projects
sachag
455
43k
Keith and Marios Guide to Fast Websites
keithpitt
411
22k
Transcript
Observability: Mackerel ʹΑΔ؍ଌ ͱ Mackerel ͷ؍ଌ 2019/09/17 NoOps Meetup Tokyo
#8 Asato WAKISAKA (Hatena)
Asato WAKISAKA GitHub: @astj / Twitter @ast_j ͯͳ ΞϓϦέʔγϣϯΤϯδχΞ Mackerel
։ൃνʔϜ (2016/08~) Mackerel ςοΫϦʔυ (2018/05~)
कඋൣғ: ϑϩϯτΤϯυ͔ΒSREing·Ͱʢࣗশʣ Scala, Golang, TypeScript, …
Mackerel
None
• Mackerel ͬͯΔਓ • ͋Γ͕ͱ͏͍͟͝·͢ʂ • Mackerel ৮ͬͨ͜ͱ͕͋Δਓ • ͋Γ͕ͱ͏͍͟͝·͢ʂʂ
• Mackerel ͬͯΔਓ • ͋Γ͕ͱ͏͍͟͝·͢ʂʂʂʂ Ξϯέʔτ
None
ࢹ
Monitoring
Observability
?
Observability ͱ Monitoring
• Observability: ʮՄ؍ଌੑʯ • γεςϜͷ෦ঢ়ଶ͕औಘͰ͖Δঢ়ଶ • 3ͭͷओཁͳߏཁૉ • Metrics, Logging,
Tracing • Mackerel ͜ͷ͏ͪ Metrics த৺
• ʮՄ؍ଌੑʯ≠ʮ؍ଌʯ • ʮՄ؍ଌੑʯγεςϜ͕࣋ͭੑ࣭ • ʮ؍ଌʯγεςϜ֎෦ͷߦಈ
Metrics Observability Log Observability Trace Observability Metrics Observation Log Observation
Trace Observation
Metrics Observability Log Observability Trace Observability Metrics Observation Log Observation
Trace Observation • Server Metrics • Middleware Metrics • Application Metrics • Business KPI • Etc… • Alerting • Dashboard Visualization • Analysis • Etc…
Metrics Observability Log Observability Trace Observability Metrics Observation Log Observation
Trace Observation ʮϝτϦοΫࢹʯ͜ͷลʁ • Server Metrics • Middleware Metrics • Application Metrics • Business KPI • Etc… • Alerting • Dashboard Visualization • Analysis • Etc…
• ͱ͍͑ɺͦͷઌͰਓ͕ؒ؍ଌ͍ͯ͠Δͣ • ݹయతʹΞϥʔτରԠͳͲ • ʮάϥϑΛݟΔʯ؍ଌ
Metrics Observability Log Observability Trace Observability Metrics Observation Log Observation
Trace Observation wਓؒ • ͋Ε • ͜Ε ˞͜Εۃ
• લஈʹʮՄ؍ଌੑʯͦͷͷ • ޙஈͷʮ؍ଌʯΛ;ͨͭʹ͚ͯଊ͑Δ • ʮ؍ଌͷ࣮ʯ • ʮʢਓؒͷʣ؍ଌʯ
Observability Observability Observability Observation Observation Observation Observe impl. Observe impl.
Observe Impl.
Observability Observability Observability Observation Observation Observation Observe impl. Observe impl.
Observe Impl. ٕज़ͱͯ͠ͷ Observability ͕ ͜ͷลΓΛࢦ͢͜ͱ͋Ε
Observability Observability Observability Observation Observation Observation Observe impl. Observe impl.
Observe Impl. ιϦϡʔγϣϯͷͰ Observability ͕ݴٴ͞ΕΔ࣌ ͜͜·Ͱؚ·ΕΔ͜ͱଟ͍
……Ͱɺ͍͍Ͱ͔͢Ͷʁ
• ٞͷ༨͍Ζ͍Ζ͋Γͦ͏ • ͔͠͠ࠓ͜ͷͷਂ͍͠·ͤΜ • ߹্ࠨΛՄ؍ଌੑͱͯ͠͠·͢
Observability Observability Observability Observation Observation Observation Observe impl. Observe impl.
Observe Impl. • ϝτϦοΫࢹ: • ϝτϦοΫΛऩू͢Δ • ऩूͨ͠ϝτϦοΫΛूੵ͢Δ • άϥϑμογϡϘʔυͰՄࢹԽ͢Δ • ΞϥʔτΛൃใ͢Δ
Observability Observability Observability Observation Observation Observation Observe impl. Observe impl.
Observe Impl. ϝτϦοΫऩू ϝτϦοΫूੵ • άϥϑ • Ξϥʔτ • μογϡϘʔυ ˞Πϝʔδ • ਓؒ • ࣗಈԽରԠ
Observability Observability Observability Observation Observation Observation Observe impl. Observe impl.
Observe Impl. • ؍ଌ͢ΔͨΊʹ؍ଌՄೳʹ͢Δඞཁ͕͋Δ • Մ؍ଌੑࢹͷલఏͱͳΔ • ʢͷͰɺࢹͷจ຺ͰՄ؍ଌੑʹݴٴ͞ΕΔʣ
None
Mackerel ͷ֓೦
ϗετ
• ࢹɾཧͷىʹͳΔ֓೦ • యܕతʹαʔόʔ • ʮαʔόʔͷΑ͏ͳͷʯϗετ • ίϯςφͷू߹ (e.g. ECS
Task / k8s pod) • ΫϥυϦιʔε (e.g. Lambda Function) ϗετ
Service:Blog Role:App Role:DB Host A Host B Host C Host
D Service:Bookmark Role:App Host E Host F … αʔϏεɾϩʔϧ
• ϗετΛʮׂʯͰྨ • μογϡϘʔυ্Ͱάϧʔϐϯά͞ΕΔ • αʔϏεϩʔϧ୯ҐͰࢹϧʔϧΛཧ αʔϏεɾϩʔϧ
None
None
None
෦ঢ়ଶͷऩू
• ࣌ܥྻͰࢀরՄೳ ϝτϦοΫ
• ୯७ͳ൱ͱϝοηʔδΛه • ฏৗ࣌ͷཤྺهʹΒͳ͍ νΣοΫࢹ
• ϩάʹର͢Δ(ϝτϦοΫ|νΣοΫ)ࢹͰදݱ • ϩάΛࠪͯ͠ύλʔϯϚον͢Δ • LTSV ܗࣜͷΞΫηεϩάΛूܭ͢Δ ϩάΛࢹ͢Δ
None
ऩूͷλʔήοτ • ΤʔδΣϯτܥ • αʔόʔ • ίϯςφ • αʔϏεϝτϦοΫ •
URL ֎ܗࢹ • ΠϯςάϨʔγϣϯ
• αʔόʔ্Ͱಈ͘ΤʔδΣϯτ • ΤʔδΣϯτ͔Β Mackerel ʹ POST ͢Δ • γεςϜϝτϦοΫࣗಈͰऩू
• CPU, memory, interface… • ϓϥάΠϯͰऔಘใΛ֦ுͰ͖Δ αʔόʔ͔Βͷऩू
Server Mackerel mackerel-agent Fluentd Golang app OS stats monitor-agent Plugins
HTTP gostats POST ྫ
• ECS 1 Task k8s 1 pod Λ1ϗετ •
αΠυΧʔίϯςφͱͯ͠ΤʔδΣϯτΛಋೖ • γεςϜϝτϦοΫΛࣗಈͰऩू • ϓϥάΠϯͰ֦ுՄೳ ίϯςφ͔Βͷऩू
Task Task Task Mackerel Agent Container Fluentd container Golang app
container monitor-agent Plugins HTTP gostats POST Orchestration Metadata ྫ
• ಛఆͷϗετʹඥ͔ͳ͍ϝτϦοΫ • Ϗδωε KPI ɺαʔϏεશମͷϨΠςϯγ • API Ͱࣗ༝ʹߘ •
e.g. fluentd αʔϏεϝτϦοΫ Service:Bookmark Role:App Host
• Mackerel ଆ͔Β URL ʹଓ • ଓՄ൱ϨεϙϯελΠϜͳͲΛऩू URL ֎ܗࢹ
• ΫϥυࣄۀऀͷϞχλϦϯά API ͔Βऔಘ • ΤʔδΣϯτͳ͠ͰऩूͰ͖Δ • ऩूϝτϦοΫΫϥυࣄۀऀґଘ AWS /
Azure ΠϯςάϨʔγϣϯ Mackerel User Cloud Services Crawl
ऩू͢Δ༰
• γεςϜϝτϦοΫࣗಈͰऩू • ϛυϧΣΞެࣜϓϥάΠϯͰऩू • ΞϓϦέʔγϣϯͷϝτϦοΫʁʁ
• γεςϜϝτϦοΫࣗಈͰऩू • ϛυϧΣΞެࣜϓϥάΠϯͰऩू • ΞϓϦέʔγϣϯͷϝτϦοΫʁʁ
• ݴޠϑϨʔϜϫʔΫ͚ͷϓϥάΠϯ͋Δ • JVM Go GC ౷ܭ؍ଌͰ͖Δ •
Γͳ͍ϝτϦοΫͷ؍ଌखஈ͕ඞཁ
Observability Observability Observability Observation Observation Observation Observe impl. Observe impl.
Observe Impl. • ؍ଌ͢ΔͨΊʹ؍ଌՄೳʹ͢Δඞཁ͕͋Δ • Մ؍ଌੑࢹͷલఏͱͳΔ • ʢͷͰɺࢹͷจ຺ͰՄ؍ଌੑʹݴٴ͢Δʣ ࠶ܝ • ؍ଌ͢ΔͨΊʹ؍ଌՄೳʹ͢Δඞཁ͕͋Δ
؍ଌ͍ͨ͠ϝτϦοΫԿΒ͔ͷखஈͰ؍ଌՄೳͰ͔͢ʁ ؍ଌՄೳʹ͢Δ ΤʔδΣϯτʢͷϓϥάΠϯʣͰ؍ଌɾऩू͢Δ NO YES
؍ଌ͍ͨ͠ϝτϦοΫԿΒ͔ͷखஈͰ؍ଌՄೳͰ͔͢ʁ ؍ଌՄೳʹ͢Δ ΤʔδΣϯτʢͷϓϥάΠϯʣͰ؍ଌɾऩू͢Δ NO YES
• ͲͷΑ͏ʹ؍ଌ͞ΕΔ͔Ͱ࣮͕ܾ·Δ ؍ଌՄೳʹ͢Δ
؍ଌ͍ͨ͠ϝτϦοΫԿΒ͔ͷखஈͰ؍ଌՄೳͰ͔͢ʁ ؍ଌՄೳʹ͢Δ ΤʔδΣϯτʢͷϓϥάΠϯʣͰ؍ଌɾऩू͢Δ NO YES
• ϓϥάΠϯΛࣗ࡞ͯ͠ҙͷखஈͰ؍ଌ • HTTP Ͱ JSON ܗࣜΛಡΈࠐΜͰ؍ଌ ؍ଌखஈ
• ϓϥάΠϯ = ࣮ߦϑΝΠϧ • ॴఆͷग़ྗͰϝτϦοΫ໊ͱΛग़ྗ͢Δ͚ͩ • ࣮ߦڥͰ“ݟ͑Δ”ͳΒԿͰऔΕΔ ࣗ࡞ϓϥάΠϯͰͷϝτϦοΫऩू $
mackerel-plugin-uptime uptime.seconds 7749392 1568662368
• ࣮ࣗମ Bash Ͱ Go ͰԿͰ • ग़ྗϑΥʔϚοτ͕ૉͳͷͰ࣮Λબͳ͍ • ؍ଌରʹ͋Θ࣮ͤͯ͢Δඞཁ͕͋Δ
• ؍ଌରଆͷࣗ༝ߴ͘ͳΔ ࣗ࡞ϓϥάΠϯͰͷϝτϦοΫऩू
HTTP ܦ༝Ͱͷऩू • mackerel-plugin-json • ؍ଌ͍ͨ͠Λ JSON Ͱ HTTP ৴͢Δ
• ϓϥάΠϯ͔ΒͦͷΤϯυϙΠϯτΛಡΈࠐΉ
ఆ • Q: Prometheus Exporter Έ͍ͨͳʁ • => ํੑ yesʢͬͱγϯϓϧʣ
• Q: Prometheus Exporter ͑ͳ͍ͷʁ • => ……!
HTTP ܦ༝Ͱͷऩू $ mackerel-plugin-json \ > -url=http://localhost/stats active3 1568666460 waiting
2 1568666460 $ curl -fsSL http://localhost/stats {“active”:3,”waiting”:2}
HTTP ܦ༝Ͱͷऩू
• ͲͪΒ͕ྑ͍͔߹ʹΑΔ • طʹHTTPΛΔͳΒHTTP͕ศར • طଘͷ؍ଌखஈ͕͋Ε࠶ར༻͍ͨ͠
ऩूͨ͠ϝτϦοΫͷར༻
• Mackerel Ͱͷओͳ͍ಓ • ࢹʢΞϥʔτʣ • ՄࢹԽʢάϥϑɾμογϡϘʔυʣ
ࢹʢΞϥʔτʣ
;ͨͭͷΞϥʔτ • ଈ࣌ରԠΛଅ͢Ξϥʔτ • ӡ༻ऀͱͯ͠θϩʹ͍ͨ͠ʂ • Information ͱͯ͠ͷΞϥʔτ
ଈ࣌ରԠͷΞϥʔτ (on-call) • γεςϜͷՔಇ͕ଛͳΘΕ͍ͯΔ • ରԠΛඞཁͱ͢Δ • “Δ·Ͱ์ஔ”͕ڐ༰ͳΒଈ࣌ରԠͰͳ͍
Information ͱͯ͠ͷΞϥʔτ • ٸ͕ͳ͍͕ԿΒ͔ͷΞΫγϣϯΛٻΊΔ • ྫ: Խϗετ͕1μϯ • γεςϜ͕ଓߦՄೳͳൣғͷʮҟৗʯ •
ಉ͡ࢦඪ͕͞ΒʹѱԽ͢Εଈ࣌ରԠ
• ࣮ࡍʹγεςϜ͕Քಇ͍ͯ͠Δ͔ Ͳ͏͔Λࢹ͢Δ • e.g. ֎ܗࢹ, ΤϥʔϨʔτ • ˱ CPU
100%Ͱಈ͍ͯΕྑ͍ ଈ࣌ରԠͷΞϥʔτ
• ͱ͍͑߹ʹΑΓ͚Γ • ͷՔಇͷࢹͰͳͯ͘ɺ ͔ͦ͜ΒՔಇঢ়گ͕ԋ៷ՄೳͳΒ༗༻
Information ͱͯ͠ͷΞϥʔτ • γεςϜՔಇҎ֎ͷࢦඪऔΓಘΔ • ଈ࣌ରԠͱಉ͡௨ܦ࿏ʹࡌͤͳ͍
Ξϥʔτશൠ • ͳΜͰ͔ΜͰΞϥʔτʹ͠ͳ͍ • ຊʹඞཁͳΞϥʔτ͕ຒΕΔ • Information ͩΒͩΒ૿͕͑ͪ
ՄࢹԽ
• ෦ͷঢ়ଶΛ࣌ܥྻ͖ͭͰه͢Δ • खಈͰऩू͠ͳͯ͘؍ଌͰ͖Δ • ݱࡏͷঢ়ଶΛѲ͢Δखஈ • ఆৗ࣌ɾো࣌Θͣ • աڈͷঢ়ଶΛৼΓฦΔखஈ
άϥϑɾμογϡϘʔυ
؍ଌΛҭͯΔ
ඞཁͳ؍ଌྔ͕࠷ॳ͔Βἧ͏ͱݶΒͳ͍
• ։ൃɾݕূதʹؾͮ͘ • ࢹ༻ҙ͠Α͏ͱͯ͠ෆʹؾͮ͘ • োى͖ͯΈͯෆʹؾͮ͘ ͨͱ͑…
ඞཁͳ؍ଌ࠷ॳ͔Βἧ͏ͱݶΒͳ͍
• ։ൃɾݕূதʹؾͮ͘ • ෛՙࢼݧͰؾͮ͘ͱඒ͍͠ʂ • োى͖ͯΈͯෆʹؾͮ͘ ͨͱ͑…
αΠΫϧΛճ͍ͯ͘͠
• ࠷ॳ͔ΒඋͰ͖Εྑ͍ • ಛʹඞਢͷ߲͍͑ͯ͘ • γεςϜͷՔಇͷࢹʂʂ • ܧଓతʹมߋɾվળ͍ͯ͘͠
None
Mackerel Ͱ Mackerel Λ؍ଌ͢Δ
Mackerel ͷٕज़ελοΫ • AWS ্ʹߏங • EC2 ӡ༻ͱ ECS Fargate
͕ࠞࡏ • ϝΠϯͷ JVM ͳͲ͕ EC2 • গͣͭ͠ίϯςφϫʔΫϩʔυʹҠߦத
ઢ
ίϯςφศར
• Πϯελϯεͷੈ͕ཁΒͳ͍ • Πϯϑϥ·ͰؔΘΔߏมߋ͕खܰ • “chef ߋ৽ͯ͠Πϯελϯε࡞Γͯ͠ೖΕ ͑ͯ…” • ʮخ͘͠ͳ͍͜ͱʯ͡ΌΜ……
ઢ͜͜·Ͱ
Mackerel ͷٕज़ελοΫ ࣌ܥྻσʔλϕʔε: ϚωʔδυαʔϏεϕʔε
ઢ
αʔόϨεΞʔΩςΫνϟศར
• ૄ݁߹ɾϦτϥΠՄೳͳઃܭ • ϝοηʔδΩϡʔΛڬΉ • Lambda ͷॲཧ༰Λႈʹઃܭ • ύϥϝʔλมߋͰεέʔϧ͢Δ •
ʢ FaaS ಛ༗ͷӡ༻՝θϩͰͳ͍ʣ
ઢ͜͜·Ͱ
Mackerel ͷٕज़ελοΫ ʢڵຯ͕͋ΕաڈͷొஃࢿྉΛ͝ཡԼ͍͞ʂʣ
JVM ΞϓϦέʔγϣϯͷ؍ଌ • ϝτϦΫεɾࢹ: mackerel • jstat ͱ jolokia ʹΑΔJVM෦ߏͷ؍ଌ
• αʔόʔ͝ͱͷϨΠςϯγࢹ • ίωΫγϣϯϓʔϧࢹ • etc…
JVM ΞϓϦέʔγϣϯͷ؍ଌ • APM: NewRelic APM • ΑΓਫ਼៛ͳ෦ੳ͕ඞཁͳͱ͖ʹ • ϩάղੳ:
fluentd => Kibana • ΞΫηεϩάɺΞϓϦέʔγϣϯϩά
։ൃڥͷ؍ଌ • ։ൃڥʢ݉εςʔδϯάڥʣ • ຊ൪ͱಉ༷ͷߏʢنখ͍͞ʣ • ಉ༷ͷ؍ଌπʔϧߏ • => ϦϦʔεલ͔Β։ൃڥͰ؍ଌ
ఆ • Q: ։ൃڥͷࢹͷίετʁ • => զʑMackerelΛແྉͰ͑Δʢଇʣ
• த֩ʹ Mackerel • ීஈͷʮࢹʯʮཧʯͷى͜͜ • ଞͷπʔϧͱΈ߹Θͤͯ؍ଌΛิڧ
None
͝ਗ਼ௌ͋Γ͕ͱ͏͍͟͝·ͨ͠