Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
Observability: Mackerel による観測と Mackerel の観測 / ...
Search
astj
PRO
September 17, 2019
Technology
2
2k
Observability: Mackerel による観測と Mackerel の観測 / NoOps Meetup Tokyo #8
https://noops.connpass.com/event/145092/
astj
PRO
September 17, 2019
Tweet
Share
More Decks by astj
See All by astj
ISUCON14公式反省会LT: 社内ISUCONの話
astj
PRO
0
150
Mackerel の時系列データベースにおける Redis Cluster の利用と Amazon ElastiCache への移行について / AWS Purpose-Built Databases Week
astj
PRO
3
5.6k
Mackerel のコンテナ監視の近況 / Mackerel Meetup #13
astj
PRO
1
2.2k
サービス開発と健全なプロダクトメンテナンスを開発チームで持続的に取り組む / Developers Boost KANSAI
astj
PRO
1
1.3k
Mackerel and Stripe / Qiita x Stripe Meetup
astj
PRO
0
5.3k
Mackerel をオンプレミスから AWS に移してからの1年半を振り返る / Hatena Engineer Seminar #11
astj
PRO
0
1.1k
AWS で実現した Mackerel 時系列データ1分粒度長期保存の裏側 / Mackerel Meetup #11 Tokyo
astj
PRO
6
29k
稼働中の Web サービスの perl のバージョンを上げていく
astj
PRO
0
7.3k
Perl 6 で Web Application Framework をつくる
astj
PRO
0
4.4k
Other Decks in Technology
See All in Technology
private spaceについてあれこれ調べてみた
operando
1
200
NOSTR, réseau social et espace de liberté décentralisé
rlifchitz
0
140
Ask! NIKKEIの運用基盤と改善に向けた取り組み / NIKKEI TECH TALK #30
kaitomajima
0
130
FastConnect の冗長性
ocise
1
9.3k
RevOpsへ至る道 データ活用による事業革新への挑戦 / path-to-revops
pei0804
3
840
これからSREになる人と、これからもSREをやっていく人へ
masayoshi
1
410
Active Directory の保護
eurekaberry
6
2.8k
20250130_『SUUMO』の裏側!第2弾 ~機械学習エンジニアリング編
recruitengineers
PRO
0
360
[TechNight #86] Oracle GoldenGate - 23ai 最新情報&プロジェクトからの学び
oracle4engineer
PRO
1
200
extensionとschema
yahonda
1
130
Bounded Context: Problem or Solution?
ewolff
1
170
Kubernetesでメールの大量配信をしている話/k8sjp-20250205
hfukamachi
0
180
Featured
See All Featured
Faster Mobile Websites
deanohume
306
31k
Speed Design
sergeychernyshev
25
760
Agile that works and the tools we love
rasmusluckow
328
21k
CSS Pre-Processors: Stylus, Less & Sass
bermonpainter
356
29k
The Illustrated Children's Guide to Kubernetes
chrisshort
48
49k
Creating an realtime collaboration tool: Agile Flush - .NET Oxford
marcduiker
27
1.9k
I Don’t Have Time: Getting Over the Fear to Launch Your Podcast
jcasabona
30
2.1k
Designing for Performance
lara
604
68k
"I'm Feeling Lucky" - Building Great Search Experiences for Today's Users (#IAC19)
danielanewman
226
22k
The Cult of Friendly URLs
andyhume
78
6.2k
Thoughts on Productivity
jonyablonski
69
4.4k
Producing Creativity
orderedlist
PRO
343
39k
Transcript
Observability: Mackerel ʹΑΔ؍ଌ ͱ Mackerel ͷ؍ଌ 2019/09/17 NoOps Meetup Tokyo
#8 Asato WAKISAKA (Hatena)
Asato WAKISAKA GitHub: @astj / Twitter @ast_j ͯͳ ΞϓϦέʔγϣϯΤϯδχΞ Mackerel
։ൃνʔϜ (2016/08~) Mackerel ςοΫϦʔυ (2018/05~)
कඋൣғ: ϑϩϯτΤϯυ͔ΒSREing·Ͱʢࣗশʣ Scala, Golang, TypeScript, …
Mackerel
None
• Mackerel ͬͯΔਓ • ͋Γ͕ͱ͏͍͟͝·͢ʂ • Mackerel ৮ͬͨ͜ͱ͕͋Δਓ • ͋Γ͕ͱ͏͍͟͝·͢ʂʂ
• Mackerel ͬͯΔਓ • ͋Γ͕ͱ͏͍͟͝·͢ʂʂʂʂ Ξϯέʔτ
None
ࢹ
Monitoring
Observability
?
Observability ͱ Monitoring
• Observability: ʮՄ؍ଌੑʯ • γεςϜͷ෦ঢ়ଶ͕औಘͰ͖Δঢ়ଶ • 3ͭͷओཁͳߏཁૉ • Metrics, Logging,
Tracing • Mackerel ͜ͷ͏ͪ Metrics த৺
• ʮՄ؍ଌੑʯ≠ʮ؍ଌʯ • ʮՄ؍ଌੑʯγεςϜ͕࣋ͭੑ࣭ • ʮ؍ଌʯγεςϜ֎෦ͷߦಈ
Metrics Observability Log Observability Trace Observability Metrics Observation Log Observation
Trace Observation
Metrics Observability Log Observability Trace Observability Metrics Observation Log Observation
Trace Observation • Server Metrics • Middleware Metrics • Application Metrics • Business KPI • Etc… • Alerting • Dashboard Visualization • Analysis • Etc…
Metrics Observability Log Observability Trace Observability Metrics Observation Log Observation
Trace Observation ʮϝτϦοΫࢹʯ͜ͷลʁ • Server Metrics • Middleware Metrics • Application Metrics • Business KPI • Etc… • Alerting • Dashboard Visualization • Analysis • Etc…
• ͱ͍͑ɺͦͷઌͰਓ͕ؒ؍ଌ͍ͯ͠Δͣ • ݹయతʹΞϥʔτରԠͳͲ • ʮάϥϑΛݟΔʯ؍ଌ
Metrics Observability Log Observability Trace Observability Metrics Observation Log Observation
Trace Observation wਓؒ • ͋Ε • ͜Ε ˞͜Εۃ
• લஈʹʮՄ؍ଌੑʯͦͷͷ • ޙஈͷʮ؍ଌʯΛ;ͨͭʹ͚ͯଊ͑Δ • ʮ؍ଌͷ࣮ʯ • ʮʢਓؒͷʣ؍ଌʯ
Observability Observability Observability Observation Observation Observation Observe impl. Observe impl.
Observe Impl.
Observability Observability Observability Observation Observation Observation Observe impl. Observe impl.
Observe Impl. ٕज़ͱͯ͠ͷ Observability ͕ ͜ͷลΓΛࢦ͢͜ͱ͋Ε
Observability Observability Observability Observation Observation Observation Observe impl. Observe impl.
Observe Impl. ιϦϡʔγϣϯͷͰ Observability ͕ݴٴ͞ΕΔ࣌ ͜͜·Ͱؚ·ΕΔ͜ͱଟ͍
……Ͱɺ͍͍Ͱ͔͢Ͷʁ
• ٞͷ༨͍Ζ͍Ζ͋Γͦ͏ • ͔͠͠ࠓ͜ͷͷਂ͍͠·ͤΜ • ߹্ࠨΛՄ؍ଌੑͱͯ͠͠·͢
Observability Observability Observability Observation Observation Observation Observe impl. Observe impl.
Observe Impl. • ϝτϦοΫࢹ: • ϝτϦοΫΛऩू͢Δ • ऩूͨ͠ϝτϦοΫΛूੵ͢Δ • άϥϑμογϡϘʔυͰՄࢹԽ͢Δ • ΞϥʔτΛൃใ͢Δ
Observability Observability Observability Observation Observation Observation Observe impl. Observe impl.
Observe Impl. ϝτϦοΫऩू ϝτϦοΫूੵ • άϥϑ • Ξϥʔτ • μογϡϘʔυ ˞Πϝʔδ • ਓؒ • ࣗಈԽରԠ
Observability Observability Observability Observation Observation Observation Observe impl. Observe impl.
Observe Impl. • ؍ଌ͢ΔͨΊʹ؍ଌՄೳʹ͢Δඞཁ͕͋Δ • Մ؍ଌੑࢹͷલఏͱͳΔ • ʢͷͰɺࢹͷจ຺ͰՄ؍ଌੑʹݴٴ͞ΕΔʣ
None
Mackerel ͷ֓೦
ϗετ
• ࢹɾཧͷىʹͳΔ֓೦ • యܕతʹαʔόʔ • ʮαʔόʔͷΑ͏ͳͷʯϗετ • ίϯςφͷू߹ (e.g. ECS
Task / k8s pod) • ΫϥυϦιʔε (e.g. Lambda Function) ϗετ
Service:Blog Role:App Role:DB Host A Host B Host C Host
D Service:Bookmark Role:App Host E Host F … αʔϏεɾϩʔϧ
• ϗετΛʮׂʯͰྨ • μογϡϘʔυ্Ͱάϧʔϐϯά͞ΕΔ • αʔϏεϩʔϧ୯ҐͰࢹϧʔϧΛཧ αʔϏεɾϩʔϧ
None
None
None
෦ঢ়ଶͷऩू
• ࣌ܥྻͰࢀরՄೳ ϝτϦοΫ
• ୯७ͳ൱ͱϝοηʔδΛه • ฏৗ࣌ͷཤྺهʹΒͳ͍ νΣοΫࢹ
• ϩάʹର͢Δ(ϝτϦοΫ|νΣοΫ)ࢹͰදݱ • ϩάΛࠪͯ͠ύλʔϯϚον͢Δ • LTSV ܗࣜͷΞΫηεϩάΛूܭ͢Δ ϩάΛࢹ͢Δ
None
ऩूͷλʔήοτ • ΤʔδΣϯτܥ • αʔόʔ • ίϯςφ • αʔϏεϝτϦοΫ •
URL ֎ܗࢹ • ΠϯςάϨʔγϣϯ
• αʔόʔ্Ͱಈ͘ΤʔδΣϯτ • ΤʔδΣϯτ͔Β Mackerel ʹ POST ͢Δ • γεςϜϝτϦοΫࣗಈͰऩू
• CPU, memory, interface… • ϓϥάΠϯͰऔಘใΛ֦ுͰ͖Δ αʔόʔ͔Βͷऩू
Server Mackerel mackerel-agent Fluentd Golang app OS stats monitor-agent Plugins
HTTP gostats POST ྫ
• ECS 1 Task k8s 1 pod Λ1ϗετ •
αΠυΧʔίϯςφͱͯ͠ΤʔδΣϯτΛಋೖ • γεςϜϝτϦοΫΛࣗಈͰऩू • ϓϥάΠϯͰ֦ுՄೳ ίϯςφ͔Βͷऩू
Task Task Task Mackerel Agent Container Fluentd container Golang app
container monitor-agent Plugins HTTP gostats POST Orchestration Metadata ྫ
• ಛఆͷϗετʹඥ͔ͳ͍ϝτϦοΫ • Ϗδωε KPI ɺαʔϏεશମͷϨΠςϯγ • API Ͱࣗ༝ʹߘ •
e.g. fluentd αʔϏεϝτϦοΫ Service:Bookmark Role:App Host
• Mackerel ଆ͔Β URL ʹଓ • ଓՄ൱ϨεϙϯελΠϜͳͲΛऩू URL ֎ܗࢹ
• ΫϥυࣄۀऀͷϞχλϦϯά API ͔Βऔಘ • ΤʔδΣϯτͳ͠ͰऩूͰ͖Δ • ऩूϝτϦοΫΫϥυࣄۀऀґଘ AWS /
Azure ΠϯςάϨʔγϣϯ Mackerel User Cloud Services Crawl
ऩू͢Δ༰
• γεςϜϝτϦοΫࣗಈͰऩू • ϛυϧΣΞެࣜϓϥάΠϯͰऩू • ΞϓϦέʔγϣϯͷϝτϦοΫʁʁ
• γεςϜϝτϦοΫࣗಈͰऩू • ϛυϧΣΞެࣜϓϥάΠϯͰऩू • ΞϓϦέʔγϣϯͷϝτϦοΫʁʁ
• ݴޠϑϨʔϜϫʔΫ͚ͷϓϥάΠϯ͋Δ • JVM Go GC ౷ܭ؍ଌͰ͖Δ •
Γͳ͍ϝτϦοΫͷ؍ଌखஈ͕ඞཁ
Observability Observability Observability Observation Observation Observation Observe impl. Observe impl.
Observe Impl. • ؍ଌ͢ΔͨΊʹ؍ଌՄೳʹ͢Δඞཁ͕͋Δ • Մ؍ଌੑࢹͷલఏͱͳΔ • ʢͷͰɺࢹͷจ຺ͰՄ؍ଌੑʹݴٴ͢Δʣ ࠶ܝ • ؍ଌ͢ΔͨΊʹ؍ଌՄೳʹ͢Δඞཁ͕͋Δ
؍ଌ͍ͨ͠ϝτϦοΫԿΒ͔ͷखஈͰ؍ଌՄೳͰ͔͢ʁ ؍ଌՄೳʹ͢Δ ΤʔδΣϯτʢͷϓϥάΠϯʣͰ؍ଌɾऩू͢Δ NO YES
؍ଌ͍ͨ͠ϝτϦοΫԿΒ͔ͷखஈͰ؍ଌՄೳͰ͔͢ʁ ؍ଌՄೳʹ͢Δ ΤʔδΣϯτʢͷϓϥάΠϯʣͰ؍ଌɾऩू͢Δ NO YES
• ͲͷΑ͏ʹ؍ଌ͞ΕΔ͔Ͱ࣮͕ܾ·Δ ؍ଌՄೳʹ͢Δ
؍ଌ͍ͨ͠ϝτϦοΫԿΒ͔ͷखஈͰ؍ଌՄೳͰ͔͢ʁ ؍ଌՄೳʹ͢Δ ΤʔδΣϯτʢͷϓϥάΠϯʣͰ؍ଌɾऩू͢Δ NO YES
• ϓϥάΠϯΛࣗ࡞ͯ͠ҙͷखஈͰ؍ଌ • HTTP Ͱ JSON ܗࣜΛಡΈࠐΜͰ؍ଌ ؍ଌखஈ
• ϓϥάΠϯ = ࣮ߦϑΝΠϧ • ॴఆͷग़ྗͰϝτϦοΫ໊ͱΛग़ྗ͢Δ͚ͩ • ࣮ߦڥͰ“ݟ͑Δ”ͳΒԿͰऔΕΔ ࣗ࡞ϓϥάΠϯͰͷϝτϦοΫऩू $
mackerel-plugin-uptime uptime.seconds 7749392 1568662368
• ࣮ࣗମ Bash Ͱ Go ͰԿͰ • ग़ྗϑΥʔϚοτ͕ૉͳͷͰ࣮Λબͳ͍ • ؍ଌରʹ͋Θ࣮ͤͯ͢Δඞཁ͕͋Δ
• ؍ଌରଆͷࣗ༝ߴ͘ͳΔ ࣗ࡞ϓϥάΠϯͰͷϝτϦοΫऩू
HTTP ܦ༝Ͱͷऩू • mackerel-plugin-json • ؍ଌ͍ͨ͠Λ JSON Ͱ HTTP ৴͢Δ
• ϓϥάΠϯ͔ΒͦͷΤϯυϙΠϯτΛಡΈࠐΉ
ఆ • Q: Prometheus Exporter Έ͍ͨͳʁ • => ํੑ yesʢͬͱγϯϓϧʣ
• Q: Prometheus Exporter ͑ͳ͍ͷʁ • => ……!
HTTP ܦ༝Ͱͷऩू $ mackerel-plugin-json \ > -url=http://localhost/stats active3 1568666460 waiting
2 1568666460 $ curl -fsSL http://localhost/stats {“active”:3,”waiting”:2}
HTTP ܦ༝Ͱͷऩू
• ͲͪΒ͕ྑ͍͔߹ʹΑΔ • طʹHTTPΛΔͳΒHTTP͕ศར • طଘͷ؍ଌखஈ͕͋Ε࠶ར༻͍ͨ͠
ऩूͨ͠ϝτϦοΫͷར༻
• Mackerel Ͱͷओͳ͍ಓ • ࢹʢΞϥʔτʣ • ՄࢹԽʢάϥϑɾμογϡϘʔυʣ
ࢹʢΞϥʔτʣ
;ͨͭͷΞϥʔτ • ଈ࣌ରԠΛଅ͢Ξϥʔτ • ӡ༻ऀͱͯ͠θϩʹ͍ͨ͠ʂ • Information ͱͯ͠ͷΞϥʔτ
ଈ࣌ରԠͷΞϥʔτ (on-call) • γεςϜͷՔಇ͕ଛͳΘΕ͍ͯΔ • ରԠΛඞཁͱ͢Δ • “Δ·Ͱ์ஔ”͕ڐ༰ͳΒଈ࣌ରԠͰͳ͍
Information ͱͯ͠ͷΞϥʔτ • ٸ͕ͳ͍͕ԿΒ͔ͷΞΫγϣϯΛٻΊΔ • ྫ: Խϗετ͕1μϯ • γεςϜ͕ଓߦՄೳͳൣғͷʮҟৗʯ •
ಉ͡ࢦඪ͕͞ΒʹѱԽ͢Εଈ࣌ରԠ
• ࣮ࡍʹγεςϜ͕Քಇ͍ͯ͠Δ͔ Ͳ͏͔Λࢹ͢Δ • e.g. ֎ܗࢹ, ΤϥʔϨʔτ • ˱ CPU
100%Ͱಈ͍ͯΕྑ͍ ଈ࣌ରԠͷΞϥʔτ
• ͱ͍͑߹ʹΑΓ͚Γ • ͷՔಇͷࢹͰͳͯ͘ɺ ͔ͦ͜ΒՔಇঢ়گ͕ԋ៷ՄೳͳΒ༗༻
Information ͱͯ͠ͷΞϥʔτ • γεςϜՔಇҎ֎ͷࢦඪऔΓಘΔ • ଈ࣌ରԠͱಉ͡௨ܦ࿏ʹࡌͤͳ͍
Ξϥʔτશൠ • ͳΜͰ͔ΜͰΞϥʔτʹ͠ͳ͍ • ຊʹඞཁͳΞϥʔτ͕ຒΕΔ • Information ͩΒͩΒ૿͕͑ͪ
ՄࢹԽ
• ෦ͷঢ়ଶΛ࣌ܥྻ͖ͭͰه͢Δ • खಈͰऩू͠ͳͯ͘؍ଌͰ͖Δ • ݱࡏͷঢ়ଶΛѲ͢Δखஈ • ఆৗ࣌ɾো࣌Θͣ • աڈͷঢ়ଶΛৼΓฦΔखஈ
άϥϑɾμογϡϘʔυ
؍ଌΛҭͯΔ
ඞཁͳ؍ଌྔ͕࠷ॳ͔Βἧ͏ͱݶΒͳ͍
• ։ൃɾݕূதʹؾͮ͘ • ࢹ༻ҙ͠Α͏ͱͯ͠ෆʹؾͮ͘ • োى͖ͯΈͯෆʹؾͮ͘ ͨͱ͑…
ඞཁͳ؍ଌ࠷ॳ͔Βἧ͏ͱݶΒͳ͍
• ։ൃɾݕূதʹؾͮ͘ • ෛՙࢼݧͰؾͮ͘ͱඒ͍͠ʂ • োى͖ͯΈͯෆʹؾͮ͘ ͨͱ͑…
αΠΫϧΛճ͍ͯ͘͠
• ࠷ॳ͔ΒඋͰ͖Εྑ͍ • ಛʹඞਢͷ߲͍͑ͯ͘ • γεςϜͷՔಇͷࢹʂʂ • ܧଓతʹมߋɾվળ͍ͯ͘͠
None
Mackerel Ͱ Mackerel Λ؍ଌ͢Δ
Mackerel ͷٕज़ελοΫ • AWS ্ʹߏங • EC2 ӡ༻ͱ ECS Fargate
͕ࠞࡏ • ϝΠϯͷ JVM ͳͲ͕ EC2 • গͣͭ͠ίϯςφϫʔΫϩʔυʹҠߦத
ઢ
ίϯςφศར
• Πϯελϯεͷੈ͕ཁΒͳ͍ • Πϯϑϥ·ͰؔΘΔߏมߋ͕खܰ • “chef ߋ৽ͯ͠Πϯελϯε࡞Γͯ͠ೖΕ ͑ͯ…” • ʮخ͘͠ͳ͍͜ͱʯ͡ΌΜ……
ઢ͜͜·Ͱ
Mackerel ͷٕज़ελοΫ ࣌ܥྻσʔλϕʔε: ϚωʔδυαʔϏεϕʔε
ઢ
αʔόϨεΞʔΩςΫνϟศར
• ૄ݁߹ɾϦτϥΠՄೳͳઃܭ • ϝοηʔδΩϡʔΛڬΉ • Lambda ͷॲཧ༰Λႈʹઃܭ • ύϥϝʔλมߋͰεέʔϧ͢Δ •
ʢ FaaS ಛ༗ͷӡ༻՝θϩͰͳ͍ʣ
ઢ͜͜·Ͱ
Mackerel ͷٕज़ελοΫ ʢڵຯ͕͋ΕաڈͷొஃࢿྉΛ͝ཡԼ͍͞ʂʣ
JVM ΞϓϦέʔγϣϯͷ؍ଌ • ϝτϦΫεɾࢹ: mackerel • jstat ͱ jolokia ʹΑΔJVM෦ߏͷ؍ଌ
• αʔόʔ͝ͱͷϨΠςϯγࢹ • ίωΫγϣϯϓʔϧࢹ • etc…
JVM ΞϓϦέʔγϣϯͷ؍ଌ • APM: NewRelic APM • ΑΓਫ਼៛ͳ෦ੳ͕ඞཁͳͱ͖ʹ • ϩάղੳ:
fluentd => Kibana • ΞΫηεϩάɺΞϓϦέʔγϣϯϩά
։ൃڥͷ؍ଌ • ։ൃڥʢ݉εςʔδϯάڥʣ • ຊ൪ͱಉ༷ͷߏʢنখ͍͞ʣ • ಉ༷ͷ؍ଌπʔϧߏ • => ϦϦʔεલ͔Β։ൃڥͰ؍ଌ
ఆ • Q: ։ൃڥͷࢹͷίετʁ • => զʑMackerelΛແྉͰ͑Δʢଇʣ
• த֩ʹ Mackerel • ීஈͷʮࢹʯʮཧʯͷى͜͜ • ଞͷπʔϧͱΈ߹Θͤͯ؍ଌΛิڧ
None
͝ਗ਼ௌ͋Γ͕ͱ͏͍͟͝·ͨ͠