Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Speaker Deck
PRO
Sign in
Sign up for free
Observability: Mackerel による観測と Mackerel の観測 / NoOps Meetup Tokyo #8
astj
PRO
September 17, 2019
Technology
2
1.3k
Observability: Mackerel による観測と Mackerel の観測 / NoOps Meetup Tokyo #8
https://noops.connpass.com/event/145092/
astj
PRO
September 17, 2019
Tweet
Share
More Decks by astj
See All by astj
Mackerel の時系列データベースにおける Redis Cluster の利用と Amazon ElastiCache への移行について / AWS Purpose-Built Databases Week
astj
PRO
3
4.3k
Mackerel のコンテナ監視の近況 / Mackerel Meetup #13
astj
PRO
1
1.2k
サービス開発と健全なプロダクトメンテナンスを開発チームで持続的に取り組む / Developers Boost KANSAI
astj
PRO
1
850
Mackerel and Stripe / Qiita x Stripe Meetup
astj
PRO
0
4.5k
Mackerel をオンプレミスから AWS に移してからの1年半を振り返る / Hatena Engineer Seminar #11
astj
PRO
0
730
AWS で実現した Mackerel 時系列データ1分粒度長期保存の裏側 / Mackerel Meetup #11 Tokyo
astj
PRO
6
26k
稼働中の Web サービスの perl のバージョンを上げていく
astj
PRO
0
6.6k
Perl 6 で Web Application Framework をつくる
astj
PRO
0
3.7k
Other Decks in Technology
See All in Technology
Micro frontends and micro services
kashif98
0
120
Istioを活用したセキュアなマイクロサービスの実現/Secure Microservices with Istio
ido_kara_deru
3
390
AutoMLを利用した機械学習モデル構築時に意識すること
sbtechnight
0
160
最先端の生成AIから考える、ビジネスにおける10年後のパラダイムシフト
sbtechnight
1
320
ソフトバンクaPaaS領域への挑戦
sbtechnight
0
310
DevelopersIO 2022 俺のTerraform Pipeline
takakuni
0
430
第22回 MLOps 勉強会:みてねのMLOps事情
tonouchi510
0
290
A3-1 IBM Championが本音で語る「IBM Cloud」
kolinz
0
300
hey BOOK
heyinc
26
290k
ECS on EC2 で Auto Scaling やってみる!
sayjoy
1
140
ソフトバンクのシナジーがもたらすクラウドソリューションについて、クラウドエンジニアが話してみた。
sbtechnight
0
300
経験者が話す!クラウド接続の3つの注意点と最新情報
sbtechnight
0
310
Featured
See All Featured
Embracing the Ebb and Flow
colly
73
3.4k
How to train your dragon (web standard)
notwaldorf
60
3.9k
It's Worth the Effort
3n
172
26k
Put a Button on it: Removing Barriers to Going Fast.
kastner
56
2.3k
Web Components: a chance to create the future
zenorocha
303
40k
Side Projects
sachag
450
37k
Fashionably flexible responsive web design (full day workshop)
malarkey
396
62k
Build your cross-platform service in a week with App Engine
jlugia
219
17k
ピンチをチャンスに:未来をつくるプロダクトロードマップ #pmconf2020
aki_i
25
15k
ReactJS: Keep Simple. Everything can be a component!
pedronauck
655
120k
Designing for humans not robots
tammielis
241
24k
Designing on Purpose - Digital PM Summit 2013
jponch
106
5.7k
Transcript
Observability: Mackerel ʹΑΔ؍ଌ ͱ Mackerel ͷ؍ଌ 2019/09/17 NoOps Meetup Tokyo
#8 Asato WAKISAKA (Hatena)
Asato WAKISAKA GitHub: @astj / Twitter @ast_j ͯͳ ΞϓϦέʔγϣϯΤϯδχΞ Mackerel
։ൃνʔϜ (2016/08~) Mackerel ςοΫϦʔυ (2018/05~)
कඋൣғ: ϑϩϯτΤϯυ͔ΒSREing·Ͱʢࣗশʣ Scala, Golang, TypeScript, …
Mackerel
None
• Mackerel ͬͯΔਓ • ͋Γ͕ͱ͏͍͟͝·͢ʂ • Mackerel ৮ͬͨ͜ͱ͕͋Δਓ • ͋Γ͕ͱ͏͍͟͝·͢ʂʂ
• Mackerel ͬͯΔਓ • ͋Γ͕ͱ͏͍͟͝·͢ʂʂʂʂ Ξϯέʔτ
None
ࢹ
Monitoring
Observability
?
Observability ͱ Monitoring
• Observability: ʮՄ؍ଌੑʯ • γεςϜͷ෦ঢ়ଶ͕औಘͰ͖Δঢ়ଶ • 3ͭͷओཁͳߏཁૉ • Metrics, Logging,
Tracing • Mackerel ͜ͷ͏ͪ Metrics த৺
• ʮՄ؍ଌੑʯ≠ʮ؍ଌʯ • ʮՄ؍ଌੑʯγεςϜ͕࣋ͭੑ࣭ • ʮ؍ଌʯγεςϜ֎෦ͷߦಈ
Metrics Observability Log Observability Trace Observability Metrics Observation Log Observation
Trace Observation
Metrics Observability Log Observability Trace Observability Metrics Observation Log Observation
Trace Observation • Server Metrics • Middleware Metrics • Application Metrics • Business KPI • Etc… • Alerting • Dashboard Visualization • Analysis • Etc…
Metrics Observability Log Observability Trace Observability Metrics Observation Log Observation
Trace Observation ʮϝτϦοΫࢹʯ͜ͷลʁ • Server Metrics • Middleware Metrics • Application Metrics • Business KPI • Etc… • Alerting • Dashboard Visualization • Analysis • Etc…
• ͱ͍͑ɺͦͷઌͰਓ͕ؒ؍ଌ͍ͯ͠Δͣ • ݹయతʹΞϥʔτରԠͳͲ • ʮάϥϑΛݟΔʯ؍ଌ
Metrics Observability Log Observability Trace Observability Metrics Observation Log Observation
Trace Observation wਓؒ • ͋Ε • ͜Ε ˞͜Εۃ
• લஈʹʮՄ؍ଌੑʯͦͷͷ • ޙஈͷʮ؍ଌʯΛ;ͨͭʹ͚ͯଊ͑Δ • ʮ؍ଌͷ࣮ʯ • ʮʢਓؒͷʣ؍ଌʯ
Observability Observability Observability Observation Observation Observation Observe impl. Observe impl.
Observe Impl.
Observability Observability Observability Observation Observation Observation Observe impl. Observe impl.
Observe Impl. ٕज़ͱͯ͠ͷ Observability ͕ ͜ͷลΓΛࢦ͢͜ͱ͋Ε
Observability Observability Observability Observation Observation Observation Observe impl. Observe impl.
Observe Impl. ιϦϡʔγϣϯͷͰ Observability ͕ݴٴ͞ΕΔ࣌ ͜͜·Ͱؚ·ΕΔ͜ͱଟ͍
……Ͱɺ͍͍Ͱ͔͢Ͷʁ
• ٞͷ༨͍Ζ͍Ζ͋Γͦ͏ • ͔͠͠ࠓ͜ͷͷਂ͍͠·ͤΜ • ߹্ࠨΛՄ؍ଌੑͱͯ͠͠·͢
Observability Observability Observability Observation Observation Observation Observe impl. Observe impl.
Observe Impl. • ϝτϦοΫࢹ: • ϝτϦοΫΛऩू͢Δ • ऩूͨ͠ϝτϦοΫΛूੵ͢Δ • άϥϑμογϡϘʔυͰՄࢹԽ͢Δ • ΞϥʔτΛൃใ͢Δ
Observability Observability Observability Observation Observation Observation Observe impl. Observe impl.
Observe Impl. ϝτϦοΫऩू ϝτϦοΫूੵ • άϥϑ • Ξϥʔτ • μογϡϘʔυ ˞Πϝʔδ • ਓؒ • ࣗಈԽରԠ
Observability Observability Observability Observation Observation Observation Observe impl. Observe impl.
Observe Impl. • ؍ଌ͢ΔͨΊʹ؍ଌՄೳʹ͢Δඞཁ͕͋Δ • Մ؍ଌੑࢹͷલఏͱͳΔ • ʢͷͰɺࢹͷจ຺ͰՄ؍ଌੑʹݴٴ͞ΕΔʣ
None
Mackerel ͷ֓೦
ϗετ
• ࢹɾཧͷىʹͳΔ֓೦ • యܕతʹαʔόʔ • ʮαʔόʔͷΑ͏ͳͷʯϗετ • ίϯςφͷू߹ (e.g. ECS
Task / k8s pod) • ΫϥυϦιʔε (e.g. Lambda Function) ϗετ
Service:Blog Role:App Role:DB Host A Host B Host C Host
D Service:Bookmark Role:App Host E Host F … αʔϏεɾϩʔϧ
• ϗετΛʮׂʯͰྨ • μογϡϘʔυ্Ͱάϧʔϐϯά͞ΕΔ • αʔϏεϩʔϧ୯ҐͰࢹϧʔϧΛཧ αʔϏεɾϩʔϧ
None
None
None
෦ঢ়ଶͷऩू
• ࣌ܥྻͰࢀরՄೳ ϝτϦοΫ
• ୯७ͳ൱ͱϝοηʔδΛه • ฏৗ࣌ͷཤྺهʹΒͳ͍ νΣοΫࢹ
• ϩάʹର͢Δ(ϝτϦοΫ|νΣοΫ)ࢹͰදݱ • ϩάΛࠪͯ͠ύλʔϯϚον͢Δ • LTSV ܗࣜͷΞΫηεϩάΛूܭ͢Δ ϩάΛࢹ͢Δ
None
ऩूͷλʔήοτ • ΤʔδΣϯτܥ • αʔόʔ • ίϯςφ • αʔϏεϝτϦοΫ •
URL ֎ܗࢹ • ΠϯςάϨʔγϣϯ
• αʔόʔ্Ͱಈ͘ΤʔδΣϯτ • ΤʔδΣϯτ͔Β Mackerel ʹ POST ͢Δ • γεςϜϝτϦοΫࣗಈͰऩू
• CPU, memory, interface… • ϓϥάΠϯͰऔಘใΛ֦ுͰ͖Δ αʔόʔ͔Βͷऩू
Server Mackerel mackerel-agent Fluentd Golang app OS stats monitor-agent Plugins
HTTP gostats POST ྫ
• ECS 1 Task k8s 1 pod Λ1ϗετ •
αΠυΧʔίϯςφͱͯ͠ΤʔδΣϯτΛಋೖ • γεςϜϝτϦοΫΛࣗಈͰऩू • ϓϥάΠϯͰ֦ுՄೳ ίϯςφ͔Βͷऩू
Task Task Task Mackerel Agent Container Fluentd container Golang app
container monitor-agent Plugins HTTP gostats POST Orchestration Metadata ྫ
• ಛఆͷϗετʹඥ͔ͳ͍ϝτϦοΫ • Ϗδωε KPI ɺαʔϏεશମͷϨΠςϯγ • API Ͱࣗ༝ʹߘ •
e.g. fluentd αʔϏεϝτϦοΫ Service:Bookmark Role:App Host
• Mackerel ଆ͔Β URL ʹଓ • ଓՄ൱ϨεϙϯελΠϜͳͲΛऩू URL ֎ܗࢹ
• ΫϥυࣄۀऀͷϞχλϦϯά API ͔Βऔಘ • ΤʔδΣϯτͳ͠ͰऩूͰ͖Δ • ऩूϝτϦοΫΫϥυࣄۀऀґଘ AWS /
Azure ΠϯςάϨʔγϣϯ Mackerel User Cloud Services Crawl
ऩू͢Δ༰
• γεςϜϝτϦοΫࣗಈͰऩू • ϛυϧΣΞެࣜϓϥάΠϯͰऩू • ΞϓϦέʔγϣϯͷϝτϦοΫʁʁ
• γεςϜϝτϦοΫࣗಈͰऩू • ϛυϧΣΞެࣜϓϥάΠϯͰऩू • ΞϓϦέʔγϣϯͷϝτϦοΫʁʁ
• ݴޠϑϨʔϜϫʔΫ͚ͷϓϥάΠϯ͋Δ • JVM Go GC ౷ܭ؍ଌͰ͖Δ •
Γͳ͍ϝτϦοΫͷ؍ଌखஈ͕ඞཁ
Observability Observability Observability Observation Observation Observation Observe impl. Observe impl.
Observe Impl. • ؍ଌ͢ΔͨΊʹ؍ଌՄೳʹ͢Δඞཁ͕͋Δ • Մ؍ଌੑࢹͷલఏͱͳΔ • ʢͷͰɺࢹͷจ຺ͰՄ؍ଌੑʹݴٴ͢Δʣ ࠶ܝ • ؍ଌ͢ΔͨΊʹ؍ଌՄೳʹ͢Δඞཁ͕͋Δ
؍ଌ͍ͨ͠ϝτϦοΫԿΒ͔ͷखஈͰ؍ଌՄೳͰ͔͢ʁ ؍ଌՄೳʹ͢Δ ΤʔδΣϯτʢͷϓϥάΠϯʣͰ؍ଌɾऩू͢Δ NO YES
؍ଌ͍ͨ͠ϝτϦοΫԿΒ͔ͷखஈͰ؍ଌՄೳͰ͔͢ʁ ؍ଌՄೳʹ͢Δ ΤʔδΣϯτʢͷϓϥάΠϯʣͰ؍ଌɾऩू͢Δ NO YES
• ͲͷΑ͏ʹ؍ଌ͞ΕΔ͔Ͱ࣮͕ܾ·Δ ؍ଌՄೳʹ͢Δ
؍ଌ͍ͨ͠ϝτϦοΫԿΒ͔ͷखஈͰ؍ଌՄೳͰ͔͢ʁ ؍ଌՄೳʹ͢Δ ΤʔδΣϯτʢͷϓϥάΠϯʣͰ؍ଌɾऩू͢Δ NO YES
• ϓϥάΠϯΛࣗ࡞ͯ͠ҙͷखஈͰ؍ଌ • HTTP Ͱ JSON ܗࣜΛಡΈࠐΜͰ؍ଌ ؍ଌखஈ
• ϓϥάΠϯ = ࣮ߦϑΝΠϧ • ॴఆͷग़ྗͰϝτϦοΫ໊ͱΛग़ྗ͢Δ͚ͩ • ࣮ߦڥͰ“ݟ͑Δ”ͳΒԿͰऔΕΔ ࣗ࡞ϓϥάΠϯͰͷϝτϦοΫऩू $
mackerel-plugin-uptime uptime.seconds 7749392 1568662368
• ࣮ࣗମ Bash Ͱ Go ͰԿͰ • ग़ྗϑΥʔϚοτ͕ૉͳͷͰ࣮Λબͳ͍ • ؍ଌରʹ͋Θ࣮ͤͯ͢Δඞཁ͕͋Δ
• ؍ଌରଆͷࣗ༝ߴ͘ͳΔ ࣗ࡞ϓϥάΠϯͰͷϝτϦοΫऩू
HTTP ܦ༝Ͱͷऩू • mackerel-plugin-json • ؍ଌ͍ͨ͠Λ JSON Ͱ HTTP ৴͢Δ
• ϓϥάΠϯ͔ΒͦͷΤϯυϙΠϯτΛಡΈࠐΉ
ఆ • Q: Prometheus Exporter Έ͍ͨͳʁ • => ํੑ yesʢͬͱγϯϓϧʣ
• Q: Prometheus Exporter ͑ͳ͍ͷʁ • => ……!
HTTP ܦ༝Ͱͷऩू $ mackerel-plugin-json \ > -url=http://localhost/stats active3 1568666460 waiting
2 1568666460 $ curl -fsSL http://localhost/stats {“active”:3,”waiting”:2}
HTTP ܦ༝Ͱͷऩू
• ͲͪΒ͕ྑ͍͔߹ʹΑΔ • طʹHTTPΛΔͳΒHTTP͕ศར • طଘͷ؍ଌखஈ͕͋Ε࠶ར༻͍ͨ͠
ऩूͨ͠ϝτϦοΫͷར༻
• Mackerel Ͱͷओͳ͍ಓ • ࢹʢΞϥʔτʣ • ՄࢹԽʢάϥϑɾμογϡϘʔυʣ
ࢹʢΞϥʔτʣ
;ͨͭͷΞϥʔτ • ଈ࣌ରԠΛଅ͢Ξϥʔτ • ӡ༻ऀͱͯ͠θϩʹ͍ͨ͠ʂ • Information ͱͯ͠ͷΞϥʔτ
ଈ࣌ରԠͷΞϥʔτ (on-call) • γεςϜͷՔಇ͕ଛͳΘΕ͍ͯΔ • ରԠΛඞཁͱ͢Δ • “Δ·Ͱ์ஔ”͕ڐ༰ͳΒଈ࣌ରԠͰͳ͍
Information ͱͯ͠ͷΞϥʔτ • ٸ͕ͳ͍͕ԿΒ͔ͷΞΫγϣϯΛٻΊΔ • ྫ: Խϗετ͕1μϯ • γεςϜ͕ଓߦՄೳͳൣғͷʮҟৗʯ •
ಉ͡ࢦඪ͕͞ΒʹѱԽ͢Εଈ࣌ରԠ
• ࣮ࡍʹγεςϜ͕Քಇ͍ͯ͠Δ͔ Ͳ͏͔Λࢹ͢Δ • e.g. ֎ܗࢹ, ΤϥʔϨʔτ • ˱ CPU
100%Ͱಈ͍ͯΕྑ͍ ଈ࣌ରԠͷΞϥʔτ
• ͱ͍͑߹ʹΑΓ͚Γ • ͷՔಇͷࢹͰͳͯ͘ɺ ͔ͦ͜ΒՔಇঢ়گ͕ԋ៷ՄೳͳΒ༗༻
Information ͱͯ͠ͷΞϥʔτ • γεςϜՔಇҎ֎ͷࢦඪऔΓಘΔ • ଈ࣌ରԠͱಉ͡௨ܦ࿏ʹࡌͤͳ͍
Ξϥʔτશൠ • ͳΜͰ͔ΜͰΞϥʔτʹ͠ͳ͍ • ຊʹඞཁͳΞϥʔτ͕ຒΕΔ • Information ͩΒͩΒ૿͕͑ͪ
ՄࢹԽ
• ෦ͷঢ়ଶΛ࣌ܥྻ͖ͭͰه͢Δ • खಈͰऩू͠ͳͯ͘؍ଌͰ͖Δ • ݱࡏͷঢ়ଶΛѲ͢Δखஈ • ఆৗ࣌ɾো࣌Θͣ • աڈͷঢ়ଶΛৼΓฦΔखஈ
άϥϑɾμογϡϘʔυ
؍ଌΛҭͯΔ
ඞཁͳ؍ଌྔ͕࠷ॳ͔Βἧ͏ͱݶΒͳ͍
• ։ൃɾݕূதʹؾͮ͘ • ࢹ༻ҙ͠Α͏ͱͯ͠ෆʹؾͮ͘ • োى͖ͯΈͯෆʹؾͮ͘ ͨͱ͑…
ඞཁͳ؍ଌ࠷ॳ͔Βἧ͏ͱݶΒͳ͍
• ։ൃɾݕূதʹؾͮ͘ • ෛՙࢼݧͰؾͮ͘ͱඒ͍͠ʂ • োى͖ͯΈͯෆʹؾͮ͘ ͨͱ͑…
αΠΫϧΛճ͍ͯ͘͠
• ࠷ॳ͔ΒඋͰ͖Εྑ͍ • ಛʹඞਢͷ߲͍͑ͯ͘ • γεςϜͷՔಇͷࢹʂʂ • ܧଓతʹมߋɾվળ͍ͯ͘͠
None
Mackerel Ͱ Mackerel Λ؍ଌ͢Δ
Mackerel ͷٕज़ελοΫ • AWS ্ʹߏங • EC2 ӡ༻ͱ ECS Fargate
͕ࠞࡏ • ϝΠϯͷ JVM ͳͲ͕ EC2 • গͣͭ͠ίϯςφϫʔΫϩʔυʹҠߦத
ઢ
ίϯςφศར
• Πϯελϯεͷੈ͕ཁΒͳ͍ • Πϯϑϥ·ͰؔΘΔߏมߋ͕खܰ • “chef ߋ৽ͯ͠Πϯελϯε࡞Γͯ͠ೖΕ ͑ͯ…” • ʮخ͘͠ͳ͍͜ͱʯ͡ΌΜ……
ઢ͜͜·Ͱ
Mackerel ͷٕज़ελοΫ ࣌ܥྻσʔλϕʔε: ϚωʔδυαʔϏεϕʔε
ઢ
αʔόϨεΞʔΩςΫνϟศར
• ૄ݁߹ɾϦτϥΠՄೳͳઃܭ • ϝοηʔδΩϡʔΛڬΉ • Lambda ͷॲཧ༰Λႈʹઃܭ • ύϥϝʔλมߋͰεέʔϧ͢Δ •
ʢ FaaS ಛ༗ͷӡ༻՝θϩͰͳ͍ʣ
ઢ͜͜·Ͱ
Mackerel ͷٕज़ελοΫ ʢڵຯ͕͋ΕաڈͷొஃࢿྉΛ͝ཡԼ͍͞ʂʣ
JVM ΞϓϦέʔγϣϯͷ؍ଌ • ϝτϦΫεɾࢹ: mackerel • jstat ͱ jolokia ʹΑΔJVM෦ߏͷ؍ଌ
• αʔόʔ͝ͱͷϨΠςϯγࢹ • ίωΫγϣϯϓʔϧࢹ • etc…
JVM ΞϓϦέʔγϣϯͷ؍ଌ • APM: NewRelic APM • ΑΓਫ਼៛ͳ෦ੳ͕ඞཁͳͱ͖ʹ • ϩάղੳ:
fluentd => Kibana • ΞΫηεϩάɺΞϓϦέʔγϣϯϩά
։ൃڥͷ؍ଌ • ։ൃڥʢ݉εςʔδϯάڥʣ • ຊ൪ͱಉ༷ͷߏʢنখ͍͞ʣ • ಉ༷ͷ؍ଌπʔϧߏ • => ϦϦʔεલ͔Β։ൃڥͰ؍ଌ
ఆ • Q: ։ൃڥͷࢹͷίετʁ • => զʑMackerelΛແྉͰ͑Δʢଇʣ
• த֩ʹ Mackerel • ීஈͷʮࢹʯʮཧʯͷى͜͜ • ଞͷπʔϧͱΈ߹Θͤͯ؍ଌΛิڧ
None
͝ਗ਼ௌ͋Γ͕ͱ͏͍͟͝·ͨ͠