Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Sign up for free
Menu
Search
Features
All features
Private URLs
Password Protection
Custom URLS
Scheduled publishing
Remove Branding
Restrict embedding
Deck Collections
Notes
Features
All features
Private URLs
Password Protection
Custom URLS
Scheduled publishing
Remove Branding
Restrict embedding
Deck Collections
Notes
Explore
Featured decks
Featured speakers
Programming
Technology
Storyboards
Explore
Featured decks
Featured speakers
Programming
Technology
Storyboards
Pricing
Search
Sign in
Sign up for free
Observability: Mackerel による観測と Mackerel の観測 / ...
Search
astj
PRO
September 17, 2019
Technology
2.4k
2
Share
Embed
Copy iframe code
Copy JS code
Copy link
Start on current slide
Observability: Mackerel による観測と Mackerel の観測 / NoOps Meetup Tokyo #8
https://noops.connpass.com/event/145092/
astj
PRO
September 17, 2019
Transcript
106 slides
Observability: Mackerel ʹΑΔ؍ଌ ͱ Mackerel ͷ؍ଌ 2019/09/17 NoOps Meetup Tokyo
#8 Asato WAKISAKA (Hatena)
Asato WAKISAKA GitHub: @astj / Twitter @ast_j ͯͳ ΞϓϦέʔγϣϯΤϯδχΞ Mackerel
։ൃνʔϜ (2016/08~) Mackerel ςοΫϦʔυ (2018/05~)
कඋൣғ: ϑϩϯτΤϯυ͔ΒSREing·Ͱʢࣗশʣ Scala, Golang, TypeScript, …
Mackerel
None
• Mackerel ͬͯΔਓ • ͋Γ͕ͱ͏͍͟͝·͢ʂ • Mackerel ৮ͬͨ͜ͱ͕͋Δਓ • ͋Γ͕ͱ͏͍͟͝·͢ʂʂ
• Mackerel ͬͯΔਓ • ͋Γ͕ͱ͏͍͟͝·͢ʂʂʂʂ Ξϯέʔτ
None
ࢹ
Monitoring
Observability
?
Observability ͱ Monitoring
• Observability: ʮՄ؍ଌੑʯ • γεςϜͷ෦ঢ়ଶ͕औಘͰ͖Δঢ়ଶ • 3ͭͷओཁͳߏཁૉ • Metrics, Logging,
Tracing • Mackerel ͜ͷ͏ͪ Metrics த৺
• ʮՄ؍ଌੑʯ≠ʮ؍ଌʯ • ʮՄ؍ଌੑʯγεςϜ͕࣋ͭੑ࣭ • ʮ؍ଌʯγεςϜ֎෦ͷߦಈ
Metrics Observability Log Observability Trace Observability Metrics Observation Log Observation
Trace Observation
Metrics Observability Log Observability Trace Observability Metrics Observation Log Observation
Trace Observation • Server Metrics • Middleware Metrics • Application Metrics • Business KPI • Etc… • Alerting • Dashboard Visualization • Analysis • Etc…
Metrics Observability Log Observability Trace Observability Metrics Observation Log Observation
Trace Observation ʮϝτϦοΫࢹʯ͜ͷลʁ • Server Metrics • Middleware Metrics • Application Metrics • Business KPI • Etc… • Alerting • Dashboard Visualization • Analysis • Etc…
• ͱ͍͑ɺͦͷઌͰਓ͕ؒ؍ଌ͍ͯ͠Δͣ • ݹయతʹΞϥʔτରԠͳͲ • ʮάϥϑΛݟΔʯ؍ଌ
Metrics Observability Log Observability Trace Observability Metrics Observation Log Observation
Trace Observation wਓؒ • ͋Ε • ͜Ε ˞͜Εۃ
• લஈʹʮՄ؍ଌੑʯͦͷͷ • ޙஈͷʮ؍ଌʯΛ;ͨͭʹ͚ͯଊ͑Δ • ʮ؍ଌͷ࣮ʯ • ʮʢਓؒͷʣ؍ଌʯ
Observability Observability Observability Observation Observation Observation Observe impl. Observe impl.
Observe Impl.
Observability Observability Observability Observation Observation Observation Observe impl. Observe impl.
Observe Impl. ٕज़ͱͯ͠ͷ Observability ͕ ͜ͷลΓΛࢦ͢͜ͱ͋Ε
Observability Observability Observability Observation Observation Observation Observe impl. Observe impl.
Observe Impl. ιϦϡʔγϣϯͷͰ Observability ͕ݴٴ͞ΕΔ࣌ ͜͜·Ͱؚ·ΕΔ͜ͱଟ͍
……Ͱɺ͍͍Ͱ͔͢Ͷʁ
• ٞͷ༨͍Ζ͍Ζ͋Γͦ͏ • ͔͠͠ࠓ͜ͷͷਂ͍͠·ͤΜ • ߹্ࠨΛՄ؍ଌੑͱͯ͠͠·͢
Observability Observability Observability Observation Observation Observation Observe impl. Observe impl.
Observe Impl. • ϝτϦοΫࢹ: • ϝτϦοΫΛऩू͢Δ • ऩूͨ͠ϝτϦοΫΛूੵ͢Δ • άϥϑμογϡϘʔυͰՄࢹԽ͢Δ • ΞϥʔτΛൃใ͢Δ
Observability Observability Observability Observation Observation Observation Observe impl. Observe impl.
Observe Impl. ϝτϦοΫऩू ϝτϦοΫूੵ • άϥϑ • Ξϥʔτ • μογϡϘʔυ ˞Πϝʔδ • ਓؒ • ࣗಈԽରԠ
Observability Observability Observability Observation Observation Observation Observe impl. Observe impl.
Observe Impl. • ؍ଌ͢ΔͨΊʹ؍ଌՄೳʹ͢Δඞཁ͕͋Δ • Մ؍ଌੑࢹͷલఏͱͳΔ • ʢͷͰɺࢹͷจ຺ͰՄ؍ଌੑʹݴٴ͞ΕΔʣ
None
Mackerel ͷ֓೦
ϗετ
• ࢹɾཧͷىʹͳΔ֓೦ • యܕతʹαʔόʔ • ʮαʔόʔͷΑ͏ͳͷʯϗετ • ίϯςφͷू߹ (e.g. ECS
Task / k8s pod) • ΫϥυϦιʔε (e.g. Lambda Function) ϗετ
Service:Blog Role:App Role:DB Host A Host B Host C Host
D Service:Bookmark Role:App Host E Host F … αʔϏεɾϩʔϧ
• ϗετΛʮׂʯͰྨ • μογϡϘʔυ্Ͱάϧʔϐϯά͞ΕΔ • αʔϏεϩʔϧ୯ҐͰࢹϧʔϧΛཧ αʔϏεɾϩʔϧ
None
None
None
෦ঢ়ଶͷऩू
• ࣌ܥྻͰࢀরՄೳ ϝτϦοΫ
• ୯७ͳ൱ͱϝοηʔδΛه • ฏৗ࣌ͷཤྺهʹΒͳ͍ νΣοΫࢹ
• ϩάʹର͢Δ(ϝτϦοΫ|νΣοΫ)ࢹͰදݱ • ϩάΛࠪͯ͠ύλʔϯϚον͢Δ • LTSV ܗࣜͷΞΫηεϩάΛूܭ͢Δ ϩάΛࢹ͢Δ
None
ऩूͷλʔήοτ • ΤʔδΣϯτܥ • αʔόʔ • ίϯςφ • αʔϏεϝτϦοΫ •
URL ֎ܗࢹ • ΠϯςάϨʔγϣϯ
• αʔόʔ্Ͱಈ͘ΤʔδΣϯτ • ΤʔδΣϯτ͔Β Mackerel ʹ POST ͢Δ • γεςϜϝτϦοΫࣗಈͰऩू
• CPU, memory, interface… • ϓϥάΠϯͰऔಘใΛ֦ுͰ͖Δ αʔόʔ͔Βͷऩू
Server Mackerel mackerel-agent Fluentd Golang app OS stats monitor-agent Plugins
HTTP gostats POST ྫ
• ECS 1 Task k8s 1 pod Λ1ϗετ •
αΠυΧʔίϯςφͱͯ͠ΤʔδΣϯτΛಋೖ • γεςϜϝτϦοΫΛࣗಈͰऩू • ϓϥάΠϯͰ֦ுՄೳ ίϯςφ͔Βͷऩू
Task Task Task Mackerel Agent Container Fluentd container Golang app
container monitor-agent Plugins HTTP gostats POST Orchestration Metadata ྫ
• ಛఆͷϗετʹඥ͔ͳ͍ϝτϦοΫ • Ϗδωε KPI ɺαʔϏεશମͷϨΠςϯγ • API Ͱࣗ༝ʹߘ •
e.g. fluentd αʔϏεϝτϦοΫ Service:Bookmark Role:App Host
• Mackerel ଆ͔Β URL ʹଓ • ଓՄ൱ϨεϙϯελΠϜͳͲΛऩू URL ֎ܗࢹ
• ΫϥυࣄۀऀͷϞχλϦϯά API ͔Βऔಘ • ΤʔδΣϯτͳ͠ͰऩूͰ͖Δ • ऩूϝτϦοΫΫϥυࣄۀऀґଘ AWS /
Azure ΠϯςάϨʔγϣϯ Mackerel User Cloud Services Crawl
ऩू͢Δ༰
• γεςϜϝτϦοΫࣗಈͰऩू • ϛυϧΣΞެࣜϓϥάΠϯͰऩू • ΞϓϦέʔγϣϯͷϝτϦοΫʁʁ
• γεςϜϝτϦοΫࣗಈͰऩू • ϛυϧΣΞެࣜϓϥάΠϯͰऩू • ΞϓϦέʔγϣϯͷϝτϦοΫʁʁ
• ݴޠϑϨʔϜϫʔΫ͚ͷϓϥάΠϯ͋Δ • JVM Go GC ౷ܭ؍ଌͰ͖Δ •
Γͳ͍ϝτϦοΫͷ؍ଌखஈ͕ඞཁ
Observability Observability Observability Observation Observation Observation Observe impl. Observe impl.
Observe Impl. • ؍ଌ͢ΔͨΊʹ؍ଌՄೳʹ͢Δඞཁ͕͋Δ • Մ؍ଌੑࢹͷલఏͱͳΔ • ʢͷͰɺࢹͷจ຺ͰՄ؍ଌੑʹݴٴ͢Δʣ ࠶ܝ • ؍ଌ͢ΔͨΊʹ؍ଌՄೳʹ͢Δඞཁ͕͋Δ
؍ଌ͍ͨ͠ϝτϦοΫԿΒ͔ͷखஈͰ؍ଌՄೳͰ͔͢ʁ ؍ଌՄೳʹ͢Δ ΤʔδΣϯτʢͷϓϥάΠϯʣͰ؍ଌɾऩू͢Δ NO YES
؍ଌ͍ͨ͠ϝτϦοΫԿΒ͔ͷखஈͰ؍ଌՄೳͰ͔͢ʁ ؍ଌՄೳʹ͢Δ ΤʔδΣϯτʢͷϓϥάΠϯʣͰ؍ଌɾऩू͢Δ NO YES
• ͲͷΑ͏ʹ؍ଌ͞ΕΔ͔Ͱ࣮͕ܾ·Δ ؍ଌՄೳʹ͢Δ
؍ଌ͍ͨ͠ϝτϦοΫԿΒ͔ͷखஈͰ؍ଌՄೳͰ͔͢ʁ ؍ଌՄೳʹ͢Δ ΤʔδΣϯτʢͷϓϥάΠϯʣͰ؍ଌɾऩू͢Δ NO YES
• ϓϥάΠϯΛࣗ࡞ͯ͠ҙͷखஈͰ؍ଌ • HTTP Ͱ JSON ܗࣜΛಡΈࠐΜͰ؍ଌ ؍ଌखஈ
• ϓϥάΠϯ = ࣮ߦϑΝΠϧ • ॴఆͷग़ྗͰϝτϦοΫ໊ͱΛग़ྗ͢Δ͚ͩ • ࣮ߦڥͰ“ݟ͑Δ”ͳΒԿͰऔΕΔ ࣗ࡞ϓϥάΠϯͰͷϝτϦοΫऩू $
mackerel-plugin-uptime uptime.seconds 7749392 1568662368
• ࣮ࣗମ Bash Ͱ Go ͰԿͰ • ग़ྗϑΥʔϚοτ͕ૉͳͷͰ࣮Λબͳ͍ • ؍ଌରʹ͋Θ࣮ͤͯ͢Δඞཁ͕͋Δ
• ؍ଌରଆͷࣗ༝ߴ͘ͳΔ ࣗ࡞ϓϥάΠϯͰͷϝτϦοΫऩू
HTTP ܦ༝Ͱͷऩू • mackerel-plugin-json • ؍ଌ͍ͨ͠Λ JSON Ͱ HTTP ৴͢Δ
• ϓϥάΠϯ͔ΒͦͷΤϯυϙΠϯτΛಡΈࠐΉ
ఆ • Q: Prometheus Exporter Έ͍ͨͳʁ • => ํੑ yesʢͬͱγϯϓϧʣ
• Q: Prometheus Exporter ͑ͳ͍ͷʁ • => ……!
HTTP ܦ༝Ͱͷऩू $ mackerel-plugin-json \ > -url=http://localhost/stats active3 1568666460 waiting
2 1568666460 $ curl -fsSL http://localhost/stats {“active”:3,”waiting”:2}
HTTP ܦ༝Ͱͷऩू
• ͲͪΒ͕ྑ͍͔߹ʹΑΔ • طʹHTTPΛΔͳΒHTTP͕ศར • طଘͷ؍ଌखஈ͕͋Ε࠶ར༻͍ͨ͠
ऩूͨ͠ϝτϦοΫͷར༻
• Mackerel Ͱͷओͳ͍ಓ • ࢹʢΞϥʔτʣ • ՄࢹԽʢάϥϑɾμογϡϘʔυʣ
ࢹʢΞϥʔτʣ
;ͨͭͷΞϥʔτ • ଈ࣌ରԠΛଅ͢Ξϥʔτ • ӡ༻ऀͱͯ͠θϩʹ͍ͨ͠ʂ • Information ͱͯ͠ͷΞϥʔτ
ଈ࣌ରԠͷΞϥʔτ (on-call) • γεςϜͷՔಇ͕ଛͳΘΕ͍ͯΔ • ରԠΛඞཁͱ͢Δ • “Δ·Ͱ์ஔ”͕ڐ༰ͳΒଈ࣌ରԠͰͳ͍
Information ͱͯ͠ͷΞϥʔτ • ٸ͕ͳ͍͕ԿΒ͔ͷΞΫγϣϯΛٻΊΔ • ྫ: Խϗετ͕1μϯ • γεςϜ͕ଓߦՄೳͳൣғͷʮҟৗʯ •
ಉ͡ࢦඪ͕͞ΒʹѱԽ͢Εଈ࣌ରԠ
• ࣮ࡍʹγεςϜ͕Քಇ͍ͯ͠Δ͔ Ͳ͏͔Λࢹ͢Δ • e.g. ֎ܗࢹ, ΤϥʔϨʔτ • ˱ CPU
100%Ͱಈ͍ͯΕྑ͍ ଈ࣌ରԠͷΞϥʔτ
• ͱ͍͑߹ʹΑΓ͚Γ • ͷՔಇͷࢹͰͳͯ͘ɺ ͔ͦ͜ΒՔಇঢ়گ͕ԋ៷ՄೳͳΒ༗༻
Information ͱͯ͠ͷΞϥʔτ • γεςϜՔಇҎ֎ͷࢦඪऔΓಘΔ • ଈ࣌ରԠͱಉ͡௨ܦ࿏ʹࡌͤͳ͍
Ξϥʔτશൠ • ͳΜͰ͔ΜͰΞϥʔτʹ͠ͳ͍ • ຊʹඞཁͳΞϥʔτ͕ຒΕΔ • Information ͩΒͩΒ૿͕͑ͪ
ՄࢹԽ
• ෦ͷঢ়ଶΛ࣌ܥྻ͖ͭͰه͢Δ • खಈͰऩू͠ͳͯ͘؍ଌͰ͖Δ • ݱࡏͷঢ়ଶΛѲ͢Δखஈ • ఆৗ࣌ɾো࣌Θͣ • աڈͷঢ়ଶΛৼΓฦΔखஈ
άϥϑɾμογϡϘʔυ
؍ଌΛҭͯΔ
ඞཁͳ؍ଌྔ͕࠷ॳ͔Βἧ͏ͱݶΒͳ͍
• ։ൃɾݕূதʹؾͮ͘ • ࢹ༻ҙ͠Α͏ͱͯ͠ෆʹؾͮ͘ • োى͖ͯΈͯෆʹؾͮ͘ ͨͱ͑…
ඞཁͳ؍ଌ࠷ॳ͔Βἧ͏ͱݶΒͳ͍
• ։ൃɾݕূதʹؾͮ͘ • ෛՙࢼݧͰؾͮ͘ͱඒ͍͠ʂ • োى͖ͯΈͯෆʹؾͮ͘ ͨͱ͑…
αΠΫϧΛճ͍ͯ͘͠
• ࠷ॳ͔ΒඋͰ͖Εྑ͍ • ಛʹඞਢͷ߲͍͑ͯ͘ • γεςϜͷՔಇͷࢹʂʂ • ܧଓతʹมߋɾվળ͍ͯ͘͠
None
Mackerel Ͱ Mackerel Λ؍ଌ͢Δ
Mackerel ͷٕज़ελοΫ • AWS ্ʹߏங • EC2 ӡ༻ͱ ECS Fargate
͕ࠞࡏ • ϝΠϯͷ JVM ͳͲ͕ EC2 • গͣͭ͠ίϯςφϫʔΫϩʔυʹҠߦத
ઢ
ίϯςφศར
• Πϯελϯεͷੈ͕ཁΒͳ͍ • Πϯϑϥ·ͰؔΘΔߏมߋ͕खܰ • “chef ߋ৽ͯ͠Πϯελϯε࡞Γͯ͠ೖΕ ͑ͯ…” • ʮخ͘͠ͳ͍͜ͱʯ͡ΌΜ……
ઢ͜͜·Ͱ
Mackerel ͷٕज़ελοΫ ࣌ܥྻσʔλϕʔε: ϚωʔδυαʔϏεϕʔε
ઢ
αʔόϨεΞʔΩςΫνϟศར
• ૄ݁߹ɾϦτϥΠՄೳͳઃܭ • ϝοηʔδΩϡʔΛڬΉ • Lambda ͷॲཧ༰Λႈʹઃܭ • ύϥϝʔλมߋͰεέʔϧ͢Δ •
ʢ FaaS ಛ༗ͷӡ༻՝θϩͰͳ͍ʣ
ઢ͜͜·Ͱ
Mackerel ͷٕज़ελοΫ ʢڵຯ͕͋ΕաڈͷొஃࢿྉΛ͝ཡԼ͍͞ʂʣ
JVM ΞϓϦέʔγϣϯͷ؍ଌ • ϝτϦΫεɾࢹ: mackerel • jstat ͱ jolokia ʹΑΔJVM෦ߏͷ؍ଌ
• αʔόʔ͝ͱͷϨΠςϯγࢹ • ίωΫγϣϯϓʔϧࢹ • etc…
JVM ΞϓϦέʔγϣϯͷ؍ଌ • APM: NewRelic APM • ΑΓਫ਼៛ͳ෦ੳ͕ඞཁͳͱ͖ʹ • ϩάղੳ:
fluentd => Kibana • ΞΫηεϩάɺΞϓϦέʔγϣϯϩά
։ൃڥͷ؍ଌ • ։ൃڥʢ݉εςʔδϯάڥʣ • ຊ൪ͱಉ༷ͷߏʢنখ͍͞ʣ • ಉ༷ͷ؍ଌπʔϧߏ • => ϦϦʔεલ͔Β։ൃڥͰ؍ଌ
ఆ • Q: ։ൃڥͷࢹͷίετʁ • => զʑMackerelΛແྉͰ͑Δʢଇʣ
• த֩ʹ Mackerel • ීஈͷʮࢹʯʮཧʯͷى͜͜ • ଞͷπʔϧͱΈ߹Θͤͯ؍ଌΛิڧ
None
͝ਗ਼ௌ͋Γ͕ͱ͏͍͟͝·ͨ͠
More Decks by astj
See All by astj
ISUCON14公式反省会LT: 社内ISUCONの話
astj
PRO
0
620
Mackerel の時系列データベースにおける Redis Cluster の利用と Amazon ElastiCache への移行について / AWS Purpose-Built Databases Week
astj
PRO
3
6.2k
Mackerel のコンテナ監視の近況 / Mackerel Meetup #13
astj
PRO
1
2.7k
サービス開発と健全なプロダクトメンテナンスを開発チームで持続的に取り組む / Developers Boost KANSAI
astj
PRO
1
1.6k
Mackerel and Stripe / Qiita x Stripe Meetup
astj
PRO
0
5.8k
Mackerel をオンプレミスから AWS に移してからの1年半を振り返る / Hatena Engineer Seminar #11
astj
PRO
0
1.4k
AWS で実現した Mackerel 時系列データ1分粒度長期保存の裏側 / Mackerel Meetup #11 Tokyo
astj
PRO
6
31k
稼働中の Web サービスの perl のバージョンを上げていく
astj
PRO
0
7.6k
Perl 6 で Web Application Framework をつくる
astj
PRO
0
4.9k
Other Decks in Technology
See All in Technology
コピーオンライト型ファイルシステムと ランダム書き込み
sat
PRO
2
130
農家は秘伝のタレを知らない - NOBODY KNOWS THE HIDDEN SOURCE - Kaggriculture 3rd Place Solution
masspeaks
3
670
Kansai WordPress Meetup@大阪『ブロックテーマを作る上で知っておくべきこと
torounit
0
260
SBOM 生成ツールで 2026 Minimum Elements はどこまで満たせるのか
gokken
0
130
20261119_AgentCore
takuyay0ne
1
160
AWS re:Invent 2026 FSI Pre-Event Networking Night
junjikoide
0
130
Taking back control of your AI development
inesmontani
PRO
0
310
MCPでネットワーク障害調査を自動化する前に直すべき3つのこと
yama3133
1
650
[関西Kaggler会 20261113] 続・コミュニティコンペのススメ
nejumi
0
190
異夢同船〜異なる夢でも同じ船でなんとかする〜
fortegp05
1
580
「個社」ではなく「業界」の課題を解く AI 対話開発 ー 価値を広げる設計と、価値を守るエンジニアリング ー
ivry_presentationmaterials
4
830
「任されビリティ」で引き受けて「よしなりてぃ」に頼らず任せる / Accept tasks with "makasarebility" and hand them over without relying on "yoshinarily"
pauli
1
880
Featured
See All Featured
Speed Design
sergeychernyshev
33
2.2k
RailsConf & Balkan Ruby 2019: The Past, Present, and Future of Rails at GitHub
eileencodes
141
35k
Test your architecture with Archunit
thirion
2
2.5k
Jess Joyce - The Pitfalls of Following Frameworks
techseoconnect
PRO
1
480
DBのスキルで生き残る技術 - AI時代におけるテーブル設計の勘所
soudai
PRO
68
60k
The SEO Collaboration Effect
kristinabergwall1
1
620
Rebuilding a faster, lazier Slack
samanthasiow
85
9.8k
Distributed Sagas: A Protocol for Coordinating Microservices
caitiem20
333
23k
Digital Projects Gone Horribly Wrong (And the UX Pros Who Still Save the Day) - Dean Schuster
uxyall
2
4k
Technical Leadership for Architectural Decision Making
baasie
4
650
Effective software design: The role of men in debugging patriarchy in IT @ Voxxed Days AMS
baasie
1
620
How To Stay Up To Date on Web Technology
chriscoyier
790
250k