https://noops.connpass.com/event/145092/
Observability:Mackerel ʹΑΔ؍ଌͱ Mackerel ͷ؍ଌ2019/09/17 NoOps Meetup Tokyo #8Asato WAKISAKA (Hatena)
View Slide
Asato WAKISAKAGitHub: @astj / Twitter @ast_jͯͳ ΞϓϦέʔγϣϯΤϯδχΞMackerel ։ൃνʔϜ (2016/08~)Mackerel ςοΫϦʔυ (2018/05~)
कඋൣғ:ϑϩϯτΤϯυ͔ΒSREing·ͰʢࣗশʣScala, Golang, TypeScript, …
Mackerel
• Mackerel ͬͯΔਓ• ͋Γ͕ͱ͏͍͟͝·͢ʂ• Mackerel ৮ͬͨ͜ͱ͕͋Δਓ• ͋Γ͕ͱ͏͍͟͝·͢ʂʂ• Mackerel ͬͯΔਓ• ͋Γ͕ͱ͏͍͟͝·͢ʂʂʂʂΞϯέʔτ
ࢹ
Monitoring
Observability
?
Observability ͱ Monitoring
• Observability: ʮՄ؍ଌੑʯ• γεςϜͷ෦ঢ়ଶ͕औಘͰ͖Δঢ়ଶ• 3ͭͷओཁͳߏཁૉ• Metrics, Logging, Tracing• Mackerel ͜ͷ͏ͪ Metrics த৺
• ʮՄ؍ଌੑʯ≠ʮ؍ଌʯ• ʮՄ؍ଌੑʯγεςϜ͕࣋ͭੑ࣭• ʮ؍ଌʯγεςϜ֎෦ͷߦಈ
Metrics ObservabilityLog ObservabilityTrace ObservabilityMetrics ObservationLog ObservationTrace Observation
Metrics ObservabilityLog ObservabilityTrace ObservabilityMetrics ObservationLog ObservationTrace Observation• Server Metrics• Middleware Metrics• Application Metrics• Business KPI• Etc…• Alerting• Dashboard Visualization• Analysis• Etc…
Metrics ObservabilityLog ObservabilityTrace ObservabilityMetrics ObservationLog ObservationTrace ObservationʮϝτϦοΫࢹʯ͜ͷลʁ• Server Metrics• Middleware Metrics• Application Metrics• Business KPI• Etc…• Alerting• Dashboard Visualization• Analysis• Etc…
• ͱ͍͑ɺͦͷઌͰਓ͕ؒ؍ଌ͍ͯ͠Δͣ• ݹయతʹΞϥʔτରԠͳͲ• ʮάϥϑΛݟΔʯ؍ଌ
Metrics ObservabilityLog ObservabilityTrace ObservabilityMetrics ObservationLog ObservationTrace Observationwਓؒ• ͋Ε• ͜Ε˞͜Εۃ
• લஈʹʮՄ؍ଌੑʯͦͷͷ• ޙஈͷʮ؍ଌʯΛ;ͨͭʹ͚ͯଊ͑Δ• ʮ؍ଌͷ࣮ʯ• ʮʢਓؒͷʣ؍ଌʯ
ObservabilityObservabilityObservabilityObservationObservationObservationObserve impl.Observe impl.Observe Impl.
ObservabilityObservabilityObservabilityObservationObservationObservationObserve impl.Observe impl.Observe Impl.ٕज़ͱͯ͠ͷ Observability ͕ ͜ͷลΓΛࢦ͢͜ͱ͋Ε
ObservabilityObservabilityObservabilityObservationObservationObservationObserve impl.Observe impl.Observe Impl.ιϦϡʔγϣϯͷͰObservability ͕ݴٴ͞ΕΔ࣌ ͜͜·Ͱؚ·ΕΔ͜ͱଟ͍
……Ͱɺ͍͍Ͱ͔͢Ͷʁ
• ٞͷ༨͍Ζ͍Ζ͋Γͦ͏• ͔͠͠ࠓ͜ͷͷਂ͍͠·ͤΜ• ߹্ࠨΛՄ؍ଌੑͱͯ͠͠·͢
ObservabilityObservabilityObservabilityObservationObservationObservationObserve impl.Observe impl.Observe Impl.• ϝτϦοΫࢹ:• ϝτϦοΫΛऩू͢Δ• ऩूͨ͠ϝτϦοΫΛूੵ͢Δ• άϥϑμογϡϘʔυͰՄࢹԽ͢Δ• ΞϥʔτΛൃใ͢Δ
ObservabilityObservabilityObservabilityObservationObservationObservationObserve impl.Observe impl.Observe Impl.ϝτϦοΫऩूϝτϦοΫूੵ • άϥϑ• Ξϥʔτ• μογϡϘʔυ˞Πϝʔδ• ਓؒ• ࣗಈԽରԠ
ObservabilityObservabilityObservabilityObservationObservationObservationObserve impl.Observe impl.Observe Impl.• ؍ଌ͢ΔͨΊʹ؍ଌՄೳʹ͢Δඞཁ͕͋Δ• Մ؍ଌੑࢹͷલఏͱͳΔ• ʢͷͰɺࢹͷจ຺ͰՄ؍ଌੑʹݴٴ͞ΕΔʣ
Mackerel ͷ֓೦
ϗετ
• ࢹɾཧͷىʹͳΔ֓೦• యܕతʹαʔόʔ• ʮαʔόʔͷΑ͏ͳͷʯϗετ• ίϯςφͷू߹ (e.g. ECS Task / k8s pod)• ΫϥυϦιʔε (e.g. Lambda Function)ϗετ
Service:BlogRole:App Role:DBHost AHost BHost CHost DService:BookmarkRole:AppHost EHost F…αʔϏεɾϩʔϧ
• ϗετΛʮׂʯͰྨ• μογϡϘʔυ্Ͱάϧʔϐϯά͞ΕΔ• αʔϏεϩʔϧ୯ҐͰࢹϧʔϧΛཧαʔϏεɾϩʔϧ
෦ঢ়ଶͷऩू
• ࣌ܥྻͰࢀরՄೳϝτϦοΫ
• ୯७ͳ൱ͱϝοηʔδΛه• ฏৗ࣌ͷཤྺهʹΒͳ͍νΣοΫࢹ
• ϩάʹର͢Δ(ϝτϦοΫ|νΣοΫ)ࢹͰදݱ• ϩάΛࠪͯ͠ύλʔϯϚον͢Δ• LTSV ܗࣜͷΞΫηεϩάΛूܭ͢ΔϩάΛࢹ͢Δ
ऩूͷλʔήοτ• ΤʔδΣϯτܥ• αʔόʔ• ίϯςφ • αʔϏεϝτϦοΫ• URL ֎ܗࢹ• ΠϯςάϨʔγϣϯ
• αʔόʔ্Ͱಈ͘ΤʔδΣϯτ• ΤʔδΣϯτ͔Β Mackerel ʹ POST ͢Δ• γεςϜϝτϦοΫࣗಈͰऩू• CPU, memory, interface…• ϓϥάΠϯͰऔಘใΛ֦ுͰ͖Δαʔόʔ͔Βͷऩू
ServerMackerelmackerel-agentFluentdGolang appOS statsmonitor-agentPluginsHTTP gostatsPOSTྫ
• ECS 1 Task k8s 1 pod Λ1ϗετ• αΠυΧʔίϯςφͱͯ͠ΤʔδΣϯτΛಋೖ• γεςϜϝτϦοΫΛࣗಈͰऩू• ϓϥάΠϯͰ֦ுՄೳίϯςφ͔Βͷऩू
TaskTaskTaskMackerelAgentContainerFluentdcontainerGolang appcontainermonitor-agentPluginsHTTP gostatsPOSTOrchestrationMetadataྫ
• ಛఆͷϗετʹඥ͔ͳ͍ϝτϦοΫ• Ϗδωε KPI ɺαʔϏεશମͷϨΠςϯγ• API Ͱࣗ༝ʹߘ• e.g. fluentdαʔϏεϝτϦοΫService:BookmarkRole:AppHost
• Mackerel ଆ͔Β URL ʹଓ• ଓՄ൱ϨεϙϯελΠϜͳͲΛऩूURL ֎ܗࢹ
• ΫϥυࣄۀऀͷϞχλϦϯά API ͔Βऔಘ• ΤʔδΣϯτͳ͠ͰऩूͰ͖Δ• ऩूϝτϦοΫΫϥυࣄۀऀґଘAWS / Azure ΠϯςάϨʔγϣϯMackerelUserCloudServicesCrawl
ऩू͢Δ༰
• γεςϜϝτϦοΫࣗಈͰऩू• ϛυϧΣΞެࣜϓϥάΠϯͰऩू• ΞϓϦέʔγϣϯͷϝτϦοΫʁʁ
• ݴޠϑϨʔϜϫʔΫ͚ͷϓϥάΠϯ͋Δ• JVM Go GC ౷ܭ؍ଌͰ͖Δ• Γͳ͍ϝτϦοΫͷ؍ଌखஈ͕ඞཁ
ObservabilityObservabilityObservabilityObservationObservationObservationObserve impl.Observe impl.Observe Impl.• ؍ଌ͢ΔͨΊʹ؍ଌՄೳʹ͢Δඞཁ͕͋Δ• Մ؍ଌੑࢹͷલఏͱͳΔ• ʢͷͰɺࢹͷจ຺ͰՄ؍ଌੑʹݴٴ͢Δʣ࠶ܝ• ؍ଌ͢ΔͨΊʹ؍ଌՄೳʹ͢Δඞཁ͕͋Δ
؍ଌ͍ͨ͠ϝτϦοΫԿΒ͔ͷखஈͰ؍ଌՄೳͰ͔͢ʁ؍ଌՄೳʹ͢ΔΤʔδΣϯτʢͷϓϥάΠϯʣͰ؍ଌɾऩू͢ΔNOYES
• ͲͷΑ͏ʹ؍ଌ͞ΕΔ͔Ͱ࣮͕ܾ·Δ؍ଌՄೳʹ͢Δ
• ϓϥάΠϯΛࣗ࡞ͯ͠ҙͷखஈͰ؍ଌ• HTTP Ͱ JSON ܗࣜΛಡΈࠐΜͰ؍ଌ؍ଌखஈ
• ϓϥάΠϯ = ࣮ߦϑΝΠϧ• ॴఆͷग़ྗͰϝτϦοΫ໊ͱΛग़ྗ͢Δ͚ͩ• ࣮ߦڥͰ“ݟ͑Δ”ͳΒԿͰऔΕΔࣗ࡞ϓϥάΠϯͰͷϝτϦοΫऩू$ mackerel-plugin-uptimeuptime.seconds 7749392 1568662368
• ࣮ࣗମ Bash Ͱ Go ͰԿͰ• ग़ྗϑΥʔϚοτ͕ૉͳͷͰ࣮Λબͳ͍• ؍ଌରʹ͋Θ࣮ͤͯ͢Δඞཁ͕͋Δ• ؍ଌରଆͷࣗ༝ߴ͘ͳΔࣗ࡞ϓϥάΠϯͰͷϝτϦοΫऩू
HTTP ܦ༝Ͱͷऩू• mackerel-plugin-json• ؍ଌ͍ͨ͠Λ JSON Ͱ HTTP ৴͢Δ• ϓϥάΠϯ͔ΒͦͷΤϯυϙΠϯτΛಡΈࠐΉ
ఆ• Q: Prometheus Exporter Έ͍ͨͳʁ• => ํੑ yesʢͬͱγϯϓϧʣ• Q: Prometheus Exporter ͑ͳ͍ͷʁ• => ……!
HTTP ܦ༝Ͱͷऩू$ mackerel-plugin-json \> -url=http://localhost/statsactive3 1568666460waiting 2 1568666460$ curl -fsSL http://localhost/stats{“active”:3,”waiting”:2}
HTTP ܦ༝Ͱͷऩू
• ͲͪΒ͕ྑ͍͔߹ʹΑΔ• طʹHTTPΛΔͳΒHTTP͕ศར• طଘͷ؍ଌखஈ͕͋Ε࠶ར༻͍ͨ͠
ऩूͨ͠ϝτϦοΫͷར༻
• Mackerel Ͱͷओͳ͍ಓ• ࢹʢΞϥʔτʣ• ՄࢹԽʢάϥϑɾμογϡϘʔυʣ
ࢹʢΞϥʔτʣ
;ͨͭͷΞϥʔτ• ଈ࣌ରԠΛଅ͢Ξϥʔτ• ӡ༻ऀͱͯ͠θϩʹ͍ͨ͠ʂ• Information ͱͯ͠ͷΞϥʔτ
ଈ࣌ରԠͷΞϥʔτ (on-call)• γεςϜͷՔಇ͕ଛͳΘΕ͍ͯΔ• ରԠΛඞཁͱ͢Δ• “Δ·Ͱ์ஔ”͕ڐ༰ͳΒଈ࣌ରԠͰͳ͍
Information ͱͯ͠ͷΞϥʔτ• ٸ͕ͳ͍͕ԿΒ͔ͷΞΫγϣϯΛٻΊΔ• ྫ: Խϗετ͕1μϯ• γεςϜ͕ଓߦՄೳͳൣғͷʮҟৗʯ• ಉ͡ࢦඪ͕͞ΒʹѱԽ͢Εଈ࣌ରԠ
• ࣮ࡍʹγεςϜ͕Քಇ͍ͯ͠Δ͔ Ͳ͏͔Λࢹ͢Δ• e.g. ֎ܗࢹ, ΤϥʔϨʔτ• ˱ CPU 100%Ͱಈ͍ͯΕྑ͍ଈ࣌ରԠͷΞϥʔτ
• ͱ͍͑߹ʹΑΓ͚Γ• ͷՔಇͷࢹͰͳͯ͘ɺ ͔ͦ͜ΒՔಇঢ়گ͕ԋ៷ՄೳͳΒ༗༻
Information ͱͯ͠ͷΞϥʔτ• γεςϜՔಇҎ֎ͷࢦඪऔΓಘΔ• ଈ࣌ରԠͱಉ͡௨ܦ࿏ʹࡌͤͳ͍
Ξϥʔτશൠ• ͳΜͰ͔ΜͰΞϥʔτʹ͠ͳ͍• ຊʹඞཁͳΞϥʔτ͕ຒΕΔ• Information ͩΒͩΒ૿͕͑ͪ
ՄࢹԽ
• ෦ͷঢ়ଶΛ࣌ܥྻ͖ͭͰه͢Δ• खಈͰऩू͠ͳͯ͘؍ଌͰ͖Δ• ݱࡏͷঢ়ଶΛѲ͢Δखஈ• ఆৗ࣌ɾো࣌Θͣ• աڈͷঢ়ଶΛৼΓฦΔखஈάϥϑɾμογϡϘʔυ
؍ଌΛҭͯΔ
ඞཁͳ؍ଌྔ͕࠷ॳ͔Βἧ͏ͱݶΒͳ͍
• ։ൃɾݕূதʹؾͮ͘• ࢹ༻ҙ͠Α͏ͱͯ͠ෆʹؾͮ͘• োى͖ͯΈͯෆʹؾͮͨ͘ͱ͑…
ඞཁͳ؍ଌ࠷ॳ͔Βἧ͏ͱݶΒͳ͍
• ։ൃɾݕূதʹؾͮ͘• ෛՙࢼݧͰؾͮ͘ͱඒ͍͠ʂ• োى͖ͯΈͯෆʹؾͮͨ͘ͱ͑…
αΠΫϧΛճ͍ͯ͘͠
• ࠷ॳ͔ΒඋͰ͖Εྑ͍• ಛʹඞਢͷ߲͍͑ͯ͘• γεςϜͷՔಇͷࢹʂʂ• ܧଓతʹมߋɾվળ͍ͯ͘͠
Mackerel ͰMackerel Λ؍ଌ͢Δ
Mackerel ͷٕज़ελοΫ• AWS ্ʹߏங• EC2 ӡ༻ͱ ECS Fargate ͕ࠞࡏ• ϝΠϯͷ JVM ͳͲ͕ EC2• গͣͭ͠ίϯςφϫʔΫϩʔυʹҠߦத
ઢ
ίϯςφศར
• Πϯελϯεͷੈ͕ཁΒͳ͍• Πϯϑϥ·ͰؔΘΔߏมߋ͕खܰ• “chef ߋ৽ͯ͠Πϯελϯε࡞Γͯ͠ೖΕ͑ͯ…”• ʮخ͘͠ͳ͍͜ͱʯ͡ΌΜ……
ઢ͜͜·Ͱ
Mackerel ͷٕज़ελοΫ࣌ܥྻσʔλϕʔε: ϚωʔδυαʔϏεϕʔε
αʔόϨεΞʔΩςΫνϟศར
• ૄ݁߹ɾϦτϥΠՄೳͳઃܭ• ϝοηʔδΩϡʔΛڬΉ• Lambda ͷॲཧ༰Λႈʹઃܭ• ύϥϝʔλมߋͰεέʔϧ͢Δ• ʢ FaaS ಛ༗ͷӡ༻՝θϩͰͳ͍ʣ
Mackerel ͷٕज़ελοΫʢڵຯ͕͋ΕաڈͷొஃࢿྉΛ͝ཡԼ͍͞ʂʣ
JVM ΞϓϦέʔγϣϯͷ؍ଌ• ϝτϦΫεɾࢹ: mackerel• jstat ͱ jolokia ʹΑΔJVM෦ߏͷ؍ଌ• αʔόʔ͝ͱͷϨΠςϯγࢹ• ίωΫγϣϯϓʔϧࢹ• etc…
JVM ΞϓϦέʔγϣϯͷ؍ଌ• APM: NewRelic APM• ΑΓਫ਼៛ͳ෦ੳ͕ඞཁͳͱ͖ʹ• ϩάղੳ: fluentd => Kibana• ΞΫηεϩάɺΞϓϦέʔγϣϯϩά
։ൃڥͷ؍ଌ• ։ൃڥʢ݉εςʔδϯάڥʣ• ຊ൪ͱಉ༷ͷߏʢنখ͍͞ʣ• ಉ༷ͷ؍ଌπʔϧߏ• => ϦϦʔεલ͔Β։ൃڥͰ؍ଌ
ఆ• Q: ։ൃڥͷࢹͷίετʁ• => զʑMackerelΛແྉͰ͑Δʢଇʣ
• த֩ʹ Mackerel• ීஈͷʮࢹʯʮཧʯͷى͜͜• ଞͷπʔϧͱΈ߹Θͤͯ؍ଌΛิڧ
͝ਗ਼ௌ͋Γ͕ͱ͏͍͟͝·ͨ͠
astj
carta_engineering
you5805
kazeburo
uzulla
sbtechnight
doradora09
koba1t
kentosuzuki
tatematsu_san
makamaka
cherdarchuk
aarron
sstephenson
cromwellryan
chriscoyier
mza
robhawkes
michaelherold
maggiecrowley
destraynor
dougneiner
sachag