Observability: Mackerel による観測と Mackerel の観測 / NoOps Meetup Tokyo #8

https://noops.connpass.com/event/145092/

Transcript

1. Observability: Mackerel ʹΑΔ؍ଌ ͱ Mackerel ͷ؍ଌ 2019/09/17 NoOps Meetup Tokyo #8 Asato WAKISAKA (Hatena)
2. Asato WAKISAKA GitHub: @astj / Twitter @ast_j ͸ͯͳ ΞϓϦέʔγϣϯΤϯδχΞ Mackerel ։ൃνʔϜ (2016/08~) Mackerel ςοΫϦʔυ (2018/05~)
3. कඋൣғ: ϑϩϯτΤϯυ͔ΒSREing·Ͱʢࣗশʣ Scala, Golang, TypeScript, …
4. Mackerel
6. • Mackerel ஌ͬͯΔਓ • ͋Γ͕ͱ͏͍͟͝·͢ʂ • Mackerel ৮ͬͨ͜ͱ͕͋Δਓ • ͋Γ͕ͱ͏͍͟͝·͢ʂʂ • Mackerel ࢖ͬͯΔਓ • ͋Γ͕ͱ͏͍͟͝·͢ʂʂʂʂ Ξϯέʔτ
8. ؂ࢹ
9. Monitoring
10. Observability
11. ?
12. Observability ͱ Monitoring
13. • Observability: ʮՄ؍ଌੑʯ • γεςϜͷ಺෦ঢ়ଶ͕औಘͰ͖Δঢ়ଶ • 3ͭͷओཁͳߏ੒ཁૉ • Metrics, Logging, Tracing • Mackerel ͸͜ͷ͏ͪ Metrics த৺
14. • ʮՄ؍ଌੑʯ≠ʮ؍ଌʯ • ʮՄ؍ଌੑʯ͸γεςϜ͕࣋ͭੑ࣭ • ʮ؍ଌʯ͸γεςϜ֎෦ͷߦಈ
15. Metrics Observability Log Observability Trace Observability Metrics Observation Log Observation Trace Observation
16. Metrics Observability Log Observability Trace Observability Metrics Observation Log Observation Trace Observation • Server Metrics • Middleware Metrics • Application Metrics • Business KPI • Etc… • Alerting • Dashboard Visualization • Analysis • Etc…
17. Metrics Observability Log Observability Trace Observability Metrics Observation Log Observation Trace Observation ʮϝτϦοΫ؂ࢹʯ͸͜ͷลʁ • Server Metrics • Middleware Metrics • Application Metrics • Business KPI • Etc… • Alerting • Dashboard Visualization • Analysis • Etc…
18. • ͱ͸͍͑ɺͦͷઌͰ͸ਓ͕ؒ؍ଌ͍ͯ͠Δ͸ͣ • ݹయతʹ͸ΞϥʔτରԠͳͲ • ʮάϥϑΛݟΔʯ΋؍ଌ
19. Metrics Observability Log Observability Trace Observability Metrics Observation Log Observation Trace Observation wਓؒ • ͋Ε΍ • ͜Ε΍ ˞͜Ε͸ۃ୺
20. • લஈʹ͸ʮՄ؍ଌੑʯͦͷ΋ͷ • ޙஈͷʮ؍ଌʯΛ;ͨͭʹ෼͚ͯଊ͑Δ • ʮ؍ଌͷ࣮૷ʯ • ʮʢਓؒͷʣ؍ଌʯ
21. Observability Observability Observability Observation Observation Observation Observe impl. Observe impl. Observe Impl.
22. Observability Observability Observability Observation Observation Observation Observe impl. Observe impl. Observe Impl. ٕज़ͱͯ͠ͷ Observability ͕
 ͜ͷลΓΛࢦ͢͜ͱ΋͋Ε͹
23. Observability Observability Observability Observation Observation Observation Observe impl. Observe impl. Observe Impl. ιϦϡʔγϣϯͷ࿩୊Ͱ Observability ͕ݴٴ͞ΕΔ࣌͸
 ͜͜·Ͱؚ·ΕΔ͜ͱ΋ଟ͍
24. ……Ͱɺ͍͍Ͱ͔͢Ͷʁ
25. • ٞ࿦ͷ༨஍͸͍Ζ͍Ζ͋Γͦ͏ • ͔͠͠ࠓ೔͸͜ͷ࿩ͷਂ௥͍͸͠·ͤΜ • ౎߹্ࠨ୺ΛՄ؍ଌੑͱͯ͠࿩͠·͢
26. Observability Observability Observability Observation Observation Observation Observe impl. Observe impl. Observe Impl. • ϝτϦοΫ؂ࢹ͸: • ϝτϦοΫΛऩू͢Δ • ऩूͨ͠ϝτϦοΫΛूੵ͢Δ • άϥϑ΍μογϡϘʔυͰՄࢹԽ͢Δ • ΞϥʔτΛൃใ͢Δ
27. Observability Observability Observability Observation Observation Observation Observe impl. Observe impl. Observe Impl. ϝτϦοΫऩू ϝτϦοΫूੵ • άϥϑ • Ξϥʔτ • μογϡϘʔυ ˞Πϝʔδ • ਓؒ • ࣗಈԽରԠ
28. Observability Observability Observability Observation Observation Observation Observe impl. Observe impl. Observe Impl. • ؍ଌ͢ΔͨΊʹ͸؍ଌՄೳʹ͢Δඞཁ͕͋Δ • Մ؍ଌੑ͸؂ࢹͷલఏͱͳΔ • ʢͷͰɺ؂ࢹͷจ຺ͰՄ؍ଌੑʹݴٴ͞ΕΔʣ
30. Mackerel ͷ֓೦
31. ϗετ
32. • ؂ࢹɾ؅ཧͷى఺ʹͳΔ֓೦ • యܕతʹ͸αʔόʔ • ʮαʔόʔͷΑ͏ͳ΋ͷʯ΋ϗετ • ίϯςφͷू߹ (e.g. ECS Task / k8s pod) • Ϋϥ΢υϦιʔε (e.g. Lambda Function) ϗετ
33. Service:Blog Role:App Role:DB Host A Host B Host C Host D Service:Bookmark Role:App Host E Host F … αʔϏεɾϩʔϧ
34. • ϗετΛʮ໾ׂʯͰ෼ྨ • μογϡϘʔυ্Ͱάϧʔϐϯά͞ΕΔ • αʔϏε΍ϩʔϧ୯ҐͰ؂ࢹϧʔϧΛ؅ཧ αʔϏεɾϩʔϧ
38. ಺෦ঢ়ଶͷऩू
39. • ࣌ܥྻͰࢀরՄೳ ϝτϦοΫ
40. • ୯७ͳ੒൱ͱϝοηʔδΛه࿥ • ฏৗ࣌ͷཤྺ͸ه࿥ʹ࢒Βͳ͍ νΣοΫ؂ࢹ
41. • ϩάʹର͢Δ(ϝτϦοΫ|νΣοΫ)؂ࢹͰදݱ • ϩάΛ૸ࠪͯ͠ύλʔϯϚον͢Δ • LTSV ܗࣜͷΞΫηεϩάΛूܭ͢Δ ϩάΛ؂ࢹ͢Δ
43. ऩूͷλʔήοτ • ΤʔδΣϯτܥ • αʔόʔ • ίϯςφ
 • αʔϏεϝτϦοΫ • URL ֎ܗ؂ࢹ • ΠϯςάϨʔγϣϯ
44. • αʔόʔ্Ͱಈ͘ΤʔδΣϯτ • ΤʔδΣϯτ͔Β Mackerel ʹ POST ͢Δ • γεςϜϝτϦοΫ͸ࣗಈͰऩू • CPU, memory, interface… • ϓϥάΠϯͰऔಘ৘ใΛ֦ுͰ͖Δ αʔόʔ͔Βͷऩू
45. Server Mackerel mackerel-agent Fluentd Golang app OS stats monitor-agent Plugins HTTP gostats POST ྫ
46. • ECS 1 Task ΍ k8s 1 pod Λ1ϗετ • αΠυΧʔίϯςφͱͯ͠ΤʔδΣϯτΛಋೖ • γεςϜϝτϦοΫΛࣗಈͰऩू • ϓϥάΠϯͰ֦ுՄೳ ίϯςφ͔Βͷऩू
47. Task Task Task Mackerel Agent Container Fluentd container Golang app container monitor-agent Plugins HTTP gostats POST Orchestration Metadata ྫ
48. • ಛఆͷϗετʹඥ෇͔ͳ͍ϝτϦοΫ • Ϗδωε KPI ɺαʔϏεશମͷϨΠςϯγ • API Ͱࣗ༝ʹ౤ߘ • e.g. fluentd αʔϏεϝτϦοΫ Service:Bookmark Role:App Host
49. • Mackerel ଆ͔Β URL ʹ઀ଓ • ઀ଓՄ൱΍ϨεϙϯελΠϜͳͲΛऩू URL ֎ܗ؂ࢹ
50. • Ϋϥ΢υࣄۀऀͷϞχλϦϯά API ͔Βऔಘ • ΤʔδΣϯτͳ͠ͰऩूͰ͖Δ • ऩूϝτϦοΫ͸Ϋϥ΢υࣄۀऀґଘ AWS / Azure ΠϯςάϨʔγϣϯ Mackerel User Cloud Services Crawl
51. ऩू͢Δ಺༰
52. • γεςϜϝτϦοΫ͸ࣗಈͰऩू • ϛυϧ΢ΣΞ͸ެࣜϓϥάΠϯͰऩू • ΞϓϦέʔγϣϯͷϝτϦοΫ͸ʁʁ
53. • γεςϜϝτϦοΫ͸ࣗಈͰऩू • ϛυϧ΢ΣΞ͸ެࣜϓϥάΠϯͰऩू • ΞϓϦέʔγϣϯͷϝτϦοΫ͸ʁʁ
54. • ݴޠ΍ϑϨʔϜϫʔΫ޲͚ͷϓϥάΠϯ͸͋Δ • JVM ΍ Go ͸ GC ౷ܭ౳΋؍ଌͰ͖Δ • ଍Γͳ͍ϝτϦοΫͷ؍ଌखஈ͕ඞཁ
55. Observability Observability Observability Observation Observation Observation Observe impl. Observe impl. Observe Impl. • ؍ଌ͢ΔͨΊʹ͸؍ଌՄೳʹ͢Δඞཁ͕͋Δ • Մ؍ଌੑ͸؂ࢹͷલఏͱͳΔ • ʢͷͰɺ؂ࢹͷจ຺ͰՄ؍ଌੑʹݴٴ͢Δʣ ࠶ܝ • ؍ଌ͢ΔͨΊʹ͸؍ଌՄೳʹ͢Δඞཁ͕͋Δ
56. ؍ଌ͍ͨ͠ϝτϦοΫ͸ԿΒ͔ͷखஈͰ؍ଌՄೳͰ͔͢ʁ ؍ଌՄೳʹ͢Δ ΤʔδΣϯτʢͷϓϥάΠϯʣͰ؍ଌɾऩू͢Δ NO YES
57. ؍ଌ͍ͨ͠ϝτϦοΫ͸ԿΒ͔ͷखஈͰ؍ଌՄೳͰ͔͢ʁ ؍ଌՄೳʹ͢Δ ΤʔδΣϯτʢͷϓϥάΠϯʣͰ؍ଌɾऩू͢Δ NO YES
58. • ͲͷΑ͏ʹ؍ଌ͞ΕΔ͔Ͱ࣮૷͕ܾ·Δ ؍ଌՄೳʹ͢Δ
59. ؍ଌ͍ͨ͠ϝτϦοΫ͸ԿΒ͔ͷखஈͰ؍ଌՄೳͰ͔͢ʁ ؍ଌՄೳʹ͢Δ ΤʔδΣϯτʢͷϓϥάΠϯʣͰ؍ଌɾऩू͢Δ NO YES
60. • ϓϥάΠϯΛࣗ࡞ͯ͠೚ҙͷखஈͰ؍ଌ • HTTP Ͱ JSON ܗࣜΛಡΈࠐΜͰ؍ଌ ؍ଌखஈ
61. • ϓϥάΠϯ = ࣮ߦϑΝΠϧ • ॴఆͷग़ྗͰϝτϦοΫ໊ͱ஋Λग़ྗ͢Δ͚ͩ • ࣮ߦ؀ڥ಺Ͱ“ݟ͑Δ”஋ͳΒԿͰ΋औΕΔ ࣗ࡞ϓϥάΠϯͰͷϝτϦοΫऩू $ mackerel-plugin-uptime uptime.seconds 7749392 1568662368
62. • ࣮૷ࣗମ͸ Bash Ͱ΋ Go Ͱ΋ԿͰ΋ • ग़ྗϑΥʔϚοτ͕ૉ๿ͳͷͰ࣮૷Λબ͹ͳ͍ • ؍ଌର৅ʹ͋Θ࣮ͤͯ૷͢Δඞཁ͕͋Δ • ؍ଌର৅ଆͷࣗ༝౓͸ߴ͘ͳΔ ࣗ࡞ϓϥάΠϯͰͷϝτϦοΫऩू
63. HTTP ܦ༝Ͱͷऩू • mackerel-plugin-json • ؍ଌ͍ͨ͠஋Λ JSON Ͱ HTTP ഑৴͢Δ • ϓϥάΠϯ͔ΒͦͷΤϯυϙΠϯτΛಡΈࠐΉ
64. ૝ఆ໰౴ • Q: Prometheus Exporter Έ͍ͨͳʁ • => ํ޲ੑ͸ yesʢ΋ͬͱγϯϓϧʣ • Q: Prometheus Exporter ࢖͑ͳ͍ͷʁ • => ……!
65. HTTP ܦ༝Ͱͷऩू $ mackerel-plugin-json \ > -url=http://localhost/stats active3 1568666460 waiting 2 1568666460 $ curl -fsSL http://localhost/stats {“active”:3,”waiting”:2}
66. HTTP ܦ༝Ͱͷऩू
67. • ͲͪΒ͕ྑ͍͔͸৔߹ʹΑΔ • طʹHTTPΛ஻ΔͳΒHTTP͕ศར • طଘͷ؍ଌखஈ͕͋Ε͹࠶ར༻͍ͨ͠
68. ऩूͨ͠ϝτϦοΫͷར༻
69. • Mackerel Ͱͷओͳ࢖͍ಓ • ؂ࢹʢΞϥʔτʣ • ՄࢹԽʢάϥϑɾμογϡϘʔυʣ
70. ؂ࢹʢΞϥʔτʣ
71. ;ͨͭͷΞϥʔτ • ଈ࣌ରԠΛଅ͢Ξϥʔτ • ӡ༻ऀͱͯ͠͸θϩʹ͍ͨ͠ʂ • Information ͱͯ͠ͷΞϥʔτ
72. ଈ࣌ରԠͷΞϥʔτ (on-call) • γεςϜͷՔಇ͕ଛͳΘΕ͍ͯΔ • ରԠΛඞཁͱ͢Δ • “௚Δ·Ͱ์ஔ”͕ڐ༰ͳΒଈ࣌ରԠͰ͸ͳ͍
73. Information ͱͯ͠ͷΞϥʔτ • ٸ͕ͳ͍͕ԿΒ͔ͷΞΫγϣϯΛٻΊΔ • ྫ: ৑௕Խϗετ͕1୆μ΢ϯ • γεςϜ͕ଓߦՄೳͳൣғͷʮҟৗʯ • ಉ͡ࢦඪ͕͞ΒʹѱԽ͢Ε͹ଈ࣌ରԠ
74. • ࣮ࡍʹγεςϜ͕Քಇ͍ͯ͠Δ͔
 Ͳ͏͔Λ؂ࢹ͢Δ • e.g. ֎ܗ؂ࢹ, ΤϥʔϨʔτ • ˱ CPU 100%Ͱ΋ಈ͍ͯΕ͹ྑ͍ ଈ࣌ରԠͷΞϥʔτ
75. • ͱ͸͍͑৔߹ʹΑΓ͚Γ • ௚઀ͷՔಇͷ؂ࢹͰͳͯ͘΋ɺ
 ͔ͦ͜ΒՔಇঢ়گ͕ԋ៷ՄೳͳΒ༗༻
76. Information ͱͯ͠ͷΞϥʔτ • γεςϜՔಇҎ֎ͷࢦඪ΋औΓಘΔ • ଈ࣌ରԠͱಉ͡௨஌ܦ࿏ʹࡌͤͳ͍
77. Ξϥʔτશൠ • ͳΜͰ΋͔ΜͰ΋Ξϥʔτʹ͠ͳ͍ • ຊ౰ʹඞཁͳΞϥʔτ͕ຒ΋ΕΔ • Information ͸ͩΒͩΒ૿͕͑ͪ
78. ՄࢹԽ
79. • ಺෦ͷঢ়ଶΛ࣌ܥྻ͖ͭͰه࿥͢Δ • खಈͰऩू͠ͳͯ͘΋؍ଌͰ͖Δ • ݱࡏͷঢ়ଶΛ೺Ѳ͢Δखஈ • ఆৗ࣌ɾো֐࣌໰Θͣ • աڈͷঢ়ଶΛৼΓฦΔखஈ άϥϑɾμογϡϘʔυ
80. ؍ଌΛҭͯΔ
81. ඞཁͳ؍ଌྔ͕࠷ॳ͔Βἧ͏ͱ͸ݶΒͳ͍
82. • ։ൃɾݕূதʹؾͮ͘ • ؂ࢹ༻ҙ͠Α͏ͱͯ͠ෆ଍ʹؾͮ͘ • ো֐ى͖ͯΈͯෆ଍ʹؾͮ͘ ͨͱ͑͹…
83. ඞཁͳ؍ଌ΋࠷ॳ͔Βἧ͏ͱ͸ݶΒͳ͍
84. • ։ൃɾݕূதʹؾͮ͘ • ෛՙࢼݧͰؾͮ͘ͱඒ͍͠ʂ • ো֐ى͖ͯΈͯෆ଍ʹؾͮ͘ ͨͱ͑͹…
85. αΠΫϧΛճ͍ͯ͘͠
86. • ໪࿦࠷ॳ͔Β੔උͰ͖Ε͹ྑ͍ • ಛʹඞਢͷ߲໨͸཈͍͑ͯ͘ • γεςϜͷՔಇͷ؂ࢹʂʂ • ܧଓతʹมߋɾվળ͍ͯ͘͠
88. Mackerel Ͱ Mackerel Λ؍ଌ͢Δ
89. Mackerel ͷٕज़ελοΫ • AWS ্ʹߏங • EC2 ӡ༻ͱ ECS Fargate ͕ࠞࡏ • ϝΠϯͷ JVM ͳͲ͕ EC2 • গͣͭ͠ίϯςφϫʔΫϩʔυʹҠߦத
90. ୤ઢ
91. ίϯςφศར
92. • Πϯελϯεͷੈ࿩͕ཁΒͳ͍ • Πϯϑϥ·ͰؔΘΔߏ੒มߋ͕खܰ • “chef ߋ৽ͯ͠Πϯελϯε࡞Γ௚ͯ͠ೖΕ ׵͑ͯ…” • ʮخ͘͠ͳ͍͜ͱʯ͡ΌΜ……
93. ୤ઢ͜͜·Ͱ
94. Mackerel ͷٕज़ελοΫ ࣌ܥྻσʔλϕʔε: ϚωʔδυαʔϏεϕʔε
95. ୤ઢ
96. αʔόϨεΞʔΩςΫνϟศར
97. • ૄ݁߹ɾϦτϥΠՄೳͳઃܭ • ϝοηʔδΩϡʔΛڬΉ • Lambda ͷॲཧ಺༰Λႈ౳ʹઃܭ • ύϥϝʔλมߋͰεέʔϧ͢Δ • ʢ໪࿦ FaaS ಛ༗ͷӡ༻՝୊͸θϩͰ͸ͳ͍ʣ
98. ୤ઢ͜͜·Ͱ
99. Mackerel ͷٕज़ελοΫ ʢڵຯ͕͋Ε͹աڈͷొஃࢿྉΛ͝ཡԼ͍͞ʂʣ
100. JVM ΞϓϦέʔγϣϯͷ؍ଌ • ϝτϦΫεɾ؂ࢹ: mackerel • jstat ͱ jolokia ʹΑΔJVM಺෦ߏ଄ͷ؍ଌ • αʔόʔ͝ͱͷϨΠςϯγ؂ࢹ • ίωΫγϣϯϓʔϧ؂ࢹ • etc…
101. JVM ΞϓϦέʔγϣϯͷ؍ଌ • APM: NewRelic APM • ΑΓਫ਼៛ͳ಺෦෼ੳ͕ඞཁͳͱ͖ʹ • ϩάղੳ: fluentd => Kibana • ΞΫηεϩάɺΞϓϦέʔγϣϯϩά
102. ։ൃ؀ڥͷ؍ଌ • ։ൃ؀ڥʢ݉εςʔδϯά؀ڥʣ • ຊ൪ͱಉ༷ͷߏ੒ʢن໛͸খ͍͞ʣ • ಉ༷ͷ؍ଌπʔϧߏ੒ • => ϦϦʔεલ͔Β։ൃ؀ڥͰ؍ଌ
103. ૝ఆ໰౴ • Q: ։ൃ؀ڥͷ؂ࢹͷίετ͸ʁ • => զʑ͸MackerelΛແྉͰ࢖͑Δʢ൓ଇʣ
104. • த֩ʹ Mackerel • ීஈͷʮ؂ࢹʯʮ؅ཧʯͷى఺͸͜͜ • ଞͷπʔϧͱ૊Έ߹Θͤͯ؍ଌΛิڧ
106. ͝ਗ਼ௌ͋Γ͕ͱ͏͍͟͝·ͨ͠