Observability: Mackerel による観測と Mackerel の観測 / NoOps Meetup Tokyo #8

81e8ac1d3a766b3d9880cd08d9a7aba4?s=47 astj
September 17, 2019

Observability: Mackerel による観測と Mackerel の観測 / NoOps Meetup Tokyo #8

81e8ac1d3a766b3d9880cd08d9a7aba4?s=128

astj

September 17, 2019
Tweet

Transcript

  1. Observability: Mackerel ʹΑΔ؍ଌ ͱ Mackerel ͷ؍ଌ 2019/09/17 NoOps Meetup Tokyo

    #8 Asato WAKISAKA (Hatena)
  2. Asato WAKISAKA GitHub: @astj / Twitter @ast_j ͸ͯͳ ΞϓϦέʔγϣϯΤϯδχΞ Mackerel

    ։ൃνʔϜ (2016/08~) Mackerel ςοΫϦʔυ (2018/05~)
  3. कඋൣғ: ϑϩϯτΤϯυ͔ΒSREing·Ͱʢࣗশʣ Scala, Golang, TypeScript, …

  4. Mackerel

  5. None
  6. • Mackerel ஌ͬͯΔਓ • ͋Γ͕ͱ͏͍͟͝·͢ʂ • Mackerel ৮ͬͨ͜ͱ͕͋Δਓ • ͋Γ͕ͱ͏͍͟͝·͢ʂʂ

    • Mackerel ࢖ͬͯΔਓ • ͋Γ͕ͱ͏͍͟͝·͢ʂʂʂʂ Ξϯέʔτ
  7. None
  8. ؂ࢹ

  9. Monitoring

  10. Observability

  11. ?

  12. Observability ͱ Monitoring

  13. • Observability: ʮՄ؍ଌੑʯ • γεςϜͷ಺෦ঢ়ଶ͕औಘͰ͖Δঢ়ଶ • 3ͭͷओཁͳߏ੒ཁૉ • Metrics, Logging,

    Tracing • Mackerel ͸͜ͷ͏ͪ Metrics த৺
  14. • ʮՄ؍ଌੑʯ≠ʮ؍ଌʯ • ʮՄ؍ଌੑʯ͸γεςϜ͕࣋ͭੑ࣭ • ʮ؍ଌʯ͸γεςϜ֎෦ͷߦಈ

  15. Metrics Observability Log Observability Trace Observability Metrics Observation Log Observation

    Trace Observation
  16. Metrics Observability Log Observability Trace Observability Metrics Observation Log Observation

    Trace Observation • Server Metrics • Middleware Metrics • Application Metrics • Business KPI • Etc… • Alerting • Dashboard Visualization • Analysis • Etc…
  17. Metrics Observability Log Observability Trace Observability Metrics Observation Log Observation

    Trace Observation ʮϝτϦοΫ؂ࢹʯ͸͜ͷลʁ • Server Metrics • Middleware Metrics • Application Metrics • Business KPI • Etc… • Alerting • Dashboard Visualization • Analysis • Etc…
  18. • ͱ͸͍͑ɺͦͷઌͰ͸ਓ͕ؒ؍ଌ͍ͯ͠Δ͸ͣ • ݹయతʹ͸ΞϥʔτରԠͳͲ • ʮάϥϑΛݟΔʯ΋؍ଌ

  19. Metrics Observability Log Observability Trace Observability Metrics Observation Log Observation

    Trace Observation wਓؒ • ͋Ε΍ • ͜Ε΍ ˞͜Ε͸ۃ୺
  20. • લஈʹ͸ʮՄ؍ଌੑʯͦͷ΋ͷ • ޙஈͷʮ؍ଌʯΛ;ͨͭʹ෼͚ͯଊ͑Δ • ʮ؍ଌͷ࣮૷ʯ • ʮʢਓؒͷʣ؍ଌʯ

  21. Observability Observability Observability Observation Observation Observation Observe impl. Observe impl.

    Observe Impl.
  22. Observability Observability Observability Observation Observation Observation Observe impl. Observe impl.

    Observe Impl. ٕज़ͱͯ͠ͷ Observability ͕
 ͜ͷลΓΛࢦ͢͜ͱ΋͋Ε͹
  23. Observability Observability Observability Observation Observation Observation Observe impl. Observe impl.

    Observe Impl. ιϦϡʔγϣϯͷ࿩୊Ͱ Observability ͕ݴٴ͞ΕΔ࣌͸
 ͜͜·Ͱؚ·ΕΔ͜ͱ΋ଟ͍
  24. ……Ͱɺ͍͍Ͱ͔͢Ͷʁ

  25. • ٞ࿦ͷ༨஍͸͍Ζ͍Ζ͋Γͦ͏ • ͔͠͠ࠓ೔͸͜ͷ࿩ͷਂ௥͍͸͠·ͤΜ • ౎߹্ࠨ୺ΛՄ؍ଌੑͱͯ͠࿩͠·͢

  26. Observability Observability Observability Observation Observation Observation Observe impl. Observe impl.

    Observe Impl. • ϝτϦοΫ؂ࢹ͸: • ϝτϦοΫΛऩू͢Δ • ऩूͨ͠ϝτϦοΫΛूੵ͢Δ • άϥϑ΍μογϡϘʔυͰՄࢹԽ͢Δ • ΞϥʔτΛൃใ͢Δ
  27. Observability Observability Observability Observation Observation Observation Observe impl. Observe impl.

    Observe Impl. ϝτϦοΫऩू ϝτϦοΫूੵ • άϥϑ • Ξϥʔτ • μογϡϘʔυ ˞Πϝʔδ • ਓؒ • ࣗಈԽରԠ
  28. Observability Observability Observability Observation Observation Observation Observe impl. Observe impl.

    Observe Impl. • ؍ଌ͢ΔͨΊʹ͸؍ଌՄೳʹ͢Δඞཁ͕͋Δ • Մ؍ଌੑ͸؂ࢹͷલఏͱͳΔ • ʢͷͰɺ؂ࢹͷจ຺ͰՄ؍ଌੑʹݴٴ͞ΕΔʣ
  29. None
  30. Mackerel ͷ֓೦

  31. ϗετ

  32. • ؂ࢹɾ؅ཧͷى఺ʹͳΔ֓೦ • యܕతʹ͸αʔόʔ • ʮαʔόʔͷΑ͏ͳ΋ͷʯ΋ϗετ • ίϯςφͷू߹ (e.g. ECS

    Task / k8s pod) • Ϋϥ΢υϦιʔε (e.g. Lambda Function) ϗετ
  33. Service:Blog Role:App Role:DB Host A Host B Host C Host

    D Service:Bookmark Role:App Host E Host F … αʔϏεɾϩʔϧ
  34. • ϗετΛʮ໾ׂʯͰ෼ྨ • μογϡϘʔυ্Ͱάϧʔϐϯά͞ΕΔ • αʔϏε΍ϩʔϧ୯ҐͰ؂ࢹϧʔϧΛ؅ཧ αʔϏεɾϩʔϧ

  35. None
  36. None
  37. None
  38. ಺෦ঢ়ଶͷऩू

  39. • ࣌ܥྻͰࢀরՄೳ ϝτϦοΫ

  40. • ୯७ͳ੒൱ͱϝοηʔδΛه࿥ • ฏৗ࣌ͷཤྺ͸ه࿥ʹ࢒Βͳ͍ νΣοΫ؂ࢹ

  41. • ϩάʹର͢Δ(ϝτϦοΫ|νΣοΫ)؂ࢹͰදݱ • ϩάΛ૸ࠪͯ͠ύλʔϯϚον͢Δ • LTSV ܗࣜͷΞΫηεϩάΛूܭ͢Δ ϩάΛ؂ࢹ͢Δ

  42. None
  43. ऩूͷλʔήοτ • ΤʔδΣϯτܥ • αʔόʔ • ίϯςφ
 • αʔϏεϝτϦοΫ •

    URL ֎ܗ؂ࢹ • ΠϯςάϨʔγϣϯ
  44. • αʔόʔ্Ͱಈ͘ΤʔδΣϯτ • ΤʔδΣϯτ͔Β Mackerel ʹ POST ͢Δ • γεςϜϝτϦοΫ͸ࣗಈͰऩू

    • CPU, memory, interface… • ϓϥάΠϯͰऔಘ৘ใΛ֦ுͰ͖Δ αʔόʔ͔Βͷऩू
  45. Server Mackerel mackerel-agent Fluentd Golang app OS stats monitor-agent Plugins

    HTTP gostats POST ྫ
  46. • ECS 1 Task ΍ k8s 1 pod Λ1ϗετ •

    αΠυΧʔίϯςφͱͯ͠ΤʔδΣϯτΛಋೖ • γεςϜϝτϦοΫΛࣗಈͰऩू • ϓϥάΠϯͰ֦ுՄೳ ίϯςφ͔Βͷऩू
  47. Task Task Task Mackerel Agent Container Fluentd container Golang app

    container monitor-agent Plugins HTTP gostats POST Orchestration Metadata ྫ
  48. • ಛఆͷϗετʹඥ෇͔ͳ͍ϝτϦοΫ • Ϗδωε KPI ɺαʔϏεશମͷϨΠςϯγ • API Ͱࣗ༝ʹ౤ߘ •

    e.g. fluentd αʔϏεϝτϦοΫ Service:Bookmark Role:App Host
  49. • Mackerel ଆ͔Β URL ʹ઀ଓ • ઀ଓՄ൱΍ϨεϙϯελΠϜͳͲΛऩू URL ֎ܗ؂ࢹ

  50. • Ϋϥ΢υࣄۀऀͷϞχλϦϯά API ͔Βऔಘ • ΤʔδΣϯτͳ͠ͰऩूͰ͖Δ • ऩूϝτϦοΫ͸Ϋϥ΢υࣄۀऀґଘ AWS /

    Azure ΠϯςάϨʔγϣϯ Mackerel User Cloud Services Crawl
  51. ऩू͢Δ಺༰

  52. • γεςϜϝτϦοΫ͸ࣗಈͰऩू • ϛυϧ΢ΣΞ͸ެࣜϓϥάΠϯͰऩू • ΞϓϦέʔγϣϯͷϝτϦοΫ͸ʁʁ

  53. • γεςϜϝτϦοΫ͸ࣗಈͰऩू • ϛυϧ΢ΣΞ͸ެࣜϓϥάΠϯͰऩू • ΞϓϦέʔγϣϯͷϝτϦοΫ͸ʁʁ

  54. • ݴޠ΍ϑϨʔϜϫʔΫ޲͚ͷϓϥάΠϯ͸͋Δ • JVM ΍ Go ͸ GC ౷ܭ౳΋؍ଌͰ͖Δ •

    ଍Γͳ͍ϝτϦοΫͷ؍ଌखஈ͕ඞཁ
  55. Observability Observability Observability Observation Observation Observation Observe impl. Observe impl.

    Observe Impl. • ؍ଌ͢ΔͨΊʹ͸؍ଌՄೳʹ͢Δඞཁ͕͋Δ • Մ؍ଌੑ͸؂ࢹͷલఏͱͳΔ • ʢͷͰɺ؂ࢹͷจ຺ͰՄ؍ଌੑʹݴٴ͢Δʣ ࠶ܝ • ؍ଌ͢ΔͨΊʹ͸؍ଌՄೳʹ͢Δඞཁ͕͋Δ
  56. ؍ଌ͍ͨ͠ϝτϦοΫ͸ԿΒ͔ͷखஈͰ؍ଌՄೳͰ͔͢ʁ ؍ଌՄೳʹ͢Δ ΤʔδΣϯτʢͷϓϥάΠϯʣͰ؍ଌɾऩू͢Δ NO YES

  57. ؍ଌ͍ͨ͠ϝτϦοΫ͸ԿΒ͔ͷखஈͰ؍ଌՄೳͰ͔͢ʁ ؍ଌՄೳʹ͢Δ ΤʔδΣϯτʢͷϓϥάΠϯʣͰ؍ଌɾऩू͢Δ NO YES

  58. • ͲͷΑ͏ʹ؍ଌ͞ΕΔ͔Ͱ࣮૷͕ܾ·Δ ؍ଌՄೳʹ͢Δ

  59. ؍ଌ͍ͨ͠ϝτϦοΫ͸ԿΒ͔ͷखஈͰ؍ଌՄೳͰ͔͢ʁ ؍ଌՄೳʹ͢Δ ΤʔδΣϯτʢͷϓϥάΠϯʣͰ؍ଌɾऩू͢Δ NO YES

  60. • ϓϥάΠϯΛࣗ࡞ͯ͠೚ҙͷखஈͰ؍ଌ • HTTP Ͱ JSON ܗࣜΛಡΈࠐΜͰ؍ଌ ؍ଌखஈ

  61. • ϓϥάΠϯ = ࣮ߦϑΝΠϧ • ॴఆͷग़ྗͰϝτϦοΫ໊ͱ஋Λग़ྗ͢Δ͚ͩ • ࣮ߦ؀ڥ಺Ͱ“ݟ͑Δ”஋ͳΒԿͰ΋औΕΔ ࣗ࡞ϓϥάΠϯͰͷϝτϦοΫऩू $

    mackerel-plugin-uptime uptime.seconds 7749392 1568662368
  62. • ࣮૷ࣗମ͸ Bash Ͱ΋ Go Ͱ΋ԿͰ΋ • ग़ྗϑΥʔϚοτ͕ૉ๿ͳͷͰ࣮૷Λબ͹ͳ͍ • ؍ଌର৅ʹ͋Θ࣮ͤͯ૷͢Δඞཁ͕͋Δ

    • ؍ଌର৅ଆͷࣗ༝౓͸ߴ͘ͳΔ ࣗ࡞ϓϥάΠϯͰͷϝτϦοΫऩू
  63. HTTP ܦ༝Ͱͷऩू • mackerel-plugin-json • ؍ଌ͍ͨ͠஋Λ JSON Ͱ HTTP ഑৴͢Δ

    • ϓϥάΠϯ͔ΒͦͷΤϯυϙΠϯτΛಡΈࠐΉ
  64. ૝ఆ໰౴ • Q: Prometheus Exporter Έ͍ͨͳʁ • => ํ޲ੑ͸ yesʢ΋ͬͱγϯϓϧʣ

    • Q: Prometheus Exporter ࢖͑ͳ͍ͷʁ • => ……!
  65. HTTP ܦ༝Ͱͷऩू $ mackerel-plugin-json \ > -url=http://localhost/stats active3 1568666460 waiting

    2 1568666460 $ curl -fsSL http://localhost/stats {“active”:3,”waiting”:2}
  66. HTTP ܦ༝Ͱͷऩू

  67. • ͲͪΒ͕ྑ͍͔͸৔߹ʹΑΔ • طʹHTTPΛ஻ΔͳΒHTTP͕ศར • طଘͷ؍ଌखஈ͕͋Ε͹࠶ར༻͍ͨ͠

  68. ऩूͨ͠ϝτϦοΫͷར༻

  69. • Mackerel Ͱͷओͳ࢖͍ಓ • ؂ࢹʢΞϥʔτʣ • ՄࢹԽʢάϥϑɾμογϡϘʔυʣ

  70. ؂ࢹʢΞϥʔτʣ

  71. ;ͨͭͷΞϥʔτ • ଈ࣌ରԠΛଅ͢Ξϥʔτ • ӡ༻ऀͱͯ͠͸θϩʹ͍ͨ͠ʂ • Information ͱͯ͠ͷΞϥʔτ

  72. ଈ࣌ରԠͷΞϥʔτ (on-call) • γεςϜͷՔಇ͕ଛͳΘΕ͍ͯΔ • ରԠΛඞཁͱ͢Δ • “௚Δ·Ͱ์ஔ”͕ڐ༰ͳΒଈ࣌ରԠͰ͸ͳ͍

  73. Information ͱͯ͠ͷΞϥʔτ • ٸ͕ͳ͍͕ԿΒ͔ͷΞΫγϣϯΛٻΊΔ • ྫ: ৑௕Խϗετ͕1୆μ΢ϯ • γεςϜ͕ଓߦՄೳͳൣғͷʮҟৗʯ •

    ಉ͡ࢦඪ͕͞ΒʹѱԽ͢Ε͹ଈ࣌ରԠ
  74. • ࣮ࡍʹγεςϜ͕Քಇ͍ͯ͠Δ͔
 Ͳ͏͔Λ؂ࢹ͢Δ • e.g. ֎ܗ؂ࢹ, ΤϥʔϨʔτ • ˱ CPU

    100%Ͱ΋ಈ͍ͯΕ͹ྑ͍ ଈ࣌ରԠͷΞϥʔτ
  75. • ͱ͸͍͑৔߹ʹΑΓ͚Γ • ௚઀ͷՔಇͷ؂ࢹͰͳͯ͘΋ɺ
 ͔ͦ͜ΒՔಇঢ়گ͕ԋ៷ՄೳͳΒ༗༻

  76. Information ͱͯ͠ͷΞϥʔτ • γεςϜՔಇҎ֎ͷࢦඪ΋औΓಘΔ • ଈ࣌ରԠͱಉ͡௨஌ܦ࿏ʹࡌͤͳ͍

  77. Ξϥʔτશൠ • ͳΜͰ΋͔ΜͰ΋Ξϥʔτʹ͠ͳ͍ • ຊ౰ʹඞཁͳΞϥʔτ͕ຒ΋ΕΔ • Information ͸ͩΒͩΒ૿͕͑ͪ

  78. ՄࢹԽ

  79. • ಺෦ͷঢ়ଶΛ࣌ܥྻ͖ͭͰه࿥͢Δ • खಈͰऩू͠ͳͯ͘΋؍ଌͰ͖Δ • ݱࡏͷঢ়ଶΛ೺Ѳ͢Δखஈ • ఆৗ࣌ɾো֐࣌໰Θͣ • աڈͷঢ়ଶΛৼΓฦΔखஈ

    άϥϑɾμογϡϘʔυ
  80. ؍ଌΛҭͯΔ

  81. ඞཁͳ؍ଌྔ͕࠷ॳ͔Βἧ͏ͱ͸ݶΒͳ͍

  82. • ։ൃɾݕূதʹؾͮ͘ • ؂ࢹ༻ҙ͠Α͏ͱͯ͠ෆ଍ʹؾͮ͘ • ো֐ى͖ͯΈͯෆ଍ʹؾͮ͘ ͨͱ͑͹…

  83. ඞཁͳ؍ଌ΋࠷ॳ͔Βἧ͏ͱ͸ݶΒͳ͍

  84. • ։ൃɾݕূதʹؾͮ͘ • ෛՙࢼݧͰؾͮ͘ͱඒ͍͠ʂ • ো֐ى͖ͯΈͯෆ଍ʹؾͮ͘ ͨͱ͑͹…

  85. αΠΫϧΛճ͍ͯ͘͠

  86. • ໪࿦࠷ॳ͔Β੔උͰ͖Ε͹ྑ͍ • ಛʹඞਢͷ߲໨͸཈͍͑ͯ͘ • γεςϜͷՔಇͷ؂ࢹʂʂ • ܧଓతʹมߋɾվળ͍ͯ͘͠

  87. None
  88. Mackerel Ͱ Mackerel Λ؍ଌ͢Δ

  89. Mackerel ͷٕज़ελοΫ • AWS ্ʹߏங • EC2 ӡ༻ͱ ECS Fargate

    ͕ࠞࡏ • ϝΠϯͷ JVM ͳͲ͕ EC2 • গͣͭ͠ίϯςφϫʔΫϩʔυʹҠߦத
  90. ୤ઢ

  91. ίϯςφศར

  92. • Πϯελϯεͷੈ࿩͕ཁΒͳ͍ • Πϯϑϥ·ͰؔΘΔߏ੒มߋ͕खܰ • “chef ߋ৽ͯ͠Πϯελϯε࡞Γ௚ͯ͠ೖΕ ׵͑ͯ…” • ʮخ͘͠ͳ͍͜ͱʯ͡ΌΜ……

  93. ୤ઢ͜͜·Ͱ

  94. Mackerel ͷٕज़ελοΫ ࣌ܥྻσʔλϕʔε: ϚωʔδυαʔϏεϕʔε

  95. ୤ઢ

  96. αʔόϨεΞʔΩςΫνϟศར

  97. • ૄ݁߹ɾϦτϥΠՄೳͳઃܭ • ϝοηʔδΩϡʔΛڬΉ • Lambda ͷॲཧ಺༰Λႈ౳ʹઃܭ • ύϥϝʔλมߋͰεέʔϧ͢Δ •

    ʢ໪࿦ FaaS ಛ༗ͷӡ༻՝୊͸θϩͰ͸ͳ͍ʣ
  98. ୤ઢ͜͜·Ͱ

  99. Mackerel ͷٕज़ελοΫ ʢڵຯ͕͋Ε͹աڈͷొஃࢿྉΛ͝ཡԼ͍͞ʂʣ

  100. JVM ΞϓϦέʔγϣϯͷ؍ଌ • ϝτϦΫεɾ؂ࢹ: mackerel • jstat ͱ jolokia ʹΑΔJVM಺෦ߏ଄ͷ؍ଌ

    • αʔόʔ͝ͱͷϨΠςϯγ؂ࢹ • ίωΫγϣϯϓʔϧ؂ࢹ • etc…
  101. JVM ΞϓϦέʔγϣϯͷ؍ଌ • APM: NewRelic APM • ΑΓਫ਼៛ͳ಺෦෼ੳ͕ඞཁͳͱ͖ʹ • ϩάղੳ:

    fluentd => Kibana • ΞΫηεϩάɺΞϓϦέʔγϣϯϩά
  102. ։ൃ؀ڥͷ؍ଌ • ։ൃ؀ڥʢ݉εςʔδϯά؀ڥʣ • ຊ൪ͱಉ༷ͷߏ੒ʢن໛͸খ͍͞ʣ • ಉ༷ͷ؍ଌπʔϧߏ੒ • => ϦϦʔεલ͔Β։ൃ؀ڥͰ؍ଌ

  103. ૝ఆ໰౴ • Q: ։ൃ؀ڥͷ؂ࢹͷίετ͸ʁ • => զʑ͸MackerelΛແྉͰ࢖͑Δʢ൓ଇʣ

  104. • த֩ʹ Mackerel • ීஈͷʮ؂ࢹʯʮ؅ཧʯͷى఺͸͜͜ • ଞͷπʔϧͱ૊Έ߹Θͤͯ؍ଌΛิڧ

  105. None
  106. ͝ਗ਼ௌ͋Γ͕ͱ͏͍͟͝·ͨ͠