Observability: Mackerel による観測と Mackerel の観測 / NoOps Meetup Tokyo #8

Transcript

1. Observability: Mackerel ʹΑΔ؍ଌ ͱ Mackerel ͷ؍ଌ 2019/09/17 NoOps Meetup Tokyo

   #8 Asato WAKISAKA (Hatena)

2. Asato WAKISAKA GitHub: @astj / Twitter @ast_j ͸ͯͳ ΞϓϦέʔγϣϯΤϯδχΞ Mackerel

   ։ൃνʔϜ (2016/08~) Mackerel ςοΫϦʔυ (2018/05~)

3. कඋൣғ: ϑϩϯτΤϯυ͔ΒSREing·Ͱʢࣗশʣ Scala, Golang, TypeScript, …

4. Mackerel

5. None

6. • Mackerel ஌ͬͯΔਓ • ͋Γ͕ͱ͏͍͟͝·͢ʂ • Mackerel ৮ͬͨ͜ͱ͕͋Δਓ • ͋Γ͕ͱ͏͍͟͝·͢ʂʂ

   • Mackerel ࢖ͬͯΔਓ • ͋Γ͕ͱ͏͍͟͝·͢ʂʂʂʂ Ξϯέʔτ
7. None

8. ؂ࢹ

9. Monitoring

10. Observability

11. ?

12. Observability ͱ Monitoring

13. • Observability: ʮՄ؍ଌੑʯ • γεςϜͷ಺෦ঢ়ଶ͕औಘͰ͖Δঢ়ଶ • 3ͭͷओཁͳߏ੒ཁૉ • Metrics, Logging,

    Tracing • Mackerel ͸͜ͷ͏ͪ Metrics த৺

14. • ʮՄ؍ଌੑʯ≠ʮ؍ଌʯ • ʮՄ؍ଌੑʯ͸γεςϜ͕࣋ͭੑ࣭ • ʮ؍ଌʯ͸γεςϜ֎෦ͷߦಈ

15. Metrics Observability Log Observability Trace Observability Metrics Observation Log Observation

    Trace Observation

16. Metrics Observability Log Observability Trace Observability Metrics Observation Log Observation

    Trace Observation • Server Metrics • Middleware Metrics • Application Metrics • Business KPI • Etc… • Alerting • Dashboard Visualization • Analysis • Etc…

17. Metrics Observability Log Observability Trace Observability Metrics Observation Log Observation

    Trace Observation ʮϝτϦοΫ؂ࢹʯ͸͜ͷลʁ • Server Metrics • Middleware Metrics • Application Metrics • Business KPI • Etc… • Alerting • Dashboard Visualization • Analysis • Etc…

18. • ͱ͸͍͑ɺͦͷઌͰ͸ਓ͕ؒ؍ଌ͍ͯ͠Δ͸ͣ • ݹయతʹ͸ΞϥʔτରԠͳͲ • ʮάϥϑΛݟΔʯ΋؍ଌ

19. Metrics Observability Log Observability Trace Observability Metrics Observation Log Observation

    Trace Observation wਓؒ • ͋Ε΍ • ͜Ε΍ ˞͜Ε͸ۃ୺

20. • લஈʹ͸ʮՄ؍ଌੑʯͦͷ΋ͷ • ޙஈͷʮ؍ଌʯΛ;ͨͭʹ෼͚ͯଊ͑Δ • ʮ؍ଌͷ࣮૷ʯ • ʮʢਓؒͷʣ؍ଌʯ

21. Observability Observability Observability Observation Observation Observation Observe impl. Observe impl.

    Observe Impl.

22. Observability Observability Observability Observation Observation Observation Observe impl. Observe impl.

    Observe Impl. ٕज़ͱͯ͠ͷ Observability ͕
 ͜ͷลΓΛࢦ͢͜ͱ΋͋Ε͹

23. Observability Observability Observability Observation Observation Observation Observe impl. Observe impl.

    Observe Impl. ιϦϡʔγϣϯͷ࿩୊Ͱ Observability ͕ݴٴ͞ΕΔ࣌͸
 ͜͜·Ͱؚ·ΕΔ͜ͱ΋ଟ͍

24. ……Ͱɺ͍͍Ͱ͔͢Ͷʁ

25. • ٞ࿦ͷ༨஍͸͍Ζ͍Ζ͋Γͦ͏ • ͔͠͠ࠓ೔͸͜ͷ࿩ͷਂ௥͍͸͠·ͤΜ • ౎߹্ࠨ୺ΛՄ؍ଌੑͱͯ͠࿩͠·͢

26. Observability Observability Observability Observation Observation Observation Observe impl. Observe impl.

    Observe Impl. • ϝτϦοΫ؂ࢹ͸: • ϝτϦοΫΛऩू͢Δ • ऩूͨ͠ϝτϦοΫΛूੵ͢Δ • άϥϑ΍μογϡϘʔυͰՄࢹԽ͢Δ • ΞϥʔτΛൃใ͢Δ

27. Observability Observability Observability Observation Observation Observation Observe impl. Observe impl.

    Observe Impl. ϝτϦοΫऩू ϝτϦοΫूੵ • άϥϑ • Ξϥʔτ • μογϡϘʔυ ˞Πϝʔδ • ਓؒ • ࣗಈԽରԠ

28. Observability Observability Observability Observation Observation Observation Observe impl. Observe impl.

    Observe Impl. • ؍ଌ͢ΔͨΊʹ͸؍ଌՄೳʹ͢Δඞཁ͕͋Δ • Մ؍ଌੑ͸؂ࢹͷલఏͱͳΔ • ʢͷͰɺ؂ࢹͷจ຺ͰՄ؍ଌੑʹݴٴ͞ΕΔʣ
29. None

30. Mackerel ͷ֓೦

31. ϗετ

32. • ؂ࢹɾ؅ཧͷى఺ʹͳΔ֓೦ • యܕతʹ͸αʔόʔ • ʮαʔόʔͷΑ͏ͳ΋ͷʯ΋ϗετ • ίϯςφͷू߹ (e.g. ECS

    Task / k8s pod) • Ϋϥ΢υϦιʔε (e.g. Lambda Function) ϗετ

33. Service:Blog Role:App Role:DB Host A Host B Host C Host

    D Service:Bookmark Role:App Host E Host F … αʔϏεɾϩʔϧ

34. • ϗετΛʮ໾ׂʯͰ෼ྨ • μογϡϘʔυ্Ͱάϧʔϐϯά͞ΕΔ • αʔϏε΍ϩʔϧ୯ҐͰ؂ࢹϧʔϧΛ؅ཧ αʔϏεɾϩʔϧ

35. None
36. None
37. None

38. ಺෦ঢ়ଶͷऩू

39. • ࣌ܥྻͰࢀরՄೳ ϝτϦοΫ

40. • ୯७ͳ੒൱ͱϝοηʔδΛه࿥ • ฏৗ࣌ͷཤྺ͸ه࿥ʹ࢒Βͳ͍ νΣοΫ؂ࢹ

41. • ϩάʹର͢Δ(ϝτϦοΫ|νΣοΫ)؂ࢹͰදݱ • ϩάΛ૸ࠪͯ͠ύλʔϯϚον͢Δ • LTSV ܗࣜͷΞΫηεϩάΛूܭ͢Δ ϩάΛ؂ࢹ͢Δ

42. None

43. ऩूͷλʔήοτ • ΤʔδΣϯτܥ • αʔόʔ • ίϯςφ
 • αʔϏεϝτϦοΫ •

    URL ֎ܗ؂ࢹ • ΠϯςάϨʔγϣϯ

44. • αʔόʔ্Ͱಈ͘ΤʔδΣϯτ • ΤʔδΣϯτ͔Β Mackerel ʹ POST ͢Δ • γεςϜϝτϦοΫ͸ࣗಈͰऩू

    • CPU, memory, interface… • ϓϥάΠϯͰऔಘ৘ใΛ֦ுͰ͖Δ αʔόʔ͔Βͷऩू

45. Server Mackerel mackerel-agent Fluentd Golang app OS stats monitor-agent Plugins

    HTTP gostats POST ྫ

46. • ECS 1 Task ΍ k8s 1 pod Λ1ϗετ •

    αΠυΧʔίϯςφͱͯ͠ΤʔδΣϯτΛಋೖ • γεςϜϝτϦοΫΛࣗಈͰऩू • ϓϥάΠϯͰ֦ுՄೳ ίϯςφ͔Βͷऩू

47. Task Task Task Mackerel Agent Container Fluentd container Golang app

    container monitor-agent Plugins HTTP gostats POST Orchestration Metadata ྫ

48. • ಛఆͷϗετʹඥ෇͔ͳ͍ϝτϦοΫ • Ϗδωε KPI ɺαʔϏεશମͷϨΠςϯγ • API Ͱࣗ༝ʹ౤ߘ •

    e.g. fluentd αʔϏεϝτϦοΫ Service:Bookmark Role:App Host

49. • Mackerel ଆ͔Β URL ʹ઀ଓ • ઀ଓՄ൱΍ϨεϙϯελΠϜͳͲΛऩू URL ֎ܗ؂ࢹ

50. • Ϋϥ΢υࣄۀऀͷϞχλϦϯά API ͔Βऔಘ • ΤʔδΣϯτͳ͠ͰऩूͰ͖Δ • ऩूϝτϦοΫ͸Ϋϥ΢υࣄۀऀґଘ AWS /

    Azure ΠϯςάϨʔγϣϯ Mackerel User Cloud Services Crawl

51. ऩू͢Δ಺༰

52. • γεςϜϝτϦοΫ͸ࣗಈͰऩू • ϛυϧ΢ΣΞ͸ެࣜϓϥάΠϯͰऩू • ΞϓϦέʔγϣϯͷϝτϦοΫ͸ʁʁ

53. • γεςϜϝτϦοΫ͸ࣗಈͰऩू • ϛυϧ΢ΣΞ͸ެࣜϓϥάΠϯͰऩू • ΞϓϦέʔγϣϯͷϝτϦοΫ͸ʁʁ

54. • ݴޠ΍ϑϨʔϜϫʔΫ޲͚ͷϓϥάΠϯ͸͋Δ • JVM ΍ Go ͸ GC ౷ܭ౳΋؍ଌͰ͖Δ •

    ଍Γͳ͍ϝτϦοΫͷ؍ଌखஈ͕ඞཁ

55. Observability Observability Observability Observation Observation Observation Observe impl. Observe impl.

    Observe Impl. • ؍ଌ͢ΔͨΊʹ͸؍ଌՄೳʹ͢Δඞཁ͕͋Δ • Մ؍ଌੑ͸؂ࢹͷલఏͱͳΔ • ʢͷͰɺ؂ࢹͷจ຺ͰՄ؍ଌੑʹݴٴ͢Δʣ ࠶ܝ • ؍ଌ͢ΔͨΊʹ͸؍ଌՄೳʹ͢Δඞཁ͕͋Δ

56. ؍ଌ͍ͨ͠ϝτϦοΫ͸ԿΒ͔ͷखஈͰ؍ଌՄೳͰ͔͢ʁ ؍ଌՄೳʹ͢Δ ΤʔδΣϯτʢͷϓϥάΠϯʣͰ؍ଌɾऩू͢Δ NO YES

57. ؍ଌ͍ͨ͠ϝτϦοΫ͸ԿΒ͔ͷखஈͰ؍ଌՄೳͰ͔͢ʁ ؍ଌՄೳʹ͢Δ ΤʔδΣϯτʢͷϓϥάΠϯʣͰ؍ଌɾऩू͢Δ NO YES

58. • ͲͷΑ͏ʹ؍ଌ͞ΕΔ͔Ͱ࣮૷͕ܾ·Δ ؍ଌՄೳʹ͢Δ

59. ؍ଌ͍ͨ͠ϝτϦοΫ͸ԿΒ͔ͷखஈͰ؍ଌՄೳͰ͔͢ʁ ؍ଌՄೳʹ͢Δ ΤʔδΣϯτʢͷϓϥάΠϯʣͰ؍ଌɾऩू͢Δ NO YES

60. • ϓϥάΠϯΛࣗ࡞ͯ͠೚ҙͷखஈͰ؍ଌ • HTTP Ͱ JSON ܗࣜΛಡΈࠐΜͰ؍ଌ ؍ଌखஈ

61. • ϓϥάΠϯ = ࣮ߦϑΝΠϧ • ॴఆͷग़ྗͰϝτϦοΫ໊ͱ஋Λग़ྗ͢Δ͚ͩ • ࣮ߦ؀ڥ಺Ͱ“ݟ͑Δ”஋ͳΒԿͰ΋औΕΔ ࣗ࡞ϓϥάΠϯͰͷϝτϦοΫऩू $

    mackerel-plugin-uptime uptime.seconds 7749392 1568662368

62. • ࣮૷ࣗମ͸ Bash Ͱ΋ Go Ͱ΋ԿͰ΋ • ग़ྗϑΥʔϚοτ͕ૉ๿ͳͷͰ࣮૷Λબ͹ͳ͍ • ؍ଌର৅ʹ͋Θ࣮ͤͯ૷͢Δඞཁ͕͋Δ

    • ؍ଌର৅ଆͷࣗ༝౓͸ߴ͘ͳΔ ࣗ࡞ϓϥάΠϯͰͷϝτϦοΫऩू

63. HTTP ܦ༝Ͱͷऩू • mackerel-plugin-json • ؍ଌ͍ͨ͠஋Λ JSON Ͱ HTTP ഑৴͢Δ

    • ϓϥάΠϯ͔ΒͦͷΤϯυϙΠϯτΛಡΈࠐΉ

64. ૝ఆ໰౴ • Q: Prometheus Exporter Έ͍ͨͳʁ • => ํ޲ੑ͸ yesʢ΋ͬͱγϯϓϧʣ

    • Q: Prometheus Exporter ࢖͑ͳ͍ͷʁ • => ……!

65. HTTP ܦ༝Ͱͷऩू $ mackerel-plugin-json \ > -url=http://localhost/stats active3 1568666460 waiting

    2 1568666460 $ curl -fsSL http://localhost/stats {“active”:3,”waiting”:2}

66. HTTP ܦ༝Ͱͷऩू

67. • ͲͪΒ͕ྑ͍͔͸৔߹ʹΑΔ • طʹHTTPΛ஻ΔͳΒHTTP͕ศར • طଘͷ؍ଌखஈ͕͋Ε͹࠶ར༻͍ͨ͠

68. ऩूͨ͠ϝτϦοΫͷར༻

69. • Mackerel Ͱͷओͳ࢖͍ಓ • ؂ࢹʢΞϥʔτʣ • ՄࢹԽʢάϥϑɾμογϡϘʔυʣ

70. ؂ࢹʢΞϥʔτʣ

71. ;ͨͭͷΞϥʔτ • ଈ࣌ରԠΛଅ͢Ξϥʔτ • ӡ༻ऀͱͯ͠͸θϩʹ͍ͨ͠ʂ • Information ͱͯ͠ͷΞϥʔτ

72. ଈ࣌ରԠͷΞϥʔτ (on-call) • γεςϜͷՔಇ͕ଛͳΘΕ͍ͯΔ • ରԠΛඞཁͱ͢Δ • “௚Δ·Ͱ์ஔ”͕ڐ༰ͳΒଈ࣌ରԠͰ͸ͳ͍

73. Information ͱͯ͠ͷΞϥʔτ • ٸ͕ͳ͍͕ԿΒ͔ͷΞΫγϣϯΛٻΊΔ • ྫ: ৑௕Խϗετ͕1୆μ΢ϯ • γεςϜ͕ଓߦՄೳͳൣғͷʮҟৗʯ •

    ಉ͡ࢦඪ͕͞ΒʹѱԽ͢Ε͹ଈ࣌ରԠ

74. • ࣮ࡍʹγεςϜ͕Քಇ͍ͯ͠Δ͔
 Ͳ͏͔Λ؂ࢹ͢Δ • e.g. ֎ܗ؂ࢹ, ΤϥʔϨʔτ • ˱ CPU

    100%Ͱ΋ಈ͍ͯΕ͹ྑ͍ ଈ࣌ରԠͷΞϥʔτ

75. • ͱ͸͍͑৔߹ʹΑΓ͚Γ • ௚઀ͷՔಇͷ؂ࢹͰͳͯ͘΋ɺ
 ͔ͦ͜ΒՔಇঢ়گ͕ԋ៷ՄೳͳΒ༗༻

76. Information ͱͯ͠ͷΞϥʔτ • γεςϜՔಇҎ֎ͷࢦඪ΋औΓಘΔ • ଈ࣌ରԠͱಉ͡௨஌ܦ࿏ʹࡌͤͳ͍

77. Ξϥʔτશൠ • ͳΜͰ΋͔ΜͰ΋Ξϥʔτʹ͠ͳ͍ • ຊ౰ʹඞཁͳΞϥʔτ͕ຒ΋ΕΔ • Information ͸ͩΒͩΒ૿͕͑ͪ

78. ՄࢹԽ

79. • ಺෦ͷঢ়ଶΛ࣌ܥྻ͖ͭͰه࿥͢Δ • खಈͰऩू͠ͳͯ͘΋؍ଌͰ͖Δ • ݱࡏͷঢ়ଶΛ೺Ѳ͢Δखஈ • ఆৗ࣌ɾো֐࣌໰Θͣ • աڈͷঢ়ଶΛৼΓฦΔखஈ

    άϥϑɾμογϡϘʔυ

80. ؍ଌΛҭͯΔ

81. ඞཁͳ؍ଌྔ͕࠷ॳ͔Βἧ͏ͱ͸ݶΒͳ͍

82. • ։ൃɾݕূதʹؾͮ͘ • ؂ࢹ༻ҙ͠Α͏ͱͯ͠ෆ଍ʹؾͮ͘ • ো֐ى͖ͯΈͯෆ଍ʹؾͮ͘ ͨͱ͑͹…

83. ඞཁͳ؍ଌ΋࠷ॳ͔Βἧ͏ͱ͸ݶΒͳ͍

84. • ։ൃɾݕূதʹؾͮ͘ • ෛՙࢼݧͰؾͮ͘ͱඒ͍͠ʂ • ো֐ى͖ͯΈͯෆ଍ʹؾͮ͘ ͨͱ͑͹…

85. αΠΫϧΛճ͍ͯ͘͠

86. • ໪࿦࠷ॳ͔Β੔උͰ͖Ε͹ྑ͍ • ಛʹඞਢͷ߲໨͸཈͍͑ͯ͘ • γεςϜͷՔಇͷ؂ࢹʂʂ • ܧଓతʹมߋɾվળ͍ͯ͘͠

87. None

88. Mackerel Ͱ Mackerel Λ؍ଌ͢Δ

89. Mackerel ͷٕज़ελοΫ • AWS ্ʹߏங • EC2 ӡ༻ͱ ECS Fargate

    ͕ࠞࡏ • ϝΠϯͷ JVM ͳͲ͕ EC2 • গͣͭ͠ίϯςφϫʔΫϩʔυʹҠߦத

90. ୤ઢ

91. ίϯςφศར

92. • Πϯελϯεͷੈ࿩͕ཁΒͳ͍ • Πϯϑϥ·ͰؔΘΔߏ੒มߋ͕खܰ • “chef ߋ৽ͯ͠Πϯελϯε࡞Γ௚ͯ͠ೖΕ ׵͑ͯ…” • ʮخ͘͠ͳ͍͜ͱʯ͡ΌΜ……

93. ୤ઢ͜͜·Ͱ

94. Mackerel ͷٕज़ελοΫ ࣌ܥྻσʔλϕʔε: ϚωʔδυαʔϏεϕʔε

95. ୤ઢ

96. αʔόϨεΞʔΩςΫνϟศར

97. • ૄ݁߹ɾϦτϥΠՄೳͳઃܭ • ϝοηʔδΩϡʔΛڬΉ • Lambda ͷॲཧ಺༰Λႈ౳ʹઃܭ • ύϥϝʔλมߋͰεέʔϧ͢Δ •

    ʢ໪࿦ FaaS ಛ༗ͷӡ༻՝୊͸θϩͰ͸ͳ͍ʣ

98. ୤ઢ͜͜·Ͱ

99. Mackerel ͷٕज़ελοΫ ʢڵຯ͕͋Ε͹աڈͷొஃࢿྉΛ͝ཡԼ͍͞ʂʣ

100. JVM ΞϓϦέʔγϣϯͷ؍ଌ • ϝτϦΫεɾ؂ࢹ: mackerel • jstat ͱ jolokia ʹΑΔJVM಺෦ߏ଄ͷ؍ଌ

     • αʔόʔ͝ͱͷϨΠςϯγ؂ࢹ • ίωΫγϣϯϓʔϧ؂ࢹ • etc…

101. JVM ΞϓϦέʔγϣϯͷ؍ଌ • APM: NewRelic APM • ΑΓਫ਼៛ͳ಺෦෼ੳ͕ඞཁͳͱ͖ʹ • ϩάղੳ:

     fluentd => Kibana • ΞΫηεϩάɺΞϓϦέʔγϣϯϩά

102. ։ൃ؀ڥͷ؍ଌ • ։ൃ؀ڥʢ݉εςʔδϯά؀ڥʣ • ຊ൪ͱಉ༷ͷߏ੒ʢن໛͸খ͍͞ʣ • ಉ༷ͷ؍ଌπʔϧߏ੒ • => ϦϦʔεલ͔Β։ൃ؀ڥͰ؍ଌ

103. ૝ఆ໰౴ • Q: ։ൃ؀ڥͷ؂ࢹͷίετ͸ʁ • => զʑ͸MackerelΛແྉͰ࢖͑Δʢ൓ଇʣ

104. • த֩ʹ Mackerel • ීஈͷʮ؂ࢹʯʮ؅ཧʯͷى఺͸͜͜ • ଞͷπʔϧͱ૊Έ߹Θͤͯ؍ଌΛิڧ

105. None

106. ͝ਗ਼ௌ͋Γ͕ͱ͏͍͟͝·ͨ͠