KARTE を支えるマルチプラットフォームインフラ監視 /karte-multi-platform-monitoring

KARTE を支えるマルチプラットフォームインフラ監視 /karte-multi-platform-monitoring

【freee × プレイド】Tech Meetup 〜インフラ監視編〜

6de30acfd736745120a3d6457dc6d45c?s=128

Daiki Matsui

January 20, 2017
Tweet

Transcript

  1. KARTEΛࢧ͑Δ ϚϧνϓϥοτϑΥʔϜΠϯϑϥ؂ࢹ @ikemonn

  2. @ikemonn ΠϯϑϥΤϯδχΞ 2015೥11݄ʹjoin

  3. ໨࣍ 1. αʔϏεͷ঺հ 2. ؂ࢹͷ໨త 3. Πϯϑϥߏ੒ 4. ؂ࢹͷํ๏ 5.

    Ξϥʔτ 6. ϋϚͬͨ͜ͱ 7. ·ͱΊ
  4. ໨࣍ 1. αʔϏεͷ঺հ 2. ؂ࢹͷ໨త 3. Πϯϑϥߏ੒ 4. ؂ࢹͷํ๏ 5.

    Ξϥʔτ 6. ϋϚͬͨ͜ͱ 7. ·ͱΊ
  5. ΢Σϒ઀٬ϓϥοτϑΥʔϜ KARTEΛ։ൃ͍ͯ͠·͢

  6. ΢Σϒ઀٬ϓϥοτϑΥʔϜ KARTEΛ։ൃ͍ͯ͠·͢

  7. ΠϯϑϥΤϯδχΞ͸2ਓ

  8. ໨࣍ 1. αʔϏεͷ঺հ 2. ؂ࢹͷ໨త 3. Πϯϑϥߏ੒ 4. ؂ࢹͷํ๏ 5.

    Ξϥʔτ 6. ϋϚͬͨ͜ͱ 7. ·ͱΊ
  9. ʮશһͰյ͠ͳ͕ΒਐΉʯ ͱ͍͏จԽΛࢧ͑Δ

  10. Startup͸։ൃεϐʔυ͕ॏཁ

  11. ৽͍͠αʔϏεɺݴޠΛੵۃతʹ ಋೖͰ͖ΔΑ͏ʹ

  12. ރΕٕͨज़Ͱ͸ͳ͍ͷͰɺ ৘ใ͕͋·Γݟ͔ͭΒͳ͍

  13. ଟগͷࣦഊ͸ڐ༰ & ࣦഊͨ࣌͠ʹ͙͢ؾ෇͚Δ

  14. ʮશһͰյ͠ͳ͕ΒਐΉʯจԽΛ ࢧ͑ΔͨΊʹ؂ࢹͷڧԽ !

  15. ໨࣍ 1. αʔϏεͷ঺հ 2. ؂ࢹͷ໨త 3. Πϯϑϥߏ੒ 4. ؂ࢹͷํ๏ 5.

    Ξϥʔτ 6. ϋϚͬͨ͜ͱ 7. ·ͱΊ
  16. ϚϧνϓϥοτϑΥʔϜ ৽͍͠αʔϏεͷੵۃར༻

  17. None
  18. ໨࣍ 1. αʔϏεͷ঺հ 2. ؂ࢹͷ໨త 3. Πϯϑϥߏ੒ 4. ؂ࢹͷํ๏ 5.

    Ξϥʔτ 6. ϋϚͬͨ͜ͱ 7. ·ͱΊ
  19. ؂ࢹํ๏

  20. ผϓϥοτϑΥʔϜͷΠϯελϯε ಉ࢜ͷϝτϦοΫ΋ൺֱ͍ͨ͠ ֤ϓϥοτϑΥʔϜʹґଘͨ͠ϝτ ϦοΫૹ৴ͷίʔυΛॻ͖ͨ͘ͳ͍ ▪Πϯϑϥଆͷ՝୊ ▪ΞϓϦଆͷ՝୊

  21. CloudWatch ৭ʑͳ؅ཧը໘Λݟͨ͘ͳ͍ http://www.irasutoya.com/2016/06/blog-post_147.html https://support.draw.io/pages/viewpage.action?pageId=1671326 https://en.wikipedia.org/wiki/File:Google_Stackdriver_logo.svg https://www.runscope.com/

  22. جຊతʹDatadogͰҰݩ؅ཧ

  23. ԿΛ؂ࢹ͢Δͷ͔ʁ

  24. σʔλͷૹ৴ݩ σʔλͷૹ৴ઌ ௨஌ઌ CloudWatch Datadog Agent library agent https://www.datadoghq.com/ https://nz.pinterest.com/pin/296393219208794919/

    https://bugsnag.com/ https://logdna.com/ https://www.pagerduty.com/resources/logo/
  25. Πϯϑϥͷ؂ࢹ

  26. σʔλͷૹ৴ݩ σʔλͷૹ৴ઌ ௨஌ઌ CloudWatch Datadog Agent library agent

  27. • CPU • Memory • Load Average • Disk Usage

    • Disk I/O • Network I/O CloudWatch Datadog Agent
  28. CloudWatch

  29. None
  30. ΞϓϦέʔγϣϯͷ؂ࢹ

  31. σʔλͷૹ৴ݩ σʔλͷૹ৴ઌ ௨஌ઌ CloudWatch Datadog Agent library agent

  32. • ղੳͨ͠Πϕϯτ਺(ࠓ೔ɺࡢ೔ɺҰिؒલɺ Ұϲ݄લ) • Ͳͷ͓٬༷͔ΒͷΠϕϯτ਺͕ଟ͍͔ • ղੳ༻ʹཷ·͍ͬͯΔΩϡʔͷ਺ • λΠϜΞ΢τͷׂ߹ •

    ֤ॲཧʹ͔͔͍ͬͯΔ࣌ؒ • ϨεϙϯελΠϜ • ΞϓϦέʔγϣϯͷόά • Τϥʔϩά
  33. var dogstatsd = require('libs-dogstatsd') const stats = dogstatsd.start(); // some

    method stats.tick(‘test’, 1, 1, [pid:1234]); libs-dogstatsd (A wrapper library of node-dogstatsd) ࣮ߦ࣌ؒ΍ॲཧճ਺ΛDatadogʹૹ৴Ͱ͖Δϥούʔ https://github.com/makinoy/libs-dogstatsd/
  34. Ͳ͏΍ͬͯ؂ࢹ͍ͯ͠Δͷ͔ʁ

  35. LayerΛ෼͚ͯDashboardΛ࡞੒

  36. • Layer1: KARTE͕ੜ͖͍ͯΔ͔ • Layer2: ॏཁͳϝτϦοΫ • Layer3: ֤roleͷৄࡉϝτϦοΫ •

    (Layer4: ΠϯελϯεຖͷϝτϦοΫ)
  37. None
  38. None
  39. None
  40. None
  41. Ұݩ؅ཧ͢Δ͜ͱ υϦϧμ΢ϯͯ͠໰୊Λ௥͍͚ͬͯΔ͜ͱ

  42. ໨࣍ 1. αʔϏεͷ঺հ 2. ؂ࢹͷ໨త 3. Πϯϑϥߏ੒ 4. ؂ࢹͷํ๏ 5.

    Ξϥʔτ 6. ϋϚͬͨ͜ͱ 7. ·ͱΊ
  43. جຊతʹ͸Ξϥʔτ΋ DatadogͰҰݩ؅ཧ͢Δ

  44. ϝτϦοΫ τϦΨʔ ௨஌ઌ MCM ᮢ஋ ҟৗ஋ ▪Πϯϑϥ

  45. ϝτϦοΫ τϦΨʔ ௨஌ઌ ᮢ஋ ҟৗ஋ ▪ΞϓϦέʔγϣϯ ᮢ஋

  46. ҟৗ஋ݕग़͕ศར

  47. None
  48. ໨࣍ 1. αʔϏεͷ঺հ 2. ؂ࢹͷ໨త 3. Πϯϑϥߏ੒ 4. ؂ࢹͷํ๏ 5.

    Ξϥʔτ 6. ϋϚͬͨ͜ͱ 7. ·ͱΊ
  49. 1.ϞχλϦϯάπʔϧͷྉ͔͔ۚΓ͗͢໰୊

  50. ɾΠϯελϯεͷىಈ͸GCPʹൺ΂ͯ஗͍ ɾASΛ࢖͏ͱҰ౓ʹىಈ/ऴྃ͢Δ୆਺ΛઃఆͰ͖Δ ɾΠϯελϯεͷىಈ͸AWSʹൺ΂ͯૣ͍ ɾASͰىಈ/ऴྃ͢Δ୆਺͸ͪ͜ΒͰ͸ઃఆͰ͖ͳ͍ ɾ੒໿ͨ͠ϗετ͋ͨΓ$18/୆ ɾ੒໿୆਺Λ௒͑Δͱ1୆͋ͨΓ$0.03/hr

  51. AutoScaler͕ҰؾʹΠϯελϯεΛ૿΍ͯ͠ɺҰ ؾʹΠϯελϯεΛݮΒ͢ͱ͍͏ڍಈΛྑ͘͢Δ Datadog Agent͕ىಈͨ͠Πϯελϯε਺͕ܹ૿ GCPΛ࢖͍࢝Ί͔ͯΒલ݄ͷྉۚͷ2ഒΛ௒͑Δྉ ͕ۚ੥ٻ͞ΕΔ "# αϙʔτʹ૬ஊͯ͠࠷దͳ՝ۚମܥΛఏҊͯ͠΋Β ͍ɺ$1000Ҏ্҆͘ͳͬͨ $

  52. Πϯελϯεͷಛੑͱ؂ࢹαʔϏεͷ՝ۚମܥ Λ͔ͬ͠Γ೺Ѳ͓ͯ͘͠ʂ

  53. 2. ϝτϦοΫૹ৴λΠϛϯά͕૝ఆͱҧ͏໰୊

  54. Քಇ͍ͯ͠ΔΠϯελϯεͷ୆਺͕ ຖ෼ૹΒΕ͍ͯΔΑ͏ʹݟ͑Δ%

  55. ࢒೦5෼ຖͰͨ͠ʂ& ΞϥʔτΛ࢓ֻ͚ͯ΋ؾͮ͘ͷ͕5෼ޙ'

  56. ϝτϦοΫͷૹ৴ස౓Λ೺Ѳ͓ͯ͘͠ʂ

  57. ໨࣍ 1. αʔϏεͷ঺հ 2. ؂ࢹͷ໨త 3. Πϯϑϥߏ੒ 4. ؂ࢹͷํ๏ 5.

    Ξϥʔτ 6. ϋϚͬͨ͜ͱ 7. ·ͱΊ
  58. 1. σʔλ΍Ξϥʔτ͸Ұݩ؅ཧ͢Δ 2. υϦϧμ΢ϯͯ͠໰୊Λ௥͍͚ͬͯΔΑ͏ʹ͢Δ 3. ֤ϓϥοτϑΥʔϜͷಛੑͱɺ؂ࢹπʔϧͷ՝ۚ ମܥΛ೺Ѳ͢Δ 4. ΠϯςάϨʔγϣϯϝτϦοΫͷૹ৴ස౓Λ೺Ѳ ͢Δ

  59. ࠓޙ

  60. 1. DatadogʹΞϥʔτΛ·ͱΊΔ 2. Datadog͕SPOFʹͳ͍ͬͯΔ݅Λ࠶ݕ౼͢Δ 3. ؂ࢹઃఆͷexportΛ͍ͨ͠

  61. ΤϯδχΞืूதͰ͢ʂ