KARTE を支えるマルチプラットフォームインフラ監視 /karte-multi-platform-monitoring

Transcript

1. KARTEΛࢧ͑Δ ϚϧνϓϥοτϑΥʔϜΠϯϑϥ؂ࢹ @ikemonn

2. @ikemonn ΠϯϑϥΤϯδχΞ 2015೥11݄ʹjoin

3. ໨࣍ 1. αʔϏεͷ঺հ 2. ؂ࢹͷ໨త 3. Πϯϑϥߏ੒ 4. ؂ࢹͷํ๏ 5.

   Ξϥʔτ 6. ϋϚͬͨ͜ͱ 7. ·ͱΊ

4. ໨࣍ 1. αʔϏεͷ঺հ 2. ؂ࢹͷ໨త 3. Πϯϑϥߏ੒ 4. ؂ࢹͷํ๏ 5.

   Ξϥʔτ 6. ϋϚͬͨ͜ͱ 7. ·ͱΊ

5. ΢Σϒ઀٬ϓϥοτϑΥʔϜ KARTEΛ։ൃ͍ͯ͠·͢

6. ΢Σϒ઀٬ϓϥοτϑΥʔϜ KARTEΛ։ൃ͍ͯ͠·͢

7. ΠϯϑϥΤϯδχΞ͸2ਓ

8. ໨࣍ 1. αʔϏεͷ঺հ 2. ؂ࢹͷ໨త 3. Πϯϑϥߏ੒ 4. ؂ࢹͷํ๏ 5.

   Ξϥʔτ 6. ϋϚͬͨ͜ͱ 7. ·ͱΊ

9. ʮશһͰյ͠ͳ͕ΒਐΉʯ ͱ͍͏จԽΛࢧ͑Δ

10. Startup͸։ൃεϐʔυ͕ॏཁ

11. ৽͍͠αʔϏεɺݴޠΛੵۃతʹ ಋೖͰ͖ΔΑ͏ʹ

12. ރΕٕͨज़Ͱ͸ͳ͍ͷͰɺ ৘ใ͕͋·Γݟ͔ͭΒͳ͍

13. ଟগͷࣦഊ͸ڐ༰ & ࣦഊͨ࣌͠ʹ͙͢ؾ෇͚Δ

14. ʮશһͰյ͠ͳ͕ΒਐΉʯจԽΛ ࢧ͑ΔͨΊʹ؂ࢹͷڧԽ !

15. ໨࣍ 1. αʔϏεͷ঺հ 2. ؂ࢹͷ໨త 3. Πϯϑϥߏ੒ 4. ؂ࢹͷํ๏ 5.

    Ξϥʔτ 6. ϋϚͬͨ͜ͱ 7. ·ͱΊ

16. ϚϧνϓϥοτϑΥʔϜ ৽͍͠αʔϏεͷੵۃར༻

17. None

18. ໨࣍ 1. αʔϏεͷ঺հ 2. ؂ࢹͷ໨త 3. Πϯϑϥߏ੒ 4. ؂ࢹͷํ๏ 5.

    Ξϥʔτ 6. ϋϚͬͨ͜ͱ 7. ·ͱΊ

19. ؂ࢹํ๏

20. ผϓϥοτϑΥʔϜͷΠϯελϯε ಉ࢜ͷϝτϦοΫ΋ൺֱ͍ͨ͠ ֤ϓϥοτϑΥʔϜʹґଘͨ͠ϝτ ϦοΫૹ৴ͷίʔυΛॻ͖ͨ͘ͳ͍ ▪Πϯϑϥଆͷ՝୊ ▪ΞϓϦଆͷ՝୊

21. CloudWatch ৭ʑͳ؅ཧը໘Λݟͨ͘ͳ͍ http://www.irasutoya.com/2016/06/blog-post_147.html https://support.draw.io/pages/viewpage.action?pageId=1671326 https://en.wikipedia.org/wiki/File:Google_Stackdriver_logo.svg https://www.runscope.com/

22. جຊతʹDatadogͰҰݩ؅ཧ

23. ԿΛ؂ࢹ͢Δͷ͔ʁ

24. σʔλͷૹ৴ݩ σʔλͷૹ৴ઌ ௨஌ઌ CloudWatch Datadog Agent library agent https://www.datadoghq.com/ https://nz.pinterest.com/pin/296393219208794919/

    https://bugsnag.com/ https://logdna.com/ https://www.pagerduty.com/resources/logo/

25. Πϯϑϥͷ؂ࢹ

26. σʔλͷૹ৴ݩ σʔλͷૹ৴ઌ ௨஌ઌ CloudWatch Datadog Agent library agent

27. • CPU • Memory • Load Average • Disk Usage

    • Disk I/O • Network I/O CloudWatch Datadog Agent

28. CloudWatch

29. None

30. ΞϓϦέʔγϣϯͷ؂ࢹ

31. σʔλͷૹ৴ݩ σʔλͷૹ৴ઌ ௨஌ઌ CloudWatch Datadog Agent library agent

32. • ղੳͨ͠Πϕϯτ਺(ࠓ೔ɺࡢ೔ɺҰिؒલɺ Ұϲ݄લ) • Ͳͷ͓٬༷͔ΒͷΠϕϯτ਺͕ଟ͍͔ • ղੳ༻ʹཷ·͍ͬͯΔΩϡʔͷ਺ • λΠϜΞ΢τͷׂ߹ •

    ֤ॲཧʹ͔͔͍ͬͯΔ࣌ؒ • ϨεϙϯελΠϜ • ΞϓϦέʔγϣϯͷόά • Τϥʔϩά

33. var dogstatsd = require('libs-dogstatsd') const stats = dogstatsd.start(); // some

    method stats.tick(‘test’, 1, 1, [pid:1234]); libs-dogstatsd (A wrapper library of node-dogstatsd) ࣮ߦ࣌ؒ΍ॲཧճ਺ΛDatadogʹૹ৴Ͱ͖Δϥούʔ https://github.com/makinoy/libs-dogstatsd/

34. Ͳ͏΍ͬͯ؂ࢹ͍ͯ͠Δͷ͔ʁ

35. LayerΛ෼͚ͯDashboardΛ࡞੒

36. • Layer1: KARTE͕ੜ͖͍ͯΔ͔ • Layer2: ॏཁͳϝτϦοΫ • Layer3: ֤roleͷৄࡉϝτϦοΫ •

    (Layer4: ΠϯελϯεຖͷϝτϦοΫ)
37. None
38. None
39. None
40. None

41. Ұݩ؅ཧ͢Δ͜ͱ υϦϧμ΢ϯͯ͠໰୊Λ௥͍͚ͬͯΔ͜ͱ

42. ໨࣍ 1. αʔϏεͷ঺հ 2. ؂ࢹͷ໨త 3. Πϯϑϥߏ੒ 4. ؂ࢹͷํ๏ 5.

    Ξϥʔτ 6. ϋϚͬͨ͜ͱ 7. ·ͱΊ

43. جຊతʹ͸Ξϥʔτ΋ DatadogͰҰݩ؅ཧ͢Δ

44. ϝτϦοΫ τϦΨʔ ௨஌ઌ MCM ᮢ஋ ҟৗ஋ ▪Πϯϑϥ

45. ϝτϦοΫ τϦΨʔ ௨஌ઌ ᮢ஋ ҟৗ஋ ▪ΞϓϦέʔγϣϯ ᮢ஋

46. ҟৗ஋ݕग़͕ศར

47. None

48. ໨࣍ 1. αʔϏεͷ঺հ 2. ؂ࢹͷ໨త 3. Πϯϑϥߏ੒ 4. ؂ࢹͷํ๏ 5.

    Ξϥʔτ 6. ϋϚͬͨ͜ͱ 7. ·ͱΊ

49. 1.ϞχλϦϯάπʔϧͷྉ͔͔ۚΓ͗͢໰୊

50. ɾΠϯελϯεͷىಈ͸GCPʹൺ΂ͯ஗͍ ɾASΛ࢖͏ͱҰ౓ʹىಈ/ऴྃ͢Δ୆਺ΛઃఆͰ͖Δ ɾΠϯελϯεͷىಈ͸AWSʹൺ΂ͯૣ͍ ɾASͰىಈ/ऴྃ͢Δ୆਺͸ͪ͜ΒͰ͸ઃఆͰ͖ͳ͍ ɾ੒໿ͨ͠ϗετ͋ͨΓ$18/୆ ɾ੒໿୆਺Λ௒͑Δͱ1୆͋ͨΓ$0.03/hr

51. AutoScaler͕ҰؾʹΠϯελϯεΛ૿΍ͯ͠ɺҰ ؾʹΠϯελϯεΛݮΒ͢ͱ͍͏ڍಈΛྑ͘͢Δ Datadog Agent͕ىಈͨ͠Πϯελϯε਺͕ܹ૿ GCPΛ࢖͍࢝Ί͔ͯΒલ݄ͷྉۚͷ2ഒΛ௒͑Δྉ ͕ۚ੥ٻ͞ΕΔ "# αϙʔτʹ૬ஊͯ͠࠷దͳ՝ۚମܥΛఏҊͯ͠΋Β ͍ɺ$1000Ҏ্҆͘ͳͬͨ $

52. Πϯελϯεͷಛੑͱ؂ࢹαʔϏεͷ՝ۚମܥ Λ͔ͬ͠Γ೺Ѳ͓ͯ͘͠ʂ

53. 2. ϝτϦοΫૹ৴λΠϛϯά͕૝ఆͱҧ͏໰୊

54. Քಇ͍ͯ͠ΔΠϯελϯεͷ୆਺͕ ຖ෼ૹΒΕ͍ͯΔΑ͏ʹݟ͑Δ%

55. ࢒೦5෼ຖͰͨ͠ʂ& ΞϥʔτΛ࢓ֻ͚ͯ΋ؾͮ͘ͷ͕5෼ޙ'

56. ϝτϦοΫͷૹ৴ස౓Λ೺Ѳ͓ͯ͘͠ʂ

57. ໨࣍ 1. αʔϏεͷ঺հ 2. ؂ࢹͷ໨త 3. Πϯϑϥߏ੒ 4. ؂ࢹͷํ๏ 5.

    Ξϥʔτ 6. ϋϚͬͨ͜ͱ 7. ·ͱΊ

58. 1. σʔλ΍Ξϥʔτ͸Ұݩ؅ཧ͢Δ 2. υϦϧμ΢ϯͯ͠໰୊Λ௥͍͚ͬͯΔΑ͏ʹ͢Δ 3. ֤ϓϥοτϑΥʔϜͷಛੑͱɺ؂ࢹπʔϧͷ՝ۚ ମܥΛ೺Ѳ͢Δ 4. ΠϯςάϨʔγϣϯϝτϦοΫͷૹ৴ස౓Λ೺Ѳ ͢Δ

59. ࠓޙ

60. 1. DatadogʹΞϥʔτΛ·ͱΊΔ 2. Datadog͕SPOFʹͳ͍ͬͯΔ݅Λ࠶ݕ౼͢Δ 3. ؂ࢹઃఆͷexportΛ͍ͨ͠

61. ΤϯδχΞืूதͰ͢ʂ