KARTE を支えるマルチプラットフォームインフラ監視 /karte-multi-platform-monitoring

【freee × プレイド】Tech Meetup 〜インフラ監視編〜

1. KARTEΛࢧ͑Δ ϚϧνϓϥοτϑΥʔϜΠϯϑϥ؂ࢹ @ikemonn
2. @ikemonn ΠϯϑϥΤϯδχΞ 2015೥11݄ʹjoin
3. ໨࣍ 1. αʔϏεͷ঺հ 2. ؂ࢹͷ໨త 3. Πϯϑϥߏ੒ 4. ؂ࢹͷํ๏ 5. Ξϥʔτ 6. ϋϚͬͨ͜ͱ 7. ·ͱΊ
4. ໨࣍ 1. αʔϏεͷ঺հ 2. ؂ࢹͷ໨త 3. Πϯϑϥߏ੒ 4. ؂ࢹͷํ๏ 5. Ξϥʔτ 6. ϋϚͬͨ͜ͱ 7. ·ͱΊ
5. ΢Σϒ઀٬ϓϥοτϑΥʔϜ KARTEΛ։ൃ͍ͯ͠·͢
6. ΢Σϒ઀٬ϓϥοτϑΥʔϜ KARTEΛ։ൃ͍ͯ͠·͢
7. ΠϯϑϥΤϯδχΞ͸2ਓ
8. ໨࣍ 1. αʔϏεͷ঺հ 2. ؂ࢹͷ໨త 3. Πϯϑϥߏ੒ 4. ؂ࢹͷํ๏ 5. Ξϥʔτ 6. ϋϚͬͨ͜ͱ 7. ·ͱΊ
9. ʮશһͰյ͠ͳ͕ΒਐΉʯ ͱ͍͏จԽΛࢧ͑Δ
10. Startup͸։ൃεϐʔυ͕ॏཁ
11. ৽͍͠αʔϏεɺݴޠΛੵۃతʹ ಋೖͰ͖ΔΑ͏ʹ
12. ރΕٕͨज़Ͱ͸ͳ͍ͷͰɺ ৘ใ͕͋·Γݟ͔ͭΒͳ͍
13. ଟগͷࣦഊ͸ڐ༰ & ࣦഊͨ࣌͠ʹ͙͢ؾ෇͚Δ
14. ʮશһͰյ͠ͳ͕ΒਐΉʯจԽΛ ࢧ͑ΔͨΊʹ؂ࢹͷڧԽ !
15. ໨࣍ 1. αʔϏεͷ঺հ 2. ؂ࢹͷ໨త 3. Πϯϑϥߏ੒ 4. ؂ࢹͷํ๏ 5. Ξϥʔτ 6. ϋϚͬͨ͜ͱ 7. ·ͱΊ
16. ϚϧνϓϥοτϑΥʔϜ ৽͍͠αʔϏεͷੵۃར༻
18. ໨࣍ 1. αʔϏεͷ঺հ 2. ؂ࢹͷ໨త 3. Πϯϑϥߏ੒ 4. ؂ࢹͷํ๏ 5. Ξϥʔτ 6. ϋϚͬͨ͜ͱ 7. ·ͱΊ
19. ؂ࢹํ๏
20. ผϓϥοτϑΥʔϜͷΠϯελϯε ಉ࢜ͷϝτϦοΫ΋ൺֱ͍ͨ͠ ֤ϓϥοτϑΥʔϜʹґଘͨ͠ϝτ ϦοΫૹ৴ͷίʔυΛॻ͖ͨ͘ͳ͍ ▪Πϯϑϥଆͷ՝୊ ▪ΞϓϦଆͷ՝୊
21. CloudWatch ৭ʑͳ؅ཧը໘Λݟͨ͘ͳ͍ http://www.irasutoya.com/2016/06/blog-post_147.html https://support.draw.io/pages/viewpage.action?pageId=1671326 https://en.wikipedia.org/wiki/File:Google_Stackdriver_logo.svg https://www.runscope.com/
22. جຊతʹDatadogͰҰݩ؅ཧ
23. ԿΛ؂ࢹ͢Δͷ͔ʁ
24. σʔλͷૹ৴ݩ σʔλͷૹ৴ઌ ௨஌ઌ CloudWatch Datadog Agent library agent https://www.datadoghq.com/ https://nz.pinterest.com/pin/296393219208794919/ https://bugsnag.com/ https://logdna.com/ https://www.pagerduty.com/resources/logo/
25. Πϯϑϥͷ؂ࢹ
26. σʔλͷૹ৴ݩ σʔλͷૹ৴ઌ ௨஌ઌ CloudWatch Datadog Agent library agent
27. • CPU • Memory • Load Average • Disk Usage • Disk I/O • Network I/O CloudWatch Datadog Agent
28. CloudWatch
30. ΞϓϦέʔγϣϯͷ؂ࢹ
31. σʔλͷૹ৴ݩ σʔλͷૹ৴ઌ ௨஌ઌ CloudWatch Datadog Agent library agent
32. • ղੳͨ͠Πϕϯτ਺(ࠓ೔ɺࡢ೔ɺҰिؒલɺ Ұϲ݄લ) • Ͳͷ͓٬༷͔ΒͷΠϕϯτ਺͕ଟ͍͔ • ղੳ༻ʹཷ·͍ͬͯΔΩϡʔͷ਺ • λΠϜΞ΢τͷׂ߹ • ֤ॲཧʹ͔͔͍ͬͯΔ࣌ؒ • ϨεϙϯελΠϜ • ΞϓϦέʔγϣϯͷόά • Τϥʔϩά
33. var dogstatsd = require('libs-dogstatsd') const stats = dogstatsd.start(); // some method stats.tick(‘test’, 1, 1, [pid:1234]); libs-dogstatsd (A wrapper library of node-dogstatsd) ࣮ߦ࣌ؒ΍ॲཧճ਺ΛDatadogʹૹ৴Ͱ͖Δϥούʔ https://github.com/makinoy/libs-dogstatsd/
34. Ͳ͏΍ͬͯ؂ࢹ͍ͯ͠Δͷ͔ʁ
35. LayerΛ෼͚ͯDashboardΛ࡞੒
36. • Layer1: KARTE͕ੜ͖͍ͯΔ͔ • Layer2: ॏཁͳϝτϦοΫ • Layer3: ֤roleͷৄࡉϝτϦοΫ • (Layer4: ΠϯελϯεຖͷϝτϦοΫ)
41. Ұݩ؅ཧ͢Δ͜ͱ υϦϧμ΢ϯͯ͠໰୊Λ௥͍͚ͬͯΔ͜ͱ
42. ໨࣍ 1. αʔϏεͷ঺հ 2. ؂ࢹͷ໨త 3. Πϯϑϥߏ੒ 4. ؂ࢹͷํ๏ 5. Ξϥʔτ 6. ϋϚͬͨ͜ͱ 7. ·ͱΊ
43. جຊతʹ͸Ξϥʔτ΋ DatadogͰҰݩ؅ཧ͢Δ
44. ϝτϦοΫ τϦΨʔ ௨஌ઌ MCM ᮢ஋ ҟৗ஋ ▪Πϯϑϥ
45. ϝτϦοΫ τϦΨʔ ௨஌ઌ ᮢ஋ ҟৗ஋ ▪ΞϓϦέʔγϣϯ ᮢ஋
46. ҟৗ஋ݕग़͕ศར
48. ໨࣍ 1. αʔϏεͷ঺հ 2. ؂ࢹͷ໨త 3. Πϯϑϥߏ੒ 4. ؂ࢹͷํ๏ 5. Ξϥʔτ 6. ϋϚͬͨ͜ͱ 7. ·ͱΊ
49. 1.ϞχλϦϯάπʔϧͷྉ͔͔ۚΓ͗͢໰୊
50. ɾΠϯελϯεͷىಈ͸GCPʹൺ΂ͯ஗͍ ɾASΛ࢖͏ͱҰ౓ʹىಈ/ऴྃ͢Δ୆਺ΛઃఆͰ͖Δ ɾΠϯελϯεͷىಈ͸AWSʹൺ΂ͯૣ͍ ɾASͰىಈ/ऴྃ͢Δ୆਺͸ͪ͜ΒͰ͸ઃఆͰ͖ͳ͍ ɾ੒໿ͨ͠ϗετ͋ͨΓ$18/୆ ɾ੒໿୆਺Λ௒͑Δͱ1୆͋ͨΓ$0.03/hr
51. AutoScaler͕ҰؾʹΠϯελϯεΛ૿΍ͯ͠ɺҰ ؾʹΠϯελϯεΛݮΒ͢ͱ͍͏ڍಈΛྑ͘͢Δ Datadog Agent͕ىಈͨ͠Πϯελϯε਺͕ܹ૿ GCPΛ࢖͍࢝Ί͔ͯΒલ݄ͷྉۚͷ2ഒΛ௒͑Δྉ ͕ۚ੥ٻ͞ΕΔ "# αϙʔτʹ૬ஊͯ͠࠷దͳ՝ۚମܥΛఏҊͯ͠΋Β ͍ɺ$1000Ҏ্҆͘ͳͬͨ $
52. Πϯελϯεͷಛੑͱ؂ࢹαʔϏεͷ՝ۚମܥ Λ͔ͬ͠Γ೺Ѳ͓ͯ͘͠ʂ
53. 2. ϝτϦοΫૹ৴λΠϛϯά͕૝ఆͱҧ͏໰୊
54. Քಇ͍ͯ͠ΔΠϯελϯεͷ୆਺͕ ຖ෼ૹΒΕ͍ͯΔΑ͏ʹݟ͑Δ%
55. ࢒೦5෼ຖͰͨ͠ʂ& ΞϥʔτΛ࢓ֻ͚ͯ΋ؾͮ͘ͷ͕5෼ޙ'
56. ϝτϦοΫͷૹ৴ස౓Λ೺Ѳ͓ͯ͘͠ʂ
57. ໨࣍ 1. αʔϏεͷ঺հ 2. ؂ࢹͷ໨త 3. Πϯϑϥߏ੒ 4. ؂ࢹͷํ๏ 5. Ξϥʔτ 6. ϋϚͬͨ͜ͱ 7. ·ͱΊ
58. 1. σʔλ΍Ξϥʔτ͸Ұݩ؅ཧ͢Δ 2. υϦϧμ΢ϯͯ͠໰୊Λ௥͍͚ͬͯΔΑ͏ʹ͢Δ 3. ֤ϓϥοτϑΥʔϜͷಛੑͱɺ؂ࢹπʔϧͷ՝ۚ ମܥΛ೺Ѳ͢Δ 4. ΠϯςάϨʔγϣϯϝτϦοΫͷૹ৴ස౓Λ೺Ѳ ͢Δ
59. ࠓޙ
60. 1. DatadogʹΞϥʔτΛ·ͱΊΔ 2. Datadog͕SPOFʹͳ͍ͬͯΔ݅Λ࠶ݕ౼͢Δ 3. ؂ࢹઃఆͷexportΛ͍ͨ͠
61. ΤϯδχΞืूதͰ͢ʂ