KARTE を支えるマルチプラットフォームインフラ監視 /karte-multi-platform-monitoring

【freee × プレイド】Tech Meetup 〜インフラ監視編〜

Transcript

1. KARTEΛࢧ͑Δ
   ϚϧνϓϥοτϑΥʔϜΠϯϑϥ؂ࢹ
   @ikemonn
   

   View Slide

2. @ikemonn
   ΠϯϑϥΤϯδχΞ
   2015೥11݄ʹjoin
   

   View Slide

3. ໨࣍
   1. αʔϏεͷ঺հ
   2. ؂ࢹͷ໨త
   3. Πϯϑϥߏ੒
   4. ؂ࢹͷํ๏
   5. Ξϥʔτ
   6. ϋϚͬͨ͜ͱ
   7. ·ͱΊ
   

   View Slide

4. ໨࣍
   1. αʔϏεͷ঺հ
   2. ؂ࢹͷ໨త
   3. Πϯϑϥߏ੒
   4. ؂ࢹͷํ๏
   5. Ξϥʔτ
   6. ϋϚͬͨ͜ͱ
   7. ·ͱΊ
   

   View Slide

5. ΢Σϒ઀٬ϓϥοτϑΥʔϜ
   KARTEΛ։ൃ͍ͯ͠·͢
   

   View Slide

6. ΢Σϒ઀٬ϓϥοτϑΥʔϜ
   KARTEΛ։ൃ͍ͯ͠·͢
   

   View Slide

7. ΠϯϑϥΤϯδχΞ͸2ਓ
   

   View Slide

8. ໨࣍
   1. αʔϏεͷ঺հ
   2. ؂ࢹͷ໨త
   3. Πϯϑϥߏ੒
   4. ؂ࢹͷํ๏
   5. Ξϥʔτ
   6. ϋϚͬͨ͜ͱ
   7. ·ͱΊ
   

   View Slide

9. ʮશһͰյ͠ͳ͕ΒਐΉʯ
   ͱ͍͏จԽΛࢧ͑Δ
   

   View Slide

10. Startup͸։ൃεϐʔυ͕ॏཁ
    

    View Slide

11. ৽͍͠αʔϏεɺݴޠΛੵۃతʹ
    ಋೖͰ͖ΔΑ͏ʹ
    

    View Slide

12. ރΕٕͨज़Ͱ͸ͳ͍ͷͰɺ
    ৘ใ͕͋·Γݟ͔ͭΒͳ͍
    

    View Slide

13. ଟগͷࣦഊ͸ڐ༰
    &
    ࣦഊͨ࣌͠ʹ͙͢ؾ෇͚Δ
    

    View Slide

14. ʮશһͰյ͠ͳ͕ΒਐΉʯจԽΛ
    ࢧ͑ΔͨΊʹ؂ࢹͷڧԽ !
    

    View Slide

15. ໨࣍
    1. αʔϏεͷ঺հ
    2. ؂ࢹͷ໨త
    3. Πϯϑϥߏ੒
    4. ؂ࢹͷํ๏
    5. Ξϥʔτ
    6. ϋϚͬͨ͜ͱ
    7. ·ͱΊ
    

    View Slide

16. ϚϧνϓϥοτϑΥʔϜ
    ৽͍͠αʔϏεͷੵۃར༻
    

    View Slide

17. View Slide

18. ໨࣍
    1. αʔϏεͷ঺հ
    2. ؂ࢹͷ໨త
    3. Πϯϑϥߏ੒
    4. ؂ࢹͷํ๏
    5. Ξϥʔτ
    6. ϋϚͬͨ͜ͱ
    7. ·ͱΊ
    

    View Slide

19. ؂ࢹํ๏
    

    View Slide

20. ผϓϥοτϑΥʔϜͷΠϯελϯε
    ಉ࢜ͷϝτϦοΫ΋ൺֱ͍ͨ͠
    ֤ϓϥοτϑΥʔϜʹґଘͨ͠ϝτ
    ϦοΫૹ৴ͷίʔυΛॻ͖ͨ͘ͳ͍
    ■Πϯϑϥଆͷ՝୊
    ■ΞϓϦଆͷ՝୊
    

    View Slide

21. CloudWatch
    ৭ʑͳ؅ཧը໘Λݟͨ͘ͳ͍
    http://www.irasutoya.com/2016/06/blog-post_147.html
    https://support.draw.io/pages/viewpage.action?pageId=1671326
    https://en.wikipedia.org/wiki/File:Google_Stackdriver_logo.svg
    https://www.runscope.com/
    

    View Slide

22. جຊతʹDatadogͰҰݩ؅ཧ
    

    View Slide

23. ԿΛ؂ࢹ͢Δͷ͔ʁ
    

    View Slide

24. σʔλͷૹ৴ݩ σʔλͷૹ৴ઌ ௨஌ઌ
    CloudWatch
    Datadog Agent
    library
    agent
    https://www.datadoghq.com/
    https://nz.pinterest.com/pin/296393219208794919/
    https://bugsnag.com/
    https://logdna.com/
    https://www.pagerduty.com/resources/logo/
    

    View Slide

25. Πϯϑϥͷ؂ࢹ
    

    View Slide

26. σʔλͷૹ৴ݩ σʔλͷૹ৴ઌ ௨஌ઌ
    CloudWatch
    Datadog Agent
    library
    agent
    

    View Slide

27. • CPU
    • Memory
    • Load Average
    • Disk Usage
    • Disk I/O
    • Network I/O
    CloudWatch
    Datadog Agent
    

    View Slide

28. CloudWatch
    

    View Slide

29. View Slide

30. ΞϓϦέʔγϣϯͷ؂ࢹ
    

    View Slide

31. σʔλͷૹ৴ݩ σʔλͷૹ৴ઌ ௨஌ઌ
    CloudWatch
    Datadog Agent
    library
    agent
    

    View Slide

32. • ղੳͨ͠Πϕϯτ਺(ࠓ೔ɺࡢ೔ɺҰिؒલɺ
    Ұϲ݄લ)
    • Ͳͷ͓٬༷͔ΒͷΠϕϯτ਺͕ଟ͍͔
    • ղੳ༻ʹཷ·͍ͬͯΔΩϡʔͷ਺
    • λΠϜΞ΢τͷׂ߹
    • ֤ॲཧʹ͔͔͍ͬͯΔ࣌ؒ
    • ϨεϙϯελΠϜ
    • ΞϓϦέʔγϣϯͷόά
    • Τϥʔϩά
    

    View Slide

33. var dogstatsd = require('libs-dogstatsd')
    const stats = dogstatsd.start();
    // some method
    stats.tick(‘test’, 1, 1, [pid:1234]);
    libs-dogstatsd
    (A wrapper library of node-dogstatsd)
    ࣮ߦ࣌ؒ΍ॲཧճ਺ΛDatadogʹૹ৴Ͱ͖Δϥούʔ
    https://github.com/makinoy/libs-dogstatsd/
    

    View Slide

34. Ͳ͏΍ͬͯ؂ࢹ͍ͯ͠Δͷ͔ʁ
    

    View Slide

35. LayerΛ෼͚ͯDashboardΛ࡞੒
    

    View Slide

36. • Layer1: KARTE͕ੜ͖͍ͯΔ͔
    • Layer2: ॏཁͳϝτϦοΫ
    • Layer3: ֤roleͷৄࡉϝτϦοΫ
    • (Layer4: ΠϯελϯεຖͷϝτϦοΫ)
    

    View Slide

37. View Slide

38. View Slide

39. View Slide

40. View Slide

41. Ұݩ؅ཧ͢Δ͜ͱ
    υϦϧμ΢ϯͯ͠໰୊Λ௥͍͚ͬͯΔ͜ͱ
    

    View Slide

42. ໨࣍
    1. αʔϏεͷ঺հ
    2. ؂ࢹͷ໨త
    3. Πϯϑϥߏ੒
    4. ؂ࢹͷํ๏
    5. Ξϥʔτ
    6. ϋϚͬͨ͜ͱ
    7. ·ͱΊ
    

    View Slide

43. جຊతʹ͸Ξϥʔτ΋
    DatadogͰҰݩ؅ཧ͢Δ
    

    View Slide

44. ϝτϦοΫ τϦΨʔ ௨஌ઌ
    MCM
    ᮢ஋
    ҟৗ஋
    ■Πϯϑϥ
    

    View Slide

45. ϝτϦοΫ τϦΨʔ ௨஌ઌ
    ᮢ஋
    ҟৗ஋
    ■ΞϓϦέʔγϣϯ
    ᮢ஋
    

    View Slide

46. ҟৗ஋ݕग़͕ศར
    

    View Slide

47. View Slide

48. ໨࣍
    1. αʔϏεͷ঺հ
    2. ؂ࢹͷ໨త
    3. Πϯϑϥߏ੒
    4. ؂ࢹͷํ๏
    5. Ξϥʔτ
    6. ϋϚͬͨ͜ͱ
    7. ·ͱΊ
    

    View Slide

49. 1.ϞχλϦϯάπʔϧͷྉ͔͔ۚΓ͗͢໰୊
    

    View Slide

50. ɾΠϯελϯεͷىಈ͸GCPʹൺ΂ͯ஗͍
    ɾASΛ࢖͏ͱҰ౓ʹىಈ/ऴྃ͢Δ୆਺ΛઃఆͰ͖Δ
    ɾΠϯελϯεͷىಈ͸AWSʹൺ΂ͯૣ͍
    ɾASͰىಈ/ऴྃ͢Δ୆਺͸ͪ͜ΒͰ͸ઃఆͰ͖ͳ͍
    ɾ੒໿ͨ͠ϗετ͋ͨΓ$18/୆
    ɾ੒໿୆਺Λ௒͑Δͱ1୆͋ͨΓ$0.03/hr
    

    View Slide

51. AutoScaler͕ҰؾʹΠϯελϯεΛ૿΍ͯ͠ɺҰ
    ؾʹΠϯελϯεΛݮΒ͢ͱ͍͏ڍಈΛྑ͘͢Δ
    Datadog Agent͕ىಈͨ͠Πϯελϯε਺͕ܹ૿
    GCPΛ࢖͍࢝Ί͔ͯΒલ݄ͷྉۚͷ2ഒΛ௒͑Δྉ
    ͕ۚ੥ٻ͞ΕΔ
    "#
    αϙʔτʹ૬ஊͯ͠࠷దͳ՝ۚମܥΛఏҊͯ͠΋Β
    ͍ɺ$1000Ҏ্҆͘ͳͬͨ
    $
    

    View Slide

52. Πϯελϯεͷಛੑͱ؂ࢹαʔϏεͷ՝ۚମܥ
    Λ͔ͬ͠Γ೺Ѳ͓ͯ͘͠ʂ
    

    View Slide

53. 2. ϝτϦοΫૹ৴λΠϛϯά͕૝ఆͱҧ͏໰୊
    

    View Slide

54. Քಇ͍ͯ͠ΔΠϯελϯεͷ୆਺͕
    ຖ෼ૹΒΕ͍ͯΔΑ͏ʹݟ͑Δ%
    

    View Slide

55. ࢒೦5෼ຖͰͨ͠ʂ&
    ΞϥʔτΛ࢓ֻ͚ͯ΋ؾͮ͘ͷ͕5෼ޙ'
    

    View Slide

56. ϝτϦοΫͷૹ৴ස౓Λ೺Ѳ͓ͯ͘͠ʂ
    

    View Slide

57. ໨࣍
    1. αʔϏεͷ঺հ
    2. ؂ࢹͷ໨త
    3. Πϯϑϥߏ੒
    4. ؂ࢹͷํ๏
    5. Ξϥʔτ
    6. ϋϚͬͨ͜ͱ
    7. ·ͱΊ
    

    View Slide

58. 1. σʔλ΍Ξϥʔτ͸Ұݩ؅ཧ͢Δ
    2. υϦϧμ΢ϯͯ͠໰୊Λ௥͍͚ͬͯΔΑ͏ʹ͢Δ
    3. ֤ϓϥοτϑΥʔϜͷಛੑͱɺ؂ࢹπʔϧͷ՝ۚ
    ମܥΛ೺Ѳ͢Δ
    4. ΠϯςάϨʔγϣϯϝτϦοΫͷૹ৴ස౓Λ೺Ѳ
    ͢Δ
    

    View Slide

59. ࠓޙ
    

    View Slide

60. 1. DatadogʹΞϥʔτΛ·ͱΊΔ
    2. Datadog͕SPOFʹͳ͍ͬͯΔ݅Λ࠶ݕ౼͢Δ
    3. ؂ࢹઃఆͷexportΛ͍ͨ͠
    

    View Slide

61. ΤϯδχΞืूதͰ͢ʂ
    

    View Slide