Upgrade to Pro — share decks privately, control downloads, hide ads and more …

KARTE を支えるマルチプラットフォームインフラ監視 /karte-multi-platform-monitoring

KARTE を支えるマルチプラットフォームインフラ監視 /karte-multi-platform-monitoring

【freee × プレイド】Tech Meetup 〜インフラ監視編〜

Daiki Matsui

January 20, 2017
Tweet

More Decks by Daiki Matsui

Other Decks in Technology

Transcript

  1. KARTEΛࢧ͑Δ
    ϚϧνϓϥοτϑΥʔϜΠϯϑϥ؂ࢹ
    @ikemonn

    View Slide

  2. @ikemonn
    ΠϯϑϥΤϯδχΞ
    2015೥11݄ʹjoin

    View Slide

  3. ໨࣍
    1. αʔϏεͷ঺հ
    2. ؂ࢹͷ໨త
    3. Πϯϑϥߏ੒
    4. ؂ࢹͷํ๏
    5. Ξϥʔτ
    6. ϋϚͬͨ͜ͱ
    7. ·ͱΊ

    View Slide

  4. ໨࣍
    1. αʔϏεͷ঺հ
    2. ؂ࢹͷ໨త
    3. Πϯϑϥߏ੒
    4. ؂ࢹͷํ๏
    5. Ξϥʔτ
    6. ϋϚͬͨ͜ͱ
    7. ·ͱΊ

    View Slide

  5. ΢Σϒ઀٬ϓϥοτϑΥʔϜ
    KARTEΛ։ൃ͍ͯ͠·͢

    View Slide

  6. ΢Σϒ઀٬ϓϥοτϑΥʔϜ
    KARTEΛ։ൃ͍ͯ͠·͢

    View Slide

  7. ΠϯϑϥΤϯδχΞ͸2ਓ

    View Slide

  8. ໨࣍
    1. αʔϏεͷ঺հ
    2. ؂ࢹͷ໨త
    3. Πϯϑϥߏ੒
    4. ؂ࢹͷํ๏
    5. Ξϥʔτ
    6. ϋϚͬͨ͜ͱ
    7. ·ͱΊ

    View Slide

  9. ʮશһͰյ͠ͳ͕ΒਐΉʯ
    ͱ͍͏จԽΛࢧ͑Δ

    View Slide

  10. Startup͸։ൃεϐʔυ͕ॏཁ

    View Slide

  11. ৽͍͠αʔϏεɺݴޠΛੵۃతʹ
    ಋೖͰ͖ΔΑ͏ʹ

    View Slide

  12. ރΕٕͨज़Ͱ͸ͳ͍ͷͰɺ
    ৘ใ͕͋·Γݟ͔ͭΒͳ͍

    View Slide

  13. ଟগͷࣦഊ͸ڐ༰
    &
    ࣦഊͨ࣌͠ʹ͙͢ؾ෇͚Δ

    View Slide

  14. ʮશһͰյ͠ͳ͕ΒਐΉʯจԽΛ
    ࢧ͑ΔͨΊʹ؂ࢹͷڧԽ !

    View Slide

  15. ໨࣍
    1. αʔϏεͷ঺հ
    2. ؂ࢹͷ໨త
    3. Πϯϑϥߏ੒
    4. ؂ࢹͷํ๏
    5. Ξϥʔτ
    6. ϋϚͬͨ͜ͱ
    7. ·ͱΊ

    View Slide

  16. ϚϧνϓϥοτϑΥʔϜ
    ৽͍͠αʔϏεͷੵۃར༻

    View Slide

  17. View Slide

  18. ໨࣍
    1. αʔϏεͷ঺հ
    2. ؂ࢹͷ໨త
    3. Πϯϑϥߏ੒
    4. ؂ࢹͷํ๏
    5. Ξϥʔτ
    6. ϋϚͬͨ͜ͱ
    7. ·ͱΊ

    View Slide

  19. ؂ࢹํ๏

    View Slide

  20. ผϓϥοτϑΥʔϜͷΠϯελϯε
    ಉ࢜ͷϝτϦοΫ΋ൺֱ͍ͨ͠
    ֤ϓϥοτϑΥʔϜʹґଘͨ͠ϝτ
    ϦοΫૹ৴ͷίʔυΛॻ͖ͨ͘ͳ͍
    ■Πϯϑϥଆͷ՝୊
    ■ΞϓϦଆͷ՝୊

    View Slide

  21. CloudWatch
    ৭ʑͳ؅ཧը໘Λݟͨ͘ͳ͍
    http://www.irasutoya.com/2016/06/blog-post_147.html
    https://support.draw.io/pages/viewpage.action?pageId=1671326
    https://en.wikipedia.org/wiki/File:Google_Stackdriver_logo.svg
    https://www.runscope.com/

    View Slide

  22. جຊతʹDatadogͰҰݩ؅ཧ

    View Slide

  23. ԿΛ؂ࢹ͢Δͷ͔ʁ

    View Slide

  24. σʔλͷૹ৴ݩ σʔλͷૹ৴ઌ ௨஌ઌ
    CloudWatch
    Datadog Agent
    library
    agent
    https://www.datadoghq.com/
    https://nz.pinterest.com/pin/296393219208794919/
    https://bugsnag.com/
    https://logdna.com/
    https://www.pagerduty.com/resources/logo/

    View Slide

  25. Πϯϑϥͷ؂ࢹ

    View Slide

  26. σʔλͷૹ৴ݩ σʔλͷૹ৴ઌ ௨஌ઌ
    CloudWatch
    Datadog Agent
    library
    agent

    View Slide

  27. • CPU
    • Memory
    • Load Average
    • Disk Usage
    • Disk I/O
    • Network I/O
    CloudWatch
    Datadog Agent

    View Slide

  28. CloudWatch

    View Slide

  29. View Slide

  30. ΞϓϦέʔγϣϯͷ؂ࢹ

    View Slide

  31. σʔλͷૹ৴ݩ σʔλͷૹ৴ઌ ௨஌ઌ
    CloudWatch
    Datadog Agent
    library
    agent

    View Slide

  32. • ղੳͨ͠Πϕϯτ਺(ࠓ೔ɺࡢ೔ɺҰिؒલɺ
    Ұϲ݄લ)
    • Ͳͷ͓٬༷͔ΒͷΠϕϯτ਺͕ଟ͍͔
    • ղੳ༻ʹཷ·͍ͬͯΔΩϡʔͷ਺
    • λΠϜΞ΢τͷׂ߹
    • ֤ॲཧʹ͔͔͍ͬͯΔ࣌ؒ
    • ϨεϙϯελΠϜ
    • ΞϓϦέʔγϣϯͷόά
    • Τϥʔϩά

    View Slide

  33. var dogstatsd = require('libs-dogstatsd')
    const stats = dogstatsd.start();
    // some method
    stats.tick(‘test’, 1, 1, [pid:1234]);
    libs-dogstatsd
    (A wrapper library of node-dogstatsd)
    ࣮ߦ࣌ؒ΍ॲཧճ਺ΛDatadogʹૹ৴Ͱ͖Δϥούʔ
    https://github.com/makinoy/libs-dogstatsd/

    View Slide

  34. Ͳ͏΍ͬͯ؂ࢹ͍ͯ͠Δͷ͔ʁ

    View Slide

  35. LayerΛ෼͚ͯDashboardΛ࡞੒

    View Slide

  36. • Layer1: KARTE͕ੜ͖͍ͯΔ͔
    • Layer2: ॏཁͳϝτϦοΫ
    • Layer3: ֤roleͷৄࡉϝτϦοΫ
    • (Layer4: ΠϯελϯεຖͷϝτϦοΫ)

    View Slide

  37. View Slide

  38. View Slide

  39. View Slide

  40. View Slide

  41. Ұݩ؅ཧ͢Δ͜ͱ
    υϦϧμ΢ϯͯ͠໰୊Λ௥͍͚ͬͯΔ͜ͱ

    View Slide

  42. ໨࣍
    1. αʔϏεͷ঺հ
    2. ؂ࢹͷ໨త
    3. Πϯϑϥߏ੒
    4. ؂ࢹͷํ๏
    5. Ξϥʔτ
    6. ϋϚͬͨ͜ͱ
    7. ·ͱΊ

    View Slide

  43. جຊతʹ͸Ξϥʔτ΋
    DatadogͰҰݩ؅ཧ͢Δ

    View Slide

  44. ϝτϦοΫ τϦΨʔ ௨஌ઌ
    MCM
    ᮢ஋
    ҟৗ஋
    ■Πϯϑϥ

    View Slide

  45. ϝτϦοΫ τϦΨʔ ௨஌ઌ
    ᮢ஋
    ҟৗ஋
    ■ΞϓϦέʔγϣϯ
    ᮢ஋

    View Slide

  46. ҟৗ஋ݕग़͕ศར

    View Slide

  47. View Slide

  48. ໨࣍
    1. αʔϏεͷ঺հ
    2. ؂ࢹͷ໨త
    3. Πϯϑϥߏ੒
    4. ؂ࢹͷํ๏
    5. Ξϥʔτ
    6. ϋϚͬͨ͜ͱ
    7. ·ͱΊ

    View Slide

  49. 1.ϞχλϦϯάπʔϧͷྉ͔͔ۚΓ͗͢໰୊

    View Slide

  50. ɾΠϯελϯεͷىಈ͸GCPʹൺ΂ͯ஗͍
    ɾASΛ࢖͏ͱҰ౓ʹىಈ/ऴྃ͢Δ୆਺ΛઃఆͰ͖Δ
    ɾΠϯελϯεͷىಈ͸AWSʹൺ΂ͯૣ͍
    ɾASͰىಈ/ऴྃ͢Δ୆਺͸ͪ͜ΒͰ͸ઃఆͰ͖ͳ͍
    ɾ੒໿ͨ͠ϗετ͋ͨΓ$18/୆
    ɾ੒໿୆਺Λ௒͑Δͱ1୆͋ͨΓ$0.03/hr

    View Slide

  51. AutoScaler͕ҰؾʹΠϯελϯεΛ૿΍ͯ͠ɺҰ
    ؾʹΠϯελϯεΛݮΒ͢ͱ͍͏ڍಈΛྑ͘͢Δ
    Datadog Agent͕ىಈͨ͠Πϯελϯε਺͕ܹ૿
    GCPΛ࢖͍࢝Ί͔ͯΒલ݄ͷྉۚͷ2ഒΛ௒͑Δྉ
    ͕ۚ੥ٻ͞ΕΔ
    "#
    αϙʔτʹ૬ஊͯ͠࠷దͳ՝ۚମܥΛఏҊͯ͠΋Β
    ͍ɺ$1000Ҏ্҆͘ͳͬͨ
    $

    View Slide

  52. Πϯελϯεͷಛੑͱ؂ࢹαʔϏεͷ՝ۚମܥ
    Λ͔ͬ͠Γ೺Ѳ͓ͯ͘͠ʂ

    View Slide

  53. 2. ϝτϦοΫૹ৴λΠϛϯά͕૝ఆͱҧ͏໰୊

    View Slide

  54. Քಇ͍ͯ͠ΔΠϯελϯεͷ୆਺͕
    ຖ෼ૹΒΕ͍ͯΔΑ͏ʹݟ͑Δ%

    View Slide

  55. ࢒೦5෼ຖͰͨ͠ʂ&
    ΞϥʔτΛ࢓ֻ͚ͯ΋ؾͮ͘ͷ͕5෼ޙ'

    View Slide

  56. ϝτϦοΫͷૹ৴ස౓Λ೺Ѳ͓ͯ͘͠ʂ

    View Slide

  57. ໨࣍
    1. αʔϏεͷ঺հ
    2. ؂ࢹͷ໨త
    3. Πϯϑϥߏ੒
    4. ؂ࢹͷํ๏
    5. Ξϥʔτ
    6. ϋϚͬͨ͜ͱ
    7. ·ͱΊ

    View Slide

  58. 1. σʔλ΍Ξϥʔτ͸Ұݩ؅ཧ͢Δ
    2. υϦϧμ΢ϯͯ͠໰୊Λ௥͍͚ͬͯΔΑ͏ʹ͢Δ
    3. ֤ϓϥοτϑΥʔϜͷಛੑͱɺ؂ࢹπʔϧͷ՝ۚ
    ମܥΛ೺Ѳ͢Δ
    4. ΠϯςάϨʔγϣϯϝτϦοΫͷૹ৴ස౓Λ೺Ѳ
    ͢Δ

    View Slide

  59. ࠓޙ

    View Slide

  60. 1. DatadogʹΞϥʔτΛ·ͱΊΔ
    2. Datadog͕SPOFʹͳ͍ͬͯΔ݅Λ࠶ݕ౼͢Δ
    3. ؂ࢹઃఆͷexportΛ͍ͨ͠

    View Slide

  61. ΤϯδχΞืूதͰ͢ʂ

    View Slide