マルチクラウドな大規模サービスKARTEを支えるDatadog/multi-cloud monitoring with Datadog

6de30acfd736745120a3d6457dc6d45c?s=47 Daiki Matsui
November 13, 2019

マルチクラウドな大規模サービスKARTEを支えるDatadog/multi-cloud monitoring with Datadog

Datadog Summit Tokyoでの登壇資料です。
動画はhttps://www.youtube.com/watch?v=_Z4HBiwODbc

6de30acfd736745120a3d6457dc6d45c?s=128

Daiki Matsui

November 13, 2019
Tweet

Transcript

  1. 1 ϚϧνΫϥ΢υͳେن໛αʔϏε KARTEΛࢧ͑ΔDatadog ɹɹʛɹɹ© 2019 PLAID Inc. 2019.11.13ɹɹʛɹɹDatadog Summit Tokyo

    2019ɹɹʛɹ PLAID, Inc. Daiki Matsui
  2. 2 2019.11.13ɹɹʛɹɹDatadog Summit Tokyo 2019ɹɹʛɹ ɹɹʛɹɹ© 2019 PLAID Inc. ࣗݾ঺հ

    দҪ େٍ גࣜձࣾϓϨΠυ SRE Senior Architect 2012೥-2015೥·ͰגࣜձࣾαΠόʔυʹͯήʔϜ։ൃʹैࣄ 2015೥ΑΓϓϨΠυʹࢀը͠ɺSREͱͯ͠ϚϧνΫϥ΢υΠϯϑϥͷߏஙɾӡ༻౳Λ୲౰ 2015೥ೖࣾ
  3. ɹɹʛɹɹ© 2019 PLAID Inc. 3 2019.11.13ɹɹʛɹɹDatadog Summit Tokyo 2019ɹɹʛɹ גࣜձࣾϓϨΠυʹ͍ͭͯ

    גࣜձࣾϓϨΠυ ౦ژ౎தԝ۠ۜ࠲6-10-1 GINZA SIX 10F ઃཱɿ2011೥10݄ ैۀһɿ130໊ ࢿຊۚɿ149,987,878ԁ ※ࢿຊ४උؚۚΉ
  4. 4 2019.11.13ɹɹʛɹɹDatadog Summit Tokyo 2019ɹɹʛɹ ɹɹʛɹɹ© 2019 PLAID Inc. KARTEʹ͍ͭͯ

    ֬ೝ͢Δ جຊαʔϏε΍͝ར༻ํ๏Λ͝঺հ͠·͢ɻ ͓ಘͳಛయ΍໾ཱͭ৘ใ͕ຬࡌͰ͢ɻ ॳΊͯͷํ΁ ְתּׅ然钠ׅ׷ 6*رؠ؎ش٦׌ֽוזַזַ♳麦׃זְծ➙ ״׶إٔؗ،حف׃׋ְהְֲ倯䗳铣דׅկ UIرؠ؎ش٦ָ濼׏גֶֻץֹ 7אךرؠ؎ٕٝ٦ٕ DESIGN RULES » CHECK ׆׏ה⢪ִ׷،؎ذي׌ֽ׾䲧ִת׃׋կ ֿך堣⠓׾ֶ鋅鷕׃זֻկ 窫㼎ծ 妜׃ְ ౙͷओ໾ɺ Ξ΢λʔ COLLECTION OUTER 嗚稊勴⟝׾㼰׃㢌刿ׅ׷׌ֽדծ֮ז׋ך椚 䟝ך暟⟝ח⳿⠓ִ׷〳腉䚍ָ넝ֻז׶תׅկ 勴⟝׾㢌ִג嗚稊׃ג׫גֻ׌ְׁկ ׀䋞劄ך暟⟝כ 鋅אַ׶תׇ׿ד׃׋ַ ꟗׄ׷ ҰਓҰਓʹ߹Θͤͨ ސ٬ମݧΛఏڙ WebαΠτͷ๚໰ऀͷߦಈΛ
 ސ٬͝ͱʹϦΞϧλΠϜʹղੳ $9 ސ٬ମݧ ϓϥοτϑΥʔϜ
  5. 5 2019.11.13ɹɹʛɹɹDatadog Summit Tokyo 2019ɹɹʛɹ ɹɹʛɹɹ© 2019 PLAID Inc. ࿩͢͜ͱ

    KARTEͷಛ௃ KARTEΛࢧ͑Δ؂ࢹͷϙΠϯτ ࠓޙ΍Γ͍ͨ͜ͱ 1 2 3
  6. 6 2019.11.13ɹɹʛɹɹDatadog Summit Tokyo 2019ɹɹʛɹ ɹɹʛɹɹ© 2019 PLAID Inc. ࿩͢͜ͱ

    KARTEͷಛ௃ KARTEΛࢧ͑Δ؂ࢹͷϙΠϯτ ࠓޙ΍Γ͍ͨ͜ͱ
  7. 7 ɹɹʛɹɹ© 2019 PLAID Inc. 2019.11.13ɹɹʛɹɹDatadog Summit Tokyo 2019ɹɹʛɹ KARTEͷಛ௃

    1.Ϛϧνςφϯτ 2.௿ϨΠςϯγʔ/ߴෛՙ 3.େن໛γεςϜ 4.ϚϧνΫϥ΢υߏ੒ ࿩͢͜ͱ →
  8. 8 ɹɹʛɹɹ© 2019 PLAID Inc. 2019.11.13ɹɹʛɹɹDatadog Summit Tokyo 2019ɹɹʛɹ KARTEͷಛ௃

    1.Ϛϧνςφϯτ 2.௿ϨΠςϯγʔ/ߴෛՙ 3.େن໛γεςϜ 4.ϚϧνΫϥ΢υߏ੒ ࿩͢͜ͱ →
  9. 9 2019.11.13ɹɹʛɹɹDatadog Summit Tokyo 2019ɹɹʛɹ ɹɹʛɹɹ© 2019 PLAID Inc. Ϛϧνςφϯτ

  10. ɹɹʛɹɹ© 2019 PLAID Inc. 10 2019.11.13ɹɹʛɹɹDatadog Summit Tokyo 2019ɹɹʛɹ ɾ༷ʑͳෛՙͷ͓٬͞Μ͕͍Δ

    ɾΞΫηεύλʔϯ΋৭ʑ Ϛϧνςφϯτ
  11. ɹɹʛɹɹ© 2019 PLAID Inc. 11 2019.11.13ɹɹʛɹɹDatadog Summit Tokyo 2019ɹɹʛɹ ɾෛՙ͕1೔Ͱ໿10ഒมԽ͢Δ

    ɾηʔϧ౳Ͱෛՙ͕ٸ૿͢Δ Ϛϧνςφϯτ 1೔ͷෛՙਪҠ
  12. 12 ɹɹʛɹɹ© 2019 PLAID Inc. 2019.11.13ɹɹʛɹɹDatadog Summit Tokyo 2019ɹɹʛɹ KARTEͷಛ௃

    1.Ϛϧνςφϯτ 2.௿ϨΠςϯγʔ/ߴෛՙ 3.େن໛γεςϜ 4.ϚϧνΫϥ΢υߏ੒ ࿩͢͜ͱ →
  13. 13 2019.11.13ɹɹʛɹɹDatadog Summit Tokyo 2019ɹɹʛɹ ɹɹʛɹɹ© 2019 PLAID Inc. ௿ϨΠςϯγʔɺߴෛՙ

    sec to real-time actions 0.x events per day 3 billion stored in BigQuery 2+ PB events (rows) / sec 65,000 streaming insert per month 1 PB analysis per month 60+ PB SRE 3 ௿ϨΠςϯγʔ/ߴෛՙ
  14. 14 2019.11.13ɹɹʛɹɹDatadog Summit Tokyo 2019ɹɹʛɹ ɹɹʛɹɹ© 2019 PLAID Inc. ௿ϨΠςϯγʔɺߴෛՙ

    sec to real-time actions 0.x events per day 3 billion stored in BigQuery 2+ PB events (rows) / sec 65,000 streaming insert per month 1 PB analysis per month 60+ PB SRE 3 ௿ϨΠςϯγʔ/ߴෛՙ
  15. 15 2019.11.13ɹɹʛɹɹDatadog Summit Tokyo 2019ɹɹʛɹ ɹɹʛɹɹ© 2019 PLAID Inc. ௿ϨΠςϯγʔɺߴෛՙ

  16. 16 2019.11.13ɹɹʛɹɹDatadog Summit Tokyo 2019ɹɹʛɹ ɹɹʛɹɹ© 2019 PLAID Inc. ௿ϨΠςϯγʔɺߴෛՙ

    sec to real-time actions 0.x events per day 3 billion stored in BigQuery 2+ PB events (rows) / sec 65,000 streaming insert per month 1 PB analysis per month 60+ PB SRE 3 ௿ϨΠςϯγʔ/ߴෛՙ
  17. 17 ɹɹʛɹɹ© 2019 PLAID Inc. 2019.11.13ɹɹʛɹɹDatadog Summit Tokyo 2019ɹɹʛɹ KARTEͷಛ௃

    1.Ϛϧνςφϯτ 2.௿ϨΠςϯγʔ/ߴෛՙ 3.େن໛γεςϜ 4.ϚϧνΫϥ΢υߏ੒ ࿩͢͜ͱ →
  18. 18 2019.11.13ɹɹʛɹɹDatadog Summit Tokyo 2019ɹɹʛɹ ɹɹʛɹɹ© 2019 PLAID Inc. େن໛γεςϜ

  19. 19 ɹɹʛɹɹ© 2019 PLAID Inc. 2019.11.13ɹɹʛɹɹDatadog Summit Tokyo 2019ɹɹʛɹ KARTEͷಛ௃

    1.Ϛϧνςφϯτ 2.௿ϨΠςϯγʔ/ߴෛՙ 3.େن໛γεςϜ 4.ϚϧνΫϥ΢υߏ੒ ࿩͢͜ͱ →
  20. 20 2019.11.13ɹɹʛɹɹDatadog Summit Tokyo 2019ɹɹʛɹ ɹɹʛɹɹ© 2019 PLAID Inc. ϚϧνΫϥ΢υߏ੒

  21. ɹɹʛɹɹ© 2019 PLAID Inc. 21 2019.11.13ɹɹʛɹɹDatadog Summit Tokyo 2019ɹɹʛɹ Pros

    1. ֤ϓϩόΠμʔͷྑ͍ͱ͜औΓ͕Ͱ͖Δ 2. ϓϩόΠμʔͷো֐ΛผϓϩόΠμʔͰΧ όʔͰ͖Δ Pros/Cons ϚϧνΫϥ΢υߏ੒ Cons 1. ӡ༻ͷख͕ؒ૿͑Δ
  22. ɹɹʛɹɹ© 2019 PLAID Inc. 22 2019.11.13ɹɹʛɹɹDatadog Summit Tokyo 2019ɹɹʛɹ ৄࡉΛ஌Γ͍ͨํ͸

    ʮσϒαϛ 2018ʯͷࢿྉͰʂ ϚϧνΫϥ΢υߏ੒ https://speakerdeck.com/tikson/zhan-lue-topointonituite
  23. 23 2019.11.13ɹɹʛɹɹDatadog Summit Tokyo 2019ɹɹʛɹ ɹɹʛɹɹ© 2019 PLAID Inc. ࿩͢͜ͱ

    KARTEͷಛ௃ KARTEΛࢧ͑Δ؂ࢹͷϙΠϯτ ࠓޙ΍Γ͍ͨ͜ͱ ✓
  24. KARTEΛࢧ͑Δ؂ࢹͷϙΠϯτ 24 ɹɹʛɹɹ© 2019 PLAID Inc. 2019.11.13ɹɹʛɹɹDatadog Summit Tokyo 2019ɹɹʛɹ

    ɾϦΞϧλΠϜੑ ɾॊೈੑ ɾεέʔϥϏϦςΟ ɾ҆ఆੑ
  25. 25 ɹɹʛɹɹ© 2019 PLAID Inc. 2019.11.13ɹɹʛɹɹDatadog Summit Tokyo 2019ɹɹʛɹ KARTEͷಛ௃

    → 1.Ϛϧνςφϯτ 2.௿ϨΠςϯγʔ/ߴෛՙ 3.େن໛γεςϜ 4.ϚϧνΫϥ΢υߏ੒
  26. 26 ɹɹʛɹɹ© 2019 PLAID Inc. 2019.11.13ɹɹʛɹɹDatadog Summit Tokyo 2019ɹɹʛɹ KARTEΛࢧ͑Δ

    ؂ࢹͷϙΠϯτ 1.Datadogʹશͯू໿ 2.Custom metricsͷੵۃ׆༻ 3.LayerΛ෼͚ͨDashboard ࿩͢͜ͱ →
  27. 27 ɹɹʛɹɹ© 2019 PLAID Inc. 2019.11.13ɹɹʛɹɹDatadog Summit Tokyo 2019ɹɹʛɹ →

    1.Datadogʹશͯू໿ 2.Custom metricsͷੵۃ׆༻ 3.LayerΛ෼͚ͨDashboard 1.Ϛϧνςφϯτ 2.௿ϨΠςϯγʔ/ߴෛՙ 3.େن໛γεςϜ 4.ϚϧνΫϥ΢υߏ੒
  28. ɹɹʛɹɹ© 2019 PLAID Inc. 28 2019.11.13ɹɹʛɹɹDatadog Summit Tokyo 2019ɹɹʛɹ ো֐ରԠ࣌ʹ͸DatadogΛݟΕ͹

    ͙͢ʹঢ়گΛ೺ѲͰ͖ΔΑ͏ʹ Datadogʹશͯू໿
  29. 29 ɹɹʛɹɹ© 2019 PLAID Inc. 2019.11.13ɹɹʛɹɹDatadog Summit Tokyo 2019ɹɹʛɹ KARTEΛࢧ͑Δ

    ؂ࢹͷϙΠϯτ ࿩͢͜ͱ → 1.Datadogʹશͯू໿ 2.Custom metricsͷੵۃ׆༻ 3.LayerΛ෼͚ͨDashboard
  30. 30 ɹɹʛɹɹ© 2019 PLAID Inc. 2019.11.13ɹɹʛɹɹDatadog Summit Tokyo 2019ɹɹʛɹ →

    1.Datadogʹશͯू໿ 2.Custom metricsͷੵۃ׆༻ 3.LayerΛ෼͚ͨDashboard 1.Ϛϧνςφϯτ 2.௿ϨΠςϯγʔ/ߴෛՙ 3.େن໛γεςϜ 4.ϚϧνΫϥ΢υߏ੒
  31. ɹɹʛɹɹ© 2019 PLAID Inc. 31 2019.11.13ɹɹʛɹɹDatadog Summit Tokyo 2019ɹɹʛɹ 32M/mon

    ΄Ͳར༻ Custom metricsͷੵۃ׆༻
  32. Custom metricsͷੵۃ׆༻ 32 ɹɹʛɹɹ© 2019 PLAID Inc. 2019.11.13ɹɹʛɹɹDatadog Summit Tokyo

    2019ɹɹʛɹ https://github.com/makinoy/libs-dogstatsd ɾlibs-dogstatsdͰɺ؆୯ʹॲཧͷ࣮ߦճ਺΍ ࣮ߦ࣌ؒΛଌఆͰ͖ΔΑ͏ʹ ɾؔ਺ΑΓ΋ࡉ͔͍୯ҐͰlatencyΛଌఆ ɾϢʔβ୯ҐͰͷύϑΥʔϚϯεΛଌఆ
 1 const dd = require('libs-dogstatsd'); 2 const stats = dd.start(); 3 // measure hoge 4 hoge(); 5 stats.tick('hoge', 1, 1, [user_id:1234]); Node.jsͷwrapper library
  33. ɹɹʛɹɹ© 2019 PLAID Inc. 33 2019.11.13ɹɹʛɹɹDatadog Summit Tokyo 2019ɹɹʛɹ tagͰϢʔβࣝผ༻IDΛૹΔ

    ࠓى͍ͬͯ͜Δ໰୊͕ ʮಛఆͷϢʔβىҼͰى͍ͬͯ͜Δʁʯ ʮKARTEશମͰى͍ͬͯ͜Δʁʯ ͔Λ੾Γ෼͚͠΍͍ͯ͘͢͠Δ Custom metricsͷੵۃ׆༻
  34. 34 ɹɹʛɹɹ© 2019 PLAID Inc. 2019.11.13ɹɹʛɹɹDatadog Summit Tokyo 2019ɹɹʛɹ KARTEΛࢧ͑Δ

    ؂ࢹͷϙΠϯτ ࿩͢͜ͱ → 1.Datadogʹશͯू໿ 2.Custom metricsͷੵۃ׆༻ 3.LayerΛ෼͚ͨDashboard
  35. 35 ɹɹʛɹɹ© 2019 PLAID Inc. 2019.11.13ɹɹʛɹɹDatadog Summit Tokyo 2019ɹɹʛɹ →

    1.Datadogʹશͯू໿ 2.Custom metricsͷੵۃ׆༻ 3.LayerΛ෼͚ͨDashboard 1.Ϛϧνςφϯτ 2.௿ϨΠςϯγʔ/ߴෛՙ 3.େن໛γεςϜ 4.ϚϧνΫϥ΢υߏ੒
  36. ɹɹʛɹɹ© 2019 PLAID Inc. 36 2019.11.13ɹɹʛɹɹDatadog Summit Tokyo 2019ɹɹʛɹ Layer

    1 ɾKARTE͕ੜ͖͍ͯΔ͔Ͳ͏͔ LayerΛ෼͚ͨDashboard
  37. ɹɹʛɹɹ© 2019 PLAID Inc. 37 2019.11.13ɹɹʛɹɹDatadog Summit Tokyo 2019ɹɹʛɹ Layer

    2 ɾॏཁͳίϯϙʔωϯτ ɾॏཁͳΞϓϦέʔγϣϯͷϝτϦοΫ LayerΛ෼͚ͨDashboard
  38. ɹɹʛɹɹ© 2019 PLAID Inc. 38 2019.11.13ɹɹʛɹɹDatadog Summit Tokyo 2019ɹɹʛɹ Layer

    3 ɾ֤αʔό܈ͷࡉ͔͍ϝτϦοΫ LayerΛ෼͚ͨDashboard
  39. 39 2019.11.13ɹɹʛɹɹDatadog Summit Tokyo 2019ɹɹʛɹ ɹɹʛɹɹ© 2019 PLAID Inc. ࿩͢͜ͱ

    KARTEͷಛ௃ KARTEΛࢧ͑Δ؂ࢹͷϙΠϯτ ࠓޙ΍Γ͍ͨ͜ͱ ✓ ✓
  40. ɹࠓޙ΍Γ͍ͨ͜ͱ 40 ɹɹʛɹɹ© 2019 PLAID Inc. 2019.11.13ɹɹʛɹɹDatadog Summit Tokyo 2019ɹɹʛɹ

    શͯͷΤϯδχΞ͕ ো֐ͷҰ࣌ରԠΛͰ͖ΔΑ͏ʹ͢Δ ɾ؂ࢹ/ো֐ରԠͷϑϩʔͷ੔ཧ ɾDatadogΛ࢖͍͜ͳͯ͠΋Β͏
  41. ɹࠓޙ΍Γ͍ͨ͜ͱ 41 ɹɹʛɹɹ© 2019 PLAID Inc. 2019.11.13ɹɹʛɹɹDatadog Summit Tokyo 2019ɹɹʛɹ

    Custom metrics 1ԯԁ໰୊ͷղܾ ɾ͋·ΓʹศརͩͬͨͷͰ࢖͍͢ ͗ͯ͠·ͬͨ… ɾCustom metricsͷ੔ཧΛߦ͏
  42. 42 2019.11.13ɹɹʛɹɹDatadog Summit Tokyo 2019ɹɹʛɹ ɹɹʛɹɹ© 2019 PLAID Inc. ·ͱΊ

    ·ͱΊ
  43. 43 ɹɹʛɹɹ© 2019 PLAID Inc. 2019.11.13ɹɹʛɹɹDatadog Summit Tokyo 2019ɹɹʛɹ KARTEΛࢧ͑Δ

    ؂ࢹͷϙΠϯτ 1.Datadogʹશͯू໿ 2.Custom metricsͷੵۃ׆༻ 3.LayerΛ෼͚ͨDashboard →
  44. None