Upgrade to Pro — share decks privately, control downloads, hide ads and more …

マルチクラウドな大規模サービスKARTEを支えるDatadog/multi-cloud mon...

Daiki Matsui
November 13, 2019

マルチクラウドな大規模サービスKARTEを支えるDatadog/multi-cloud monitoring with Datadog

Datadog Summit Tokyoでの登壇資料です。
動画はhttps://www.youtube.com/watch?v=_Z4HBiwODbc

Daiki Matsui

November 13, 2019
Tweet

More Decks by Daiki Matsui

Other Decks in Technology

Transcript

  1. 2 2019.11.13ɹɹʛɹɹDatadog Summit Tokyo 2019ɹɹʛɹ ɹɹʛɹɹ© 2019 PLAID Inc. ࣗݾ঺հ

    দҪ େٍ גࣜձࣾϓϨΠυ SRE Senior Architect 2012೥-2015೥·ͰגࣜձࣾαΠόʔυʹͯήʔϜ։ൃʹैࣄ 2015೥ΑΓϓϨΠυʹࢀը͠ɺSREͱͯ͠ϚϧνΫϥ΢υΠϯϑϥͷߏஙɾӡ༻౳Λ୲౰ 2015೥ೖࣾ
  2. ɹɹʛɹɹ© 2019 PLAID Inc. 3 2019.11.13ɹɹʛɹɹDatadog Summit Tokyo 2019ɹɹʛɹ גࣜձࣾϓϨΠυʹ͍ͭͯ

    גࣜձࣾϓϨΠυ ౦ژ౎தԝ۠ۜ࠲6-10-1 GINZA SIX 10F ઃཱɿ2011೥10݄ ैۀһɿ130໊ ࢿຊۚɿ149,987,878ԁ ※ࢿຊ४උؚۚΉ
  3. 4 2019.11.13ɹɹʛɹɹDatadog Summit Tokyo 2019ɹɹʛɹ ɹɹʛɹɹ© 2019 PLAID Inc. KARTEʹ͍ͭͯ

    ֬ೝ͢Δ جຊαʔϏε΍͝ར༻ํ๏Λ͝঺հ͠·͢ɻ ͓ಘͳಛయ΍໾ཱͭ৘ใ͕ຬࡌͰ͢ɻ ॳΊͯͷํ΁ ְתּׅ然钠ׅ׷ 6*رؠ؎ش٦׌ֽוזַזַ♳麦׃זְծ➙ ״׶إٔؗ،حف׃׋ְהְֲ倯䗳铣דׅկ UIرؠ؎ش٦ָ濼׏גֶֻץֹ 7אךرؠ؎ٕٝ٦ٕ DESIGN RULES » CHECK ׆׏ה⢪ִ׷،؎ذي׌ֽ׾䲧ִת׃׋կ ֿך堣⠓׾ֶ鋅鷕׃זֻկ 窫㼎ծ 妜׃ְ ౙͷओ໾ɺ Ξ΢λʔ COLLECTION OUTER 嗚稊勴⟝׾㼰׃㢌刿ׅ׷׌ֽדծ֮ז׋ך椚 䟝ך暟⟝ח⳿⠓ִ׷〳腉䚍ָ넝ֻז׶תׅկ 勴⟝׾㢌ִג嗚稊׃ג׫גֻ׌ְׁկ ׀䋞劄ך暟⟝כ 鋅אַ׶תׇ׿ד׃׋ַ ꟗׄ׷ ҰਓҰਓʹ߹Θͤͨ ސ٬ମݧΛఏڙ WebαΠτͷ๚໰ऀͷߦಈΛ
 ސ٬͝ͱʹϦΞϧλΠϜʹղੳ $9 ސ٬ମݧ ϓϥοτϑΥʔϜ
  4. 5 2019.11.13ɹɹʛɹɹDatadog Summit Tokyo 2019ɹɹʛɹ ɹɹʛɹɹ© 2019 PLAID Inc. ࿩͢͜ͱ

    KARTEͷಛ௃ KARTEΛࢧ͑Δ؂ࢹͷϙΠϯτ ࠓޙ΍Γ͍ͨ͜ͱ 1 2 3
  5. 6 2019.11.13ɹɹʛɹɹDatadog Summit Tokyo 2019ɹɹʛɹ ɹɹʛɹɹ© 2019 PLAID Inc. ࿩͢͜ͱ

    KARTEͷಛ௃ KARTEΛࢧ͑Δ؂ࢹͷϙΠϯτ ࠓޙ΍Γ͍ͨ͜ͱ
  6. 7 ɹɹʛɹɹ© 2019 PLAID Inc. 2019.11.13ɹɹʛɹɹDatadog Summit Tokyo 2019ɹɹʛɹ KARTEͷಛ௃

    1.Ϛϧνςφϯτ 2.௿ϨΠςϯγʔ/ߴෛՙ 3.େن໛γεςϜ 4.ϚϧνΫϥ΢υߏ੒ ࿩͢͜ͱ →
  7. 8 ɹɹʛɹɹ© 2019 PLAID Inc. 2019.11.13ɹɹʛɹɹDatadog Summit Tokyo 2019ɹɹʛɹ KARTEͷಛ௃

    1.Ϛϧνςφϯτ 2.௿ϨΠςϯγʔ/ߴෛՙ 3.େن໛γεςϜ 4.ϚϧνΫϥ΢υߏ੒ ࿩͢͜ͱ →
  8. 12 ɹɹʛɹɹ© 2019 PLAID Inc. 2019.11.13ɹɹʛɹɹDatadog Summit Tokyo 2019ɹɹʛɹ KARTEͷಛ௃

    1.Ϛϧνςφϯτ 2.௿ϨΠςϯγʔ/ߴෛՙ 3.େن໛γεςϜ 4.ϚϧνΫϥ΢υߏ੒ ࿩͢͜ͱ →
  9. 13 2019.11.13ɹɹʛɹɹDatadog Summit Tokyo 2019ɹɹʛɹ ɹɹʛɹɹ© 2019 PLAID Inc. ௿ϨΠςϯγʔɺߴෛՙ

    sec to real-time actions 0.x events per day 3 billion stored in BigQuery 2+ PB events (rows) / sec 65,000 streaming insert per month 1 PB analysis per month 60+ PB SRE 3 ௿ϨΠςϯγʔ/ߴෛՙ
  10. 14 2019.11.13ɹɹʛɹɹDatadog Summit Tokyo 2019ɹɹʛɹ ɹɹʛɹɹ© 2019 PLAID Inc. ௿ϨΠςϯγʔɺߴෛՙ

    sec to real-time actions 0.x events per day 3 billion stored in BigQuery 2+ PB events (rows) / sec 65,000 streaming insert per month 1 PB analysis per month 60+ PB SRE 3 ௿ϨΠςϯγʔ/ߴෛՙ
  11. 16 2019.11.13ɹɹʛɹɹDatadog Summit Tokyo 2019ɹɹʛɹ ɹɹʛɹɹ© 2019 PLAID Inc. ௿ϨΠςϯγʔɺߴෛՙ

    sec to real-time actions 0.x events per day 3 billion stored in BigQuery 2+ PB events (rows) / sec 65,000 streaming insert per month 1 PB analysis per month 60+ PB SRE 3 ௿ϨΠςϯγʔ/ߴෛՙ
  12. 17 ɹɹʛɹɹ© 2019 PLAID Inc. 2019.11.13ɹɹʛɹɹDatadog Summit Tokyo 2019ɹɹʛɹ KARTEͷಛ௃

    1.Ϛϧνςφϯτ 2.௿ϨΠςϯγʔ/ߴෛՙ 3.େن໛γεςϜ 4.ϚϧνΫϥ΢υߏ੒ ࿩͢͜ͱ →
  13. 19 ɹɹʛɹɹ© 2019 PLAID Inc. 2019.11.13ɹɹʛɹɹDatadog Summit Tokyo 2019ɹɹʛɹ KARTEͷಛ௃

    1.Ϛϧνςφϯτ 2.௿ϨΠςϯγʔ/ߴෛՙ 3.େن໛γεςϜ 4.ϚϧνΫϥ΢υߏ੒ ࿩͢͜ͱ →
  14. ɹɹʛɹɹ© 2019 PLAID Inc. 21 2019.11.13ɹɹʛɹɹDatadog Summit Tokyo 2019ɹɹʛɹ Pros

    1. ֤ϓϩόΠμʔͷྑ͍ͱ͜औΓ͕Ͱ͖Δ 2. ϓϩόΠμʔͷো֐ΛผϓϩόΠμʔͰΧ όʔͰ͖Δ Pros/Cons ϚϧνΫϥ΢υߏ੒ Cons 1. ӡ༻ͷख͕ؒ૿͑Δ
  15. ɹɹʛɹɹ© 2019 PLAID Inc. 22 2019.11.13ɹɹʛɹɹDatadog Summit Tokyo 2019ɹɹʛɹ ৄࡉΛ஌Γ͍ͨํ͸

    ʮσϒαϛ 2018ʯͷࢿྉͰʂ ϚϧνΫϥ΢υߏ੒ https://speakerdeck.com/tikson/zhan-lue-topointonituite
  16. 23 2019.11.13ɹɹʛɹɹDatadog Summit Tokyo 2019ɹɹʛɹ ɹɹʛɹɹ© 2019 PLAID Inc. ࿩͢͜ͱ

    KARTEͷಛ௃ KARTEΛࢧ͑Δ؂ࢹͷϙΠϯτ ࠓޙ΍Γ͍ͨ͜ͱ ✓
  17. 25 ɹɹʛɹɹ© 2019 PLAID Inc. 2019.11.13ɹɹʛɹɹDatadog Summit Tokyo 2019ɹɹʛɹ KARTEͷಛ௃

    → 1.Ϛϧνςφϯτ 2.௿ϨΠςϯγʔ/ߴෛՙ 3.େن໛γεςϜ 4.ϚϧνΫϥ΢υߏ੒
  18. 26 ɹɹʛɹɹ© 2019 PLAID Inc. 2019.11.13ɹɹʛɹɹDatadog Summit Tokyo 2019ɹɹʛɹ KARTEΛࢧ͑Δ

    ؂ࢹͷϙΠϯτ 1.Datadogʹશͯू໿ 2.Custom metricsͷੵۃ׆༻ 3.LayerΛ෼͚ͨDashboard ࿩͢͜ͱ →
  19. 27 ɹɹʛɹɹ© 2019 PLAID Inc. 2019.11.13ɹɹʛɹɹDatadog Summit Tokyo 2019ɹɹʛɹ →

    1.Datadogʹશͯू໿ 2.Custom metricsͷੵۃ׆༻ 3.LayerΛ෼͚ͨDashboard 1.Ϛϧνςφϯτ 2.௿ϨΠςϯγʔ/ߴෛՙ 3.େن໛γεςϜ 4.ϚϧνΫϥ΢υߏ੒
  20. 29 ɹɹʛɹɹ© 2019 PLAID Inc. 2019.11.13ɹɹʛɹɹDatadog Summit Tokyo 2019ɹɹʛɹ KARTEΛࢧ͑Δ

    ؂ࢹͷϙΠϯτ ࿩͢͜ͱ → 1.Datadogʹશͯू໿ 2.Custom metricsͷੵۃ׆༻ 3.LayerΛ෼͚ͨDashboard
  21. 30 ɹɹʛɹɹ© 2019 PLAID Inc. 2019.11.13ɹɹʛɹɹDatadog Summit Tokyo 2019ɹɹʛɹ →

    1.Datadogʹશͯू໿ 2.Custom metricsͷੵۃ׆༻ 3.LayerΛ෼͚ͨDashboard 1.Ϛϧνςφϯτ 2.௿ϨΠςϯγʔ/ߴෛՙ 3.େن໛γεςϜ 4.ϚϧνΫϥ΢υߏ੒
  22. Custom metricsͷੵۃ׆༻ 32 ɹɹʛɹɹ© 2019 PLAID Inc. 2019.11.13ɹɹʛɹɹDatadog Summit Tokyo

    2019ɹɹʛɹ https://github.com/makinoy/libs-dogstatsd ɾlibs-dogstatsdͰɺ؆୯ʹॲཧͷ࣮ߦճ਺΍ ࣮ߦ࣌ؒΛଌఆͰ͖ΔΑ͏ʹ ɾؔ਺ΑΓ΋ࡉ͔͍୯ҐͰlatencyΛଌఆ ɾϢʔβ୯ҐͰͷύϑΥʔϚϯεΛଌఆ
 1 const dd = require('libs-dogstatsd'); 2 const stats = dd.start(); 3 // measure hoge 4 hoge(); 5 stats.tick('hoge', 1, 1, [user_id:1234]); Node.jsͷwrapper library
  23. ɹɹʛɹɹ© 2019 PLAID Inc. 33 2019.11.13ɹɹʛɹɹDatadog Summit Tokyo 2019ɹɹʛɹ tagͰϢʔβࣝผ༻IDΛૹΔ

    ࠓى͍ͬͯ͜Δ໰୊͕ ʮಛఆͷϢʔβىҼͰى͍ͬͯ͜Δʁʯ ʮKARTEશମͰى͍ͬͯ͜Δʁʯ ͔Λ੾Γ෼͚͠΍͍ͯ͘͢͠Δ Custom metricsͷੵۃ׆༻
  24. 34 ɹɹʛɹɹ© 2019 PLAID Inc. 2019.11.13ɹɹʛɹɹDatadog Summit Tokyo 2019ɹɹʛɹ KARTEΛࢧ͑Δ

    ؂ࢹͷϙΠϯτ ࿩͢͜ͱ → 1.Datadogʹશͯू໿ 2.Custom metricsͷੵۃ׆༻ 3.LayerΛ෼͚ͨDashboard
  25. 35 ɹɹʛɹɹ© 2019 PLAID Inc. 2019.11.13ɹɹʛɹɹDatadog Summit Tokyo 2019ɹɹʛɹ →

    1.Datadogʹશͯू໿ 2.Custom metricsͷੵۃ׆༻ 3.LayerΛ෼͚ͨDashboard 1.Ϛϧνςφϯτ 2.௿ϨΠςϯγʔ/ߴෛՙ 3.େن໛γεςϜ 4.ϚϧνΫϥ΢υߏ੒
  26. ɹɹʛɹɹ© 2019 PLAID Inc. 37 2019.11.13ɹɹʛɹɹDatadog Summit Tokyo 2019ɹɹʛɹ Layer

    2 ɾॏཁͳίϯϙʔωϯτ ɾॏཁͳΞϓϦέʔγϣϯͷϝτϦοΫ LayerΛ෼͚ͨDashboard
  27. 39 2019.11.13ɹɹʛɹɹDatadog Summit Tokyo 2019ɹɹʛɹ ɹɹʛɹɹ© 2019 PLAID Inc. ࿩͢͜ͱ

    KARTEͷಛ௃ KARTEΛࢧ͑Δ؂ࢹͷϙΠϯτ ࠓޙ΍Γ͍ͨ͜ͱ ✓ ✓
  28. ɹࠓޙ΍Γ͍ͨ͜ͱ 40 ɹɹʛɹɹ© 2019 PLAID Inc. 2019.11.13ɹɹʛɹɹDatadog Summit Tokyo 2019ɹɹʛɹ

    શͯͷΤϯδχΞ͕ ো֐ͷҰ࣌ରԠΛͰ͖ΔΑ͏ʹ͢Δ ɾ؂ࢹ/ো֐ରԠͷϑϩʔͷ੔ཧ ɾDatadogΛ࢖͍͜ͳͯ͠΋Β͏
  29. ɹࠓޙ΍Γ͍ͨ͜ͱ 41 ɹɹʛɹɹ© 2019 PLAID Inc. 2019.11.13ɹɹʛɹɹDatadog Summit Tokyo 2019ɹɹʛɹ

    Custom metrics 1ԯԁ໰୊ͷղܾ ɾ͋·ΓʹศརͩͬͨͷͰ࢖͍͢ ͗ͯ͠·ͬͨ… ɾCustom metricsͷ੔ཧΛߦ͏
  30. 43 ɹɹʛɹɹ© 2019 PLAID Inc. 2019.11.13ɹɹʛɹɹDatadog Summit Tokyo 2019ɹɹʛɹ KARTEΛࢧ͑Δ

    ؂ࢹͷϙΠϯτ 1.Datadogʹશͯू໿ 2.Custom metricsͷੵۃ׆༻ 3.LayerΛ෼͚ͨDashboard →