マルチクラウドな大規模サービスKARTEを支えるDatadog/multi-cloud monitoring with Datadog

Datadog Summit Tokyoでの登壇資料です。
動画はhttps://www.youtube.com/watch?v=_Z4HBiwODbc

Transcript

1. 1 ϚϧνΫϥ΢υͳେن໛αʔϏε KARTEΛࢧ͑ΔDatadog ɹɹʛɹɹ© 2019 PLAID Inc. 2019.11.13ɹɹʛɹɹDatadog Summit Tokyo 2019ɹɹʛɹ PLAID, Inc. Daiki Matsui
2. 2 2019.11.13ɹɹʛɹɹDatadog Summit Tokyo 2019ɹɹʛɹ ɹɹʛɹɹ© 2019 PLAID Inc. ࣗݾ঺հ দҪ େٍ גࣜձࣾϓϨΠυ SRE Senior Architect 2012೥-2015೥·ͰגࣜձࣾαΠόʔυʹͯήʔϜ։ൃʹैࣄ 2015೥ΑΓϓϨΠυʹࢀը͠ɺSREͱͯ͠ϚϧνΫϥ΢υΠϯϑϥͷߏஙɾӡ༻౳Λ୲౰ 2015೥ೖࣾ
3. ɹɹʛɹɹ© 2019 PLAID Inc. 3 2019.11.13ɹɹʛɹɹDatadog Summit Tokyo 2019ɹɹʛɹ גࣜձࣾϓϨΠυʹ͍ͭͯ גࣜձࣾϓϨΠυ ౦ژ౎தԝ۠ۜ࠲6-10-1 GINZA SIX 10F ઃཱɿ2011೥10݄ ैۀһɿ130໊ ࢿຊۚɿ149,987,878ԁ ※ࢿຊ४උؚۚΉ
4. 4 2019.11.13ɹɹʛɹɹDatadog Summit Tokyo 2019ɹɹʛɹ ɹɹʛɹɹ© 2019 PLAID Inc. KARTEʹ͍ͭͯ ֬ೝ͢Δ جຊαʔϏε΍͝ར༻ํ๏Λ͝঺հ͠·͢ɻ ͓ಘͳಛయ΍໾ཱͭ৘ใ͕ຬࡌͰ͢ɻ ॳΊͯͷํ΁ ְתּׅ然钠ׅ׷ 6*رؠ؎ش٦׌ֽוזַזַ♳麦׃זְծ➙ ״׶إٔؗ،حف׃׋ְהְֲ倯䗳铣דׅկ UIرؠ؎ش٦ָ濼׏גֶֻץֹ 7אךرؠ؎ٕٝ٦ٕ DESIGN RULES » CHECK ׆׏ה⢪ִ׷،؎ذي׌ֽ׾䲧ִת׃׋կ ֿך堣⠓׾ֶ鋅鷕׃זֻկ 窫㼎ծ 妜׃ְ ౙͷओ໾ɺ Ξ΢λʔ COLLECTION OUTER 嗚稊勴⟝׾㼰׃㢌刿ׅ׷׌ֽדծ֮ז׋ך椚 䟝ך暟⟝ח⳿⠓ִ׷〳腉䚍ָ넝ֻז׶תׅկ 勴⟝׾㢌ִג嗚稊׃ג׫גֻ׌ְׁկ ׀䋞劄ך暟⟝כ 鋅אַ׶תׇ׿ד׃׋ַ ꟗׄ׷ ҰਓҰਓʹ߹Θͤͨ ސ٬ମݧΛఏڙ WebαΠτͷ๚໰ऀͷߦಈΛ
 ސ٬͝ͱʹϦΞϧλΠϜʹղੳ $9 ސ٬ମݧ ϓϥοτϑΥʔϜ
5. 5 2019.11.13ɹɹʛɹɹDatadog Summit Tokyo 2019ɹɹʛɹ ɹɹʛɹɹ© 2019 PLAID Inc. ࿩͢͜ͱ KARTEͷಛ௃ KARTEΛࢧ͑Δ؂ࢹͷϙΠϯτ ࠓޙ΍Γ͍ͨ͜ͱ 1 2 3
6. 6 2019.11.13ɹɹʛɹɹDatadog Summit Tokyo 2019ɹɹʛɹ ɹɹʛɹɹ© 2019 PLAID Inc. ࿩͢͜ͱ KARTEͷಛ௃ KARTEΛࢧ͑Δ؂ࢹͷϙΠϯτ ࠓޙ΍Γ͍ͨ͜ͱ
7. 7 ɹɹʛɹɹ© 2019 PLAID Inc. 2019.11.13ɹɹʛɹɹDatadog Summit Tokyo 2019ɹɹʛɹ KARTEͷಛ௃ 1.Ϛϧνςφϯτ 2.௿ϨΠςϯγʔ/ߴෛՙ 3.େن໛γεςϜ 4.ϚϧνΫϥ΢υߏ੒ ࿩͢͜ͱ →
8. 8 ɹɹʛɹɹ© 2019 PLAID Inc. 2019.11.13ɹɹʛɹɹDatadog Summit Tokyo 2019ɹɹʛɹ KARTEͷಛ௃ 1.Ϛϧνςφϯτ 2.௿ϨΠςϯγʔ/ߴෛՙ 3.େن໛γεςϜ 4.ϚϧνΫϥ΢υߏ੒ ࿩͢͜ͱ →
9. 9 2019.11.13ɹɹʛɹɹDatadog Summit Tokyo 2019ɹɹʛɹ ɹɹʛɹɹ© 2019 PLAID Inc. Ϛϧνςφϯτ
10. ɹɹʛɹɹ© 2019 PLAID Inc. 10 2019.11.13ɹɹʛɹɹDatadog Summit Tokyo 2019ɹɹʛɹ ɾ༷ʑͳෛՙͷ͓٬͞Μ͕͍Δ ɾΞΫηεύλʔϯ΋৭ʑ Ϛϧνςφϯτ
11. ɹɹʛɹɹ© 2019 PLAID Inc. 11 2019.11.13ɹɹʛɹɹDatadog Summit Tokyo 2019ɹɹʛɹ ɾෛՙ͕1೔Ͱ໿10ഒมԽ͢Δ ɾηʔϧ౳Ͱෛՙ͕ٸ૿͢Δ Ϛϧνςφϯτ 1೔ͷෛՙਪҠ
12. 12 ɹɹʛɹɹ© 2019 PLAID Inc. 2019.11.13ɹɹʛɹɹDatadog Summit Tokyo 2019ɹɹʛɹ KARTEͷಛ௃ 1.Ϛϧνςφϯτ 2.௿ϨΠςϯγʔ/ߴෛՙ 3.େن໛γεςϜ 4.ϚϧνΫϥ΢υߏ੒ ࿩͢͜ͱ →
13. 13 2019.11.13ɹɹʛɹɹDatadog Summit Tokyo 2019ɹɹʛɹ ɹɹʛɹɹ© 2019 PLAID Inc. ௿ϨΠςϯγʔɺߴෛՙ sec to real-time actions 0.x events per day 3 billion stored in BigQuery 2+ PB events (rows) / sec 65,000 streaming insert per month 1 PB analysis per month 60+ PB SRE 3 ௿ϨΠςϯγʔ/ߴෛՙ
14. 14 2019.11.13ɹɹʛɹɹDatadog Summit Tokyo 2019ɹɹʛɹ ɹɹʛɹɹ© 2019 PLAID Inc. ௿ϨΠςϯγʔɺߴෛՙ sec to real-time actions 0.x events per day 3 billion stored in BigQuery 2+ PB events (rows) / sec 65,000 streaming insert per month 1 PB analysis per month 60+ PB SRE 3 ௿ϨΠςϯγʔ/ߴෛՙ
15. 15 2019.11.13ɹɹʛɹɹDatadog Summit Tokyo 2019ɹɹʛɹ ɹɹʛɹɹ© 2019 PLAID Inc. ௿ϨΠςϯγʔɺߴෛՙ
16. 16 2019.11.13ɹɹʛɹɹDatadog Summit Tokyo 2019ɹɹʛɹ ɹɹʛɹɹ© 2019 PLAID Inc. ௿ϨΠςϯγʔɺߴෛՙ sec to real-time actions 0.x events per day 3 billion stored in BigQuery 2+ PB events (rows) / sec 65,000 streaming insert per month 1 PB analysis per month 60+ PB SRE 3 ௿ϨΠςϯγʔ/ߴෛՙ
17. 17 ɹɹʛɹɹ© 2019 PLAID Inc. 2019.11.13ɹɹʛɹɹDatadog Summit Tokyo 2019ɹɹʛɹ KARTEͷಛ௃ 1.Ϛϧνςφϯτ 2.௿ϨΠςϯγʔ/ߴෛՙ 3.େن໛γεςϜ 4.ϚϧνΫϥ΢υߏ੒ ࿩͢͜ͱ →
18. 18 2019.11.13ɹɹʛɹɹDatadog Summit Tokyo 2019ɹɹʛɹ ɹɹʛɹɹ© 2019 PLAID Inc. େن໛γεςϜ
19. 19 ɹɹʛɹɹ© 2019 PLAID Inc. 2019.11.13ɹɹʛɹɹDatadog Summit Tokyo 2019ɹɹʛɹ KARTEͷಛ௃ 1.Ϛϧνςφϯτ 2.௿ϨΠςϯγʔ/ߴෛՙ 3.େن໛γεςϜ 4.ϚϧνΫϥ΢υߏ੒ ࿩͢͜ͱ →
20. 20 2019.11.13ɹɹʛɹɹDatadog Summit Tokyo 2019ɹɹʛɹ ɹɹʛɹɹ© 2019 PLAID Inc. ϚϧνΫϥ΢υߏ੒
21. ɹɹʛɹɹ© 2019 PLAID Inc. 21 2019.11.13ɹɹʛɹɹDatadog Summit Tokyo 2019ɹɹʛɹ Pros 1. ֤ϓϩόΠμʔͷྑ͍ͱ͜औΓ͕Ͱ͖Δ 2. ϓϩόΠμʔͷো֐ΛผϓϩόΠμʔͰΧ όʔͰ͖Δ Pros/Cons ϚϧνΫϥ΢υߏ੒ Cons 1. ӡ༻ͷख͕ؒ૿͑Δ
22. ɹɹʛɹɹ© 2019 PLAID Inc. 22 2019.11.13ɹɹʛɹɹDatadog Summit Tokyo 2019ɹɹʛɹ ৄࡉΛ஌Γ͍ͨํ͸ ʮσϒαϛ 2018ʯͷࢿྉͰʂ ϚϧνΫϥ΢υߏ੒ https://speakerdeck.com/tikson/zhan-lue-topointonituite
23. 23 2019.11.13ɹɹʛɹɹDatadog Summit Tokyo 2019ɹɹʛɹ ɹɹʛɹɹ© 2019 PLAID Inc. ࿩͢͜ͱ KARTEͷಛ௃ KARTEΛࢧ͑Δ؂ࢹͷϙΠϯτ ࠓޙ΍Γ͍ͨ͜ͱ ✓
24. KARTEΛࢧ͑Δ؂ࢹͷϙΠϯτ 24 ɹɹʛɹɹ© 2019 PLAID Inc. 2019.11.13ɹɹʛɹɹDatadog Summit Tokyo 2019ɹɹʛɹ ɾϦΞϧλΠϜੑ ɾॊೈੑ ɾεέʔϥϏϦςΟ ɾ҆ఆੑ
25. 25 ɹɹʛɹɹ© 2019 PLAID Inc. 2019.11.13ɹɹʛɹɹDatadog Summit Tokyo 2019ɹɹʛɹ KARTEͷಛ௃ → 1.Ϛϧνςφϯτ 2.௿ϨΠςϯγʔ/ߴෛՙ 3.େن໛γεςϜ 4.ϚϧνΫϥ΢υߏ੒
26. 26 ɹɹʛɹɹ© 2019 PLAID Inc. 2019.11.13ɹɹʛɹɹDatadog Summit Tokyo 2019ɹɹʛɹ KARTEΛࢧ͑Δ ؂ࢹͷϙΠϯτ 1.Datadogʹશͯू໿ 2.Custom metricsͷੵۃ׆༻ 3.LayerΛ෼͚ͨDashboard ࿩͢͜ͱ →
27. 27 ɹɹʛɹɹ© 2019 PLAID Inc. 2019.11.13ɹɹʛɹɹDatadog Summit Tokyo 2019ɹɹʛɹ → 1.Datadogʹશͯू໿ 2.Custom metricsͷੵۃ׆༻ 3.LayerΛ෼͚ͨDashboard 1.Ϛϧνςφϯτ 2.௿ϨΠςϯγʔ/ߴෛՙ 3.େن໛γεςϜ 4.ϚϧνΫϥ΢υߏ੒
28. ɹɹʛɹɹ© 2019 PLAID Inc. 28 2019.11.13ɹɹʛɹɹDatadog Summit Tokyo 2019ɹɹʛɹ ো֐ରԠ࣌ʹ͸DatadogΛݟΕ͹ ͙͢ʹঢ়گΛ೺ѲͰ͖ΔΑ͏ʹ Datadogʹશͯू໿
29. 29 ɹɹʛɹɹ© 2019 PLAID Inc. 2019.11.13ɹɹʛɹɹDatadog Summit Tokyo 2019ɹɹʛɹ KARTEΛࢧ͑Δ ؂ࢹͷϙΠϯτ ࿩͢͜ͱ → 1.Datadogʹશͯू໿ 2.Custom metricsͷੵۃ׆༻ 3.LayerΛ෼͚ͨDashboard
30. 30 ɹɹʛɹɹ© 2019 PLAID Inc. 2019.11.13ɹɹʛɹɹDatadog Summit Tokyo 2019ɹɹʛɹ → 1.Datadogʹશͯू໿ 2.Custom metricsͷੵۃ׆༻ 3.LayerΛ෼͚ͨDashboard 1.Ϛϧνςφϯτ 2.௿ϨΠςϯγʔ/ߴෛՙ 3.େن໛γεςϜ 4.ϚϧνΫϥ΢υߏ੒
31. ɹɹʛɹɹ© 2019 PLAID Inc. 31 2019.11.13ɹɹʛɹɹDatadog Summit Tokyo 2019ɹɹʛɹ 32M/mon ΄Ͳར༻ Custom metricsͷੵۃ׆༻
32. Custom metricsͷੵۃ׆༻ 32 ɹɹʛɹɹ© 2019 PLAID Inc. 2019.11.13ɹɹʛɹɹDatadog Summit Tokyo 2019ɹɹʛɹ https://github.com/makinoy/libs-dogstatsd ɾlibs-dogstatsdͰɺ؆୯ʹॲཧͷ࣮ߦճ਺΍ ࣮ߦ࣌ؒΛଌఆͰ͖ΔΑ͏ʹ ɾؔ਺ΑΓ΋ࡉ͔͍୯ҐͰlatencyΛଌఆ ɾϢʔβ୯ҐͰͷύϑΥʔϚϯεΛଌఆ
 1 const dd = require('libs-dogstatsd'); 2 const stats = dd.start(); 3 // measure hoge 4 hoge(); 5 stats.tick('hoge', 1, 1, [user_id:1234]); Node.jsͷwrapper library
33. ɹɹʛɹɹ© 2019 PLAID Inc. 33 2019.11.13ɹɹʛɹɹDatadog Summit Tokyo 2019ɹɹʛɹ tagͰϢʔβࣝผ༻IDΛૹΔ ࠓى͍ͬͯ͜Δ໰୊͕ ʮಛఆͷϢʔβىҼͰى͍ͬͯ͜Δʁʯ ʮKARTEશମͰى͍ͬͯ͜Δʁʯ ͔Λ੾Γ෼͚͠΍͍ͯ͘͢͠Δ Custom metricsͷੵۃ׆༻
34. 34 ɹɹʛɹɹ© 2019 PLAID Inc. 2019.11.13ɹɹʛɹɹDatadog Summit Tokyo 2019ɹɹʛɹ KARTEΛࢧ͑Δ ؂ࢹͷϙΠϯτ ࿩͢͜ͱ → 1.Datadogʹશͯू໿ 2.Custom metricsͷੵۃ׆༻ 3.LayerΛ෼͚ͨDashboard
35. 35 ɹɹʛɹɹ© 2019 PLAID Inc. 2019.11.13ɹɹʛɹɹDatadog Summit Tokyo 2019ɹɹʛɹ → 1.Datadogʹશͯू໿ 2.Custom metricsͷੵۃ׆༻ 3.LayerΛ෼͚ͨDashboard 1.Ϛϧνςφϯτ 2.௿ϨΠςϯγʔ/ߴෛՙ 3.େن໛γεςϜ 4.ϚϧνΫϥ΢υߏ੒
36. ɹɹʛɹɹ© 2019 PLAID Inc. 36 2019.11.13ɹɹʛɹɹDatadog Summit Tokyo 2019ɹɹʛɹ Layer 1 ɾKARTE͕ੜ͖͍ͯΔ͔Ͳ͏͔ LayerΛ෼͚ͨDashboard
37. ɹɹʛɹɹ© 2019 PLAID Inc. 37 2019.11.13ɹɹʛɹɹDatadog Summit Tokyo 2019ɹɹʛɹ Layer 2 ɾॏཁͳίϯϙʔωϯτ ɾॏཁͳΞϓϦέʔγϣϯͷϝτϦοΫ LayerΛ෼͚ͨDashboard
38. ɹɹʛɹɹ© 2019 PLAID Inc. 38 2019.11.13ɹɹʛɹɹDatadog Summit Tokyo 2019ɹɹʛɹ Layer 3 ɾ֤αʔό܈ͷࡉ͔͍ϝτϦοΫ LayerΛ෼͚ͨDashboard
39. 39 2019.11.13ɹɹʛɹɹDatadog Summit Tokyo 2019ɹɹʛɹ ɹɹʛɹɹ© 2019 PLAID Inc. ࿩͢͜ͱ KARTEͷಛ௃ KARTEΛࢧ͑Δ؂ࢹͷϙΠϯτ ࠓޙ΍Γ͍ͨ͜ͱ ✓ ✓
40. ɹࠓޙ΍Γ͍ͨ͜ͱ 40 ɹɹʛɹɹ© 2019 PLAID Inc. 2019.11.13ɹɹʛɹɹDatadog Summit Tokyo 2019ɹɹʛɹ શͯͷΤϯδχΞ͕ ো֐ͷҰ࣌ରԠΛͰ͖ΔΑ͏ʹ͢Δ ɾ؂ࢹ/ো֐ରԠͷϑϩʔͷ੔ཧ ɾDatadogΛ࢖͍͜ͳͯ͠΋Β͏
41. ɹࠓޙ΍Γ͍ͨ͜ͱ 41 ɹɹʛɹɹ© 2019 PLAID Inc. 2019.11.13ɹɹʛɹɹDatadog Summit Tokyo 2019ɹɹʛɹ Custom metrics 1ԯԁ໰୊ͷղܾ ɾ͋·ΓʹศརͩͬͨͷͰ࢖͍͢ ͗ͯ͠·ͬͨ… ɾCustom metricsͷ੔ཧΛߦ͏
42. 42 2019.11.13ɹɹʛɹɹDatadog Summit Tokyo 2019ɹɹʛɹ ɹɹʛɹɹ© 2019 PLAID Inc. ·ͱΊ ·ͱΊ
43. 43 ɹɹʛɹɹ© 2019 PLAID Inc. 2019.11.13ɹɹʛɹɹDatadog Summit Tokyo 2019ɹɹʛɹ KARTEΛࢧ͑Δ ؂ࢹͷϙΠϯτ 1.Datadogʹશͯू໿ 2.Custom metricsͷੵۃ׆༻ 3.LayerΛ෼͚ͨDashboard →