マルチクラウドな大規模サービスKARTEを支えるDatadog/multi-cloud monitoring with Datadog

6de30acfd736745120a3d6457dc6d45c?s=47 Daiki Matsui
November 13, 2019
Technology
0
300

マルチクラウドな大規模サービスKARTEを支えるDatadog/multi-cloud monitoring with Datadog

Datadog Summit Tokyoでの登壇資料です。
動画はhttps://www.youtube.com/watch?v=_Z4HBiwODbc

6de30acfd736745120a3d6457dc6d45c?s=128

Daiki Matsui

November 13, 2019
Tweet

More Decks by Daiki Matsui

See All by Daiki Matsui
6de30acfd736745120a3d6457dc6d45c?s=48 ikemonn
2
2.8k
6de30acfd736745120a3d6457dc6d45c?s=48 ikemonn
2
12k
6de30acfd736745120a3d6457dc6d45c?s=48 ikemonn
0
83

Other Decks in Technology

See All in Technology
7a0bc9670ff654b04d4361a9afd1bba8?s=48 hcrane
18
1.7k
5bc984d78def65048378c439d77c379f?s=48 tomoyamachi
0
130
88964b936e864ca7d326272eaa70fa9a?s=48 hgsgtk
0
700
Cd931bc05e7cee46b9a950a63e47ba4c?s=48 you
0
130
40301c0affdf359eaca771713e22b71a?s=48 harshbothra
0
580
43a37af3a99834cd9c4b15806b5b4081?s=48 yoshiokacb
0
260
8a84268593355816432ceaf78777d585?s=48 dena_tech
1
2.2k
703c34db2db0e609fddec6a5585ceccc?s=48 kenta_sato3
2
160
5059d3f370ad7e1f4d8de4be79ae1a2c?s=48 rvirus0817
2
330
A2cd163fd3cfca5204b0d8f0c17eea29?s=48 yujif
2
130
Df739988f32c42e8c318ef844d6e902b?s=48 oliver_diary
2
110
Cb88f765dd38cd942c39aacb5abbea06?s=48 ufoo68
0
110

Featured

See All Featured
5f50f94346fbcb23706f0707169317a4?s=48 aarron
254
36k
48c098b6579220a67a6ba949be6e4b5a?s=48 revolveconf
195
9.5k
1b75d89772b7eea1f6c89fbf362607d9?s=48 moore
124
21k
1177e050db6bafe62885362edf6e3537?s=48 lauravandoore
436
28k
Ecdea9b9714877b86cee08458f085481?s=48 trallard
9
380
245cee81a9c424266e5e401d844ea881?s=48 lara
13
2.3k
7fa6101cd43067653cf4ac6c7ca54305?s=48 akmur
252
19k
A16eb159db895e3b01d3dc95767ad595?s=48 jonyablonski
6
640
B83d5b590577969ee79a5ad845411a7d?s=48 ammeep
653
54k
Eb8975af8e49e19e3dd6b6b84a542e26?s=48 qrush
282
18k
Dafc4723e9a1c067796c0fec6f509774?s=48 lemiorhan
621
40k
Ef32e0b1ac5082450dc8c1fca3a26b5c?s=48 cherdarchuk
66
250k

Transcript

  1. 1 ϚϧνΫϥ΢υͳେن໛αʔϏε KARTEΛࢧ͑ΔDatadog ɹɹʛɹɹ© 2019 PLAID Inc. 2019.11.13ɹɹʛɹɹDatadog Summit Tokyo

    2019ɹɹʛɹ PLAID, Inc. Daiki Matsui

  2. 2 2019.11.13ɹɹʛɹɹDatadog Summit Tokyo 2019ɹɹʛɹ ɹɹʛɹɹ© 2019 PLAID Inc. ࣗݾ঺հ

    দҪ େٍ גࣜձࣾϓϨΠυ SRE Senior Architect 2012೥-2015೥·ͰגࣜձࣾαΠόʔυʹͯήʔϜ։ൃʹैࣄ 2015೥ΑΓϓϨΠυʹࢀը͠ɺSREͱͯ͠ϚϧνΫϥ΢υΠϯϑϥͷߏஙɾӡ༻౳Λ୲౰ 2015೥ೖࣾ

  3. ɹɹʛɹɹ© 2019 PLAID Inc. 3 2019.11.13ɹɹʛɹɹDatadog Summit Tokyo 2019ɹɹʛɹ גࣜձࣾϓϨΠυʹ͍ͭͯ

    גࣜձࣾϓϨΠυ ౦ژ౎தԝ۠ۜ࠲6-10-1 GINZA SIX 10F ઃཱɿ2011೥10݄ ैۀһɿ130໊ ࢿຊۚɿ149,987,878ԁ ※ࢿຊ४උؚۚΉ

  4. 4 2019.11.13ɹɹʛɹɹDatadog Summit Tokyo 2019ɹɹʛɹ ɹɹʛɹɹ© 2019 PLAID Inc. KARTEʹ͍ͭͯ

    ֬ೝ͢Δ جຊαʔϏε΍͝ར༻ํ๏Λ͝঺հ͠·͢ɻ ͓ಘͳಛయ΍໾ཱͭ৘ใ͕ຬࡌͰ͢ɻ ॳΊͯͷํ΁ ְתּׅ然钠ׅ׷ 6*رؠ؎ش٦׌ֽוזַזַ♳麦׃זְծ➙ ״׶إٔؗ،حف׃׋ְהְֲ倯䗳铣דׅկ UIرؠ؎ش٦ָ濼׏גֶֻץֹ 7אךرؠ؎ٕٝ٦ٕ DESIGN RULES » CHECK ׆׏ה⢪ִ׷،؎ذي׌ֽ׾䲧ִת׃׋կ ֿך堣⠓׾ֶ鋅鷕׃זֻկ 窫㼎ծ 妜׃ְ ౙͷओ໾ɺ Ξ΢λʔ COLLECTION OUTER 嗚稊勴⟝׾㼰׃㢌刿ׅ׷׌ֽדծ֮ז׋ך椚 䟝ך暟⟝ח⳿⠓ִ׷〳腉䚍ָ넝ֻז׶תׅկ 勴⟝׾㢌ִג嗚稊׃ג׫גֻ׌ְׁկ ׀䋞劄ך暟⟝כ 鋅אַ׶תׇ׿ד׃׋ַ ꟗׄ׷ ҰਓҰਓʹ߹Θͤͨ ސ٬ମݧΛఏڙ WebαΠτͷ๚໰ऀͷߦಈΛ
 ސ٬͝ͱʹϦΞϧλΠϜʹղੳ $9 ސ٬ମݧ ϓϥοτϑΥʔϜ

  5. 5 2019.11.13ɹɹʛɹɹDatadog Summit Tokyo 2019ɹɹʛɹ ɹɹʛɹɹ© 2019 PLAID Inc. ࿩͢͜ͱ

    KARTEͷಛ௃ KARTEΛࢧ͑Δ؂ࢹͷϙΠϯτ ࠓޙ΍Γ͍ͨ͜ͱ 1 2 3

  6. 6 2019.11.13ɹɹʛɹɹDatadog Summit Tokyo 2019ɹɹʛɹ ɹɹʛɹɹ© 2019 PLAID Inc. ࿩͢͜ͱ

    KARTEͷಛ௃ KARTEΛࢧ͑Δ؂ࢹͷϙΠϯτ ࠓޙ΍Γ͍ͨ͜ͱ

  7. 7 ɹɹʛɹɹ© 2019 PLAID Inc. 2019.11.13ɹɹʛɹɹDatadog Summit Tokyo 2019ɹɹʛɹ KARTEͷಛ௃

    1.Ϛϧνςφϯτ 2.௿ϨΠςϯγʔ/ߴෛՙ 3.େن໛γεςϜ 4.ϚϧνΫϥ΢υߏ੒ ࿩͢͜ͱ →

  8. 8 ɹɹʛɹɹ© 2019 PLAID Inc. 2019.11.13ɹɹʛɹɹDatadog Summit Tokyo 2019ɹɹʛɹ KARTEͷಛ௃

    1.Ϛϧνςφϯτ 2.௿ϨΠςϯγʔ/ߴෛՙ 3.େن໛γεςϜ 4.ϚϧνΫϥ΢υߏ੒ ࿩͢͜ͱ →

  9. 9 2019.11.13ɹɹʛɹɹDatadog Summit Tokyo 2019ɹɹʛɹ ɹɹʛɹɹ© 2019 PLAID Inc. Ϛϧνςφϯτ

  10. ɹɹʛɹɹ© 2019 PLAID Inc. 10 2019.11.13ɹɹʛɹɹDatadog Summit Tokyo 2019ɹɹʛɹ ɾ༷ʑͳෛՙͷ͓٬͞Μ͕͍Δ

    ɾΞΫηεύλʔϯ΋৭ʑ Ϛϧνςφϯτ

  11. ɹɹʛɹɹ© 2019 PLAID Inc. 11 2019.11.13ɹɹʛɹɹDatadog Summit Tokyo 2019ɹɹʛɹ ɾෛՙ͕1೔Ͱ໿10ഒมԽ͢Δ

    ɾηʔϧ౳Ͱෛՙ͕ٸ૿͢Δ Ϛϧνςφϯτ 1೔ͷෛՙਪҠ

  12. 12 ɹɹʛɹɹ© 2019 PLAID Inc. 2019.11.13ɹɹʛɹɹDatadog Summit Tokyo 2019ɹɹʛɹ KARTEͷಛ௃

    1.Ϛϧνςφϯτ 2.௿ϨΠςϯγʔ/ߴෛՙ 3.େن໛γεςϜ 4.ϚϧνΫϥ΢υߏ੒ ࿩͢͜ͱ →

  13. 13 2019.11.13ɹɹʛɹɹDatadog Summit Tokyo 2019ɹɹʛɹ ɹɹʛɹɹ© 2019 PLAID Inc. ௿ϨΠςϯγʔɺߴෛՙ

    sec to real-time actions 0.x events per day 3 billion stored in BigQuery 2+ PB events (rows) / sec 65,000 streaming insert per month 1 PB analysis per month 60+ PB SRE 3 ௿ϨΠςϯγʔ/ߴෛՙ

  14. 14 2019.11.13ɹɹʛɹɹDatadog Summit Tokyo 2019ɹɹʛɹ ɹɹʛɹɹ© 2019 PLAID Inc. ௿ϨΠςϯγʔɺߴෛՙ

    sec to real-time actions 0.x events per day 3 billion stored in BigQuery 2+ PB events (rows) / sec 65,000 streaming insert per month 1 PB analysis per month 60+ PB SRE 3 ௿ϨΠςϯγʔ/ߴෛՙ

  15. 15 2019.11.13ɹɹʛɹɹDatadog Summit Tokyo 2019ɹɹʛɹ ɹɹʛɹɹ© 2019 PLAID Inc. ௿ϨΠςϯγʔɺߴෛՙ

  16. 16 2019.11.13ɹɹʛɹɹDatadog Summit Tokyo 2019ɹɹʛɹ ɹɹʛɹɹ© 2019 PLAID Inc. ௿ϨΠςϯγʔɺߴෛՙ

    sec to real-time actions 0.x events per day 3 billion stored in BigQuery 2+ PB events (rows) / sec 65,000 streaming insert per month 1 PB analysis per month 60+ PB SRE 3 ௿ϨΠςϯγʔ/ߴෛՙ

  17. 17 ɹɹʛɹɹ© 2019 PLAID Inc. 2019.11.13ɹɹʛɹɹDatadog Summit Tokyo 2019ɹɹʛɹ KARTEͷಛ௃

    1.Ϛϧνςφϯτ 2.௿ϨΠςϯγʔ/ߴෛՙ 3.େن໛γεςϜ 4.ϚϧνΫϥ΢υߏ੒ ࿩͢͜ͱ →

  18. 18 2019.11.13ɹɹʛɹɹDatadog Summit Tokyo 2019ɹɹʛɹ ɹɹʛɹɹ© 2019 PLAID Inc. େن໛γεςϜ

  19. 19 ɹɹʛɹɹ© 2019 PLAID Inc. 2019.11.13ɹɹʛɹɹDatadog Summit Tokyo 2019ɹɹʛɹ KARTEͷಛ௃

    1.Ϛϧνςφϯτ 2.௿ϨΠςϯγʔ/ߴෛՙ 3.େن໛γεςϜ 4.ϚϧνΫϥ΢υߏ੒ ࿩͢͜ͱ →

  20. 20 2019.11.13ɹɹʛɹɹDatadog Summit Tokyo 2019ɹɹʛɹ ɹɹʛɹɹ© 2019 PLAID Inc. ϚϧνΫϥ΢υߏ੒

  21. ɹɹʛɹɹ© 2019 PLAID Inc. 21 2019.11.13ɹɹʛɹɹDatadog Summit Tokyo 2019ɹɹʛɹ Pros

    1. ֤ϓϩόΠμʔͷྑ͍ͱ͜औΓ͕Ͱ͖Δ 2. ϓϩόΠμʔͷো֐ΛผϓϩόΠμʔͰΧ όʔͰ͖Δ Pros/Cons ϚϧνΫϥ΢υߏ੒ Cons 1. ӡ༻ͷख͕ؒ૿͑Δ

  22. ɹɹʛɹɹ© 2019 PLAID Inc. 22 2019.11.13ɹɹʛɹɹDatadog Summit Tokyo 2019ɹɹʛɹ ৄࡉΛ஌Γ͍ͨํ͸

    ʮσϒαϛ 2018ʯͷࢿྉͰʂ ϚϧνΫϥ΢υߏ੒ https://speakerdeck.com/tikson/zhan-lue-topointonituite

  23. 23 2019.11.13ɹɹʛɹɹDatadog Summit Tokyo 2019ɹɹʛɹ ɹɹʛɹɹ© 2019 PLAID Inc. ࿩͢͜ͱ

    KARTEͷಛ௃ KARTEΛࢧ͑Δ؂ࢹͷϙΠϯτ ࠓޙ΍Γ͍ͨ͜ͱ ✓

  24. KARTEΛࢧ͑Δ؂ࢹͷϙΠϯτ 24 ɹɹʛɹɹ© 2019 PLAID Inc. 2019.11.13ɹɹʛɹɹDatadog Summit Tokyo 2019ɹɹʛɹ

    ɾϦΞϧλΠϜੑ ɾॊೈੑ ɾεέʔϥϏϦςΟ ɾ҆ఆੑ

  25. 25 ɹɹʛɹɹ© 2019 PLAID Inc. 2019.11.13ɹɹʛɹɹDatadog Summit Tokyo 2019ɹɹʛɹ KARTEͷಛ௃

    → 1.Ϛϧνςφϯτ 2.௿ϨΠςϯγʔ/ߴෛՙ 3.େن໛γεςϜ 4.ϚϧνΫϥ΢υߏ੒

  26. 26 ɹɹʛɹɹ© 2019 PLAID Inc. 2019.11.13ɹɹʛɹɹDatadog Summit Tokyo 2019ɹɹʛɹ KARTEΛࢧ͑Δ

    ؂ࢹͷϙΠϯτ 1.Datadogʹશͯू໿ 2.Custom metricsͷੵۃ׆༻ 3.LayerΛ෼͚ͨDashboard ࿩͢͜ͱ →

  27. 27 ɹɹʛɹɹ© 2019 PLAID Inc. 2019.11.13ɹɹʛɹɹDatadog Summit Tokyo 2019ɹɹʛɹ →

    1.Datadogʹશͯू໿ 2.Custom metricsͷੵۃ׆༻ 3.LayerΛ෼͚ͨDashboard 1.Ϛϧνςφϯτ 2.௿ϨΠςϯγʔ/ߴෛՙ 3.େن໛γεςϜ 4.ϚϧνΫϥ΢υߏ੒

  28. ɹɹʛɹɹ© 2019 PLAID Inc. 28 2019.11.13ɹɹʛɹɹDatadog Summit Tokyo 2019ɹɹʛɹ ো֐ରԠ࣌ʹ͸DatadogΛݟΕ͹

    ͙͢ʹঢ়گΛ೺ѲͰ͖ΔΑ͏ʹ Datadogʹશͯू໿

  29. 29 ɹɹʛɹɹ© 2019 PLAID Inc. 2019.11.13ɹɹʛɹɹDatadog Summit Tokyo 2019ɹɹʛɹ KARTEΛࢧ͑Δ

    ؂ࢹͷϙΠϯτ ࿩͢͜ͱ → 1.Datadogʹશͯू໿ 2.Custom metricsͷੵۃ׆༻ 3.LayerΛ෼͚ͨDashboard

  30. 30 ɹɹʛɹɹ© 2019 PLAID Inc. 2019.11.13ɹɹʛɹɹDatadog Summit Tokyo 2019ɹɹʛɹ →

    1.Datadogʹશͯू໿ 2.Custom metricsͷੵۃ׆༻ 3.LayerΛ෼͚ͨDashboard 1.Ϛϧνςφϯτ 2.௿ϨΠςϯγʔ/ߴෛՙ 3.େن໛γεςϜ 4.ϚϧνΫϥ΢υߏ੒

  31. ɹɹʛɹɹ© 2019 PLAID Inc. 31 2019.11.13ɹɹʛɹɹDatadog Summit Tokyo 2019ɹɹʛɹ 32M/mon

    ΄Ͳར༻ Custom metricsͷੵۃ׆༻

  32. Custom metricsͷੵۃ׆༻ 32 ɹɹʛɹɹ© 2019 PLAID Inc. 2019.11.13ɹɹʛɹɹDatadog Summit Tokyo

    2019ɹɹʛɹ https://github.com/makinoy/libs-dogstatsd ɾlibs-dogstatsdͰɺ؆୯ʹॲཧͷ࣮ߦճ਺΍ ࣮ߦ࣌ؒΛଌఆͰ͖ΔΑ͏ʹ ɾؔ਺ΑΓ΋ࡉ͔͍୯ҐͰlatencyΛଌఆ ɾϢʔβ୯ҐͰͷύϑΥʔϚϯεΛଌఆ
 1 const dd = require('libs-dogstatsd'); 2 const stats = dd.start(); 3 // measure hoge 4 hoge(); 5 stats.tick('hoge', 1, 1, [user_id:1234]); Node.jsͷwrapper library

  33. ɹɹʛɹɹ© 2019 PLAID Inc. 33 2019.11.13ɹɹʛɹɹDatadog Summit Tokyo 2019ɹɹʛɹ tagͰϢʔβࣝผ༻IDΛૹΔ

    ࠓى͍ͬͯ͜Δ໰୊͕ ʮಛఆͷϢʔβىҼͰى͍ͬͯ͜Δʁʯ ʮKARTEશମͰى͍ͬͯ͜Δʁʯ ͔Λ੾Γ෼͚͠΍͍ͯ͘͢͠Δ Custom metricsͷੵۃ׆༻

  34. 34 ɹɹʛɹɹ© 2019 PLAID Inc. 2019.11.13ɹɹʛɹɹDatadog Summit Tokyo 2019ɹɹʛɹ KARTEΛࢧ͑Δ

    ؂ࢹͷϙΠϯτ ࿩͢͜ͱ → 1.Datadogʹશͯू໿ 2.Custom metricsͷੵۃ׆༻ 3.LayerΛ෼͚ͨDashboard

  35. 35 ɹɹʛɹɹ© 2019 PLAID Inc. 2019.11.13ɹɹʛɹɹDatadog Summit Tokyo 2019ɹɹʛɹ →

    1.Datadogʹશͯू໿ 2.Custom metricsͷੵۃ׆༻ 3.LayerΛ෼͚ͨDashboard 1.Ϛϧνςφϯτ 2.௿ϨΠςϯγʔ/ߴෛՙ 3.େن໛γεςϜ 4.ϚϧνΫϥ΢υߏ੒

  36. ɹɹʛɹɹ© 2019 PLAID Inc. 36 2019.11.13ɹɹʛɹɹDatadog Summit Tokyo 2019ɹɹʛɹ Layer

    1 ɾKARTE͕ੜ͖͍ͯΔ͔Ͳ͏͔ LayerΛ෼͚ͨDashboard

  37. ɹɹʛɹɹ© 2019 PLAID Inc. 37 2019.11.13ɹɹʛɹɹDatadog Summit Tokyo 2019ɹɹʛɹ Layer

    2 ɾॏཁͳίϯϙʔωϯτ ɾॏཁͳΞϓϦέʔγϣϯͷϝτϦοΫ LayerΛ෼͚ͨDashboard

  38. ɹɹʛɹɹ© 2019 PLAID Inc. 38 2019.11.13ɹɹʛɹɹDatadog Summit Tokyo 2019ɹɹʛɹ Layer

    3 ɾ֤αʔό܈ͷࡉ͔͍ϝτϦοΫ LayerΛ෼͚ͨDashboard

  39. 39 2019.11.13ɹɹʛɹɹDatadog Summit Tokyo 2019ɹɹʛɹ ɹɹʛɹɹ© 2019 PLAID Inc. ࿩͢͜ͱ

    KARTEͷಛ௃ KARTEΛࢧ͑Δ؂ࢹͷϙΠϯτ ࠓޙ΍Γ͍ͨ͜ͱ ✓ ✓

  40. ɹࠓޙ΍Γ͍ͨ͜ͱ 40 ɹɹʛɹɹ© 2019 PLAID Inc. 2019.11.13ɹɹʛɹɹDatadog Summit Tokyo 2019ɹɹʛɹ

    શͯͷΤϯδχΞ͕ ো֐ͷҰ࣌ରԠΛͰ͖ΔΑ͏ʹ͢Δ ɾ؂ࢹ/ো֐ରԠͷϑϩʔͷ੔ཧ ɾDatadogΛ࢖͍͜ͳͯ͠΋Β͏

  41. ɹࠓޙ΍Γ͍ͨ͜ͱ 41 ɹɹʛɹɹ© 2019 PLAID Inc. 2019.11.13ɹɹʛɹɹDatadog Summit Tokyo 2019ɹɹʛɹ

    Custom metrics 1ԯԁ໰୊ͷղܾ ɾ͋·ΓʹศརͩͬͨͷͰ࢖͍͢ ͗ͯ͠·ͬͨ… ɾCustom metricsͷ੔ཧΛߦ͏

  42. 42 2019.11.13ɹɹʛɹɹDatadog Summit Tokyo 2019ɹɹʛɹ ɹɹʛɹɹ© 2019 PLAID Inc. ·ͱΊ

    ·ͱΊ

  43. 43 ɹɹʛɹɹ© 2019 PLAID Inc. 2019.11.13ɹɹʛɹɹDatadog Summit Tokyo 2019ɹɹʛɹ KARTEΛࢧ͑Δ

    ؂ࢹͷϙΠϯτ 1.Datadogʹશͯू໿ 2.Custom metricsͷੵۃ׆༻ 3.LayerΛ෼͚ͨDashboard →
  44. None