Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
マルチクラウドな大規模サービスKARTEを支えるDatadog/multi-cloud mon...
Search
Daiki Matsui
November 13, 2019
Technology
0
2.3k
マルチクラウドな大規模サービスKARTEを支えるDatadog/multi-cloud monitoring with Datadog
Datadog Summit Tokyoでの登壇資料です。
動画は
https://www.youtube.com/watch?v=_Z4HBiwODbc
Daiki Matsui
November 13, 2019
Tweet
Share
More Decks by Daiki Matsui
See All by Daiki Matsui
The Fourth Age of SRE
ikemonn
0
210
Multicloud deploy with Spinnaker / builderscon tokyo 2018
ikemonn
2
4.4k
KARTE を支えるマルチプラットフォームインフラ監視 /karte-multi-platform-monitoring
ikemonn
2
15k
Node Apps Performance Tuning
ikemonn
0
130
Other Decks in Technology
See All in Technology
バクラクにおける可観測性向上の取り組み
yuu26
3
410
[JAWS-UG金沢支部×コンテナ支部合同企画]コンテナとは何か
furuton
3
230
APIテスト自動化の勘所
yokawasa
7
4.1k
【若手エンジニア応援LT会】AWSで繋がり、共に成長! ~コミュニティ活動と新人教育への挑戦~
kazushi_ohata
0
170
一休.comレストランにおけるRustの活用
kymmt90
3
580
Autify Company Deck
autifyhq
1
39k
30万人が利用するチャットをFirebase Realtime DatabaseからActionCableへ移行する方法
ryosk7
5
330
グローバル展開を見据えたサービスにおける機械翻訳プラクティス / dp-ai-translating
cyberagentdevelopers
PRO
1
150
visionOSでの空間表現実装とImmersive Video表示について / ai-immersive-visionos
cyberagentdevelopers
PRO
1
100
Emacs x Nostr
hakkadaikon
1
150
失敗しないOpenJDKの非互換調査
tabatad
0
270
Apple/Google/Amazonの決済システムの違いを踏まえた定期購読課金システムの構築 / abema-billing-system
cyberagentdevelopers
PRO
1
210
Featured
See All Featured
Testing 201, or: Great Expectations
jmmastey
38
7k
Large-scale JavaScript Application Architecture
addyosmani
510
110k
Visualizing Your Data: Incorporating Mongo into Loggly Infrastructure
mongodb
42
9.2k
For a Future-Friendly Web
brad_frost
175
9.4k
Gamification - CAS2011
davidbonilla
80
5k
VelocityConf: Rendering Performance Case Studies
addyosmani
325
24k
[Rails World 2023 - Day 1 Closing Keynote] - The Magic of Rails
eileencodes
32
1.8k
The Illustrated Children's Guide to Kubernetes
chrisshort
48
48k
Docker and Python
trallard
40
3.1k
A Tale of Four Properties
chriscoyier
156
23k
Side Projects
sachag
452
42k
ReactJS: Keep Simple. Everything can be a component!
pedronauck
664
120k
Transcript
1 ϚϧνΫϥυͳେنαʔϏε KARTEΛࢧ͑ΔDatadog ɹɹʛɹɹ© 2019 PLAID Inc. 2019.11.13ɹɹʛɹɹDatadog Summit Tokyo
2019ɹɹʛɹ PLAID, Inc. Daiki Matsui
2 2019.11.13ɹɹʛɹɹDatadog Summit Tokyo 2019ɹɹʛɹ ɹɹʛɹɹ© 2019 PLAID Inc. ࣗݾհ
দҪ େٍ גࣜձࣾϓϨΠυ SRE Senior Architect 2012-2015·ͰגࣜձࣾαΠόʔυʹͯήʔϜ։ൃʹैࣄ 2015ΑΓϓϨΠυʹࢀը͠ɺSREͱͯ͠ϚϧνΫϥυΠϯϑϥͷߏஙɾӡ༻Λ୲ 2015ೖࣾ
ɹɹʛɹɹ© 2019 PLAID Inc. 3 2019.11.13ɹɹʛɹɹDatadog Summit Tokyo 2019ɹɹʛɹ גࣜձࣾϓϨΠυʹ͍ͭͯ
גࣜձࣾϓϨΠυ ౦ژதԝ۠ۜ࠲6-10-1 GINZA SIX 10F ઃཱɿ201110݄ ैۀһɿ130໊ ࢿຊۚɿ149,987,878ԁ ※ࢿຊ४උؚۚΉ
4 2019.11.13ɹɹʛɹɹDatadog Summit Tokyo 2019ɹɹʛɹ ɹɹʛɹɹ© 2019 PLAID Inc. KARTEʹ͍ͭͯ
֬ೝ͢Δ جຊαʔϏε͝ར༻ํ๏Λ͝հ͠·͢ɻ ͓ಘͳಛయཱͭใ͕ຬࡌͰ͢ɻ ॳΊͯͷํ ְתּׅ然钠ׅ 6*رؠ؎ش٦ֽוזַזַ♳麦׃זְծ➙ ״إٔؗ،حف׃ְהְֲ倯䗳铣דׅկ UIرؠ؎ش٦ָ濼גֶֻץֹ 7אךرؠ؎ٕٝ٦ٕ DESIGN RULES » CHECK ׆ה⢪ִ،؎ذيֽ䲧ִת׃կ ֿך堣⠓ֶ鋅鷕׃זֻկ 窫㼎ծ 妜׃ְ ౙͷओɺ Ξλʔ COLLECTION OUTER 嗚稊勴⟝㼰׃㢌刿ֽׅדծ֮זך椚 䟝ך暟⟝ח⳿⠓ִ〳腉䚍ָ넝ֻזתׅկ 勴⟝㢌ִג嗚稊׃גגְֻׁկ ׀䋞劄ך暟⟝כ 鋅אַתׇד׃ַ ꟗׄ ҰਓҰਓʹ߹Θͤͨ ސ٬ମݧΛఏڙ WebαΠτͷ๚ऀͷߦಈΛ ސ٬͝ͱʹϦΞϧλΠϜʹղੳ $9 ސ٬ମݧ ϓϥοτϑΥʔϜ
5 2019.11.13ɹɹʛɹɹDatadog Summit Tokyo 2019ɹɹʛɹ ɹɹʛɹɹ© 2019 PLAID Inc. ͢͜ͱ
KARTEͷಛ KARTEΛࢧ͑ΔࢹͷϙΠϯτ ࠓޙΓ͍ͨ͜ͱ 1 2 3
6 2019.11.13ɹɹʛɹɹDatadog Summit Tokyo 2019ɹɹʛɹ ɹɹʛɹɹ© 2019 PLAID Inc. ͢͜ͱ
KARTEͷಛ KARTEΛࢧ͑ΔࢹͷϙΠϯτ ࠓޙΓ͍ͨ͜ͱ
7 ɹɹʛɹɹ© 2019 PLAID Inc. 2019.11.13ɹɹʛɹɹDatadog Summit Tokyo 2019ɹɹʛɹ KARTEͷಛ
1.Ϛϧνςφϯτ 2.ϨΠςϯγʔ/ߴෛՙ 3.େنγεςϜ 4.ϚϧνΫϥυߏ ͢͜ͱ →
8 ɹɹʛɹɹ© 2019 PLAID Inc. 2019.11.13ɹɹʛɹɹDatadog Summit Tokyo 2019ɹɹʛɹ KARTEͷಛ
1.Ϛϧνςφϯτ 2.ϨΠςϯγʔ/ߴෛՙ 3.େنγεςϜ 4.ϚϧνΫϥυߏ ͢͜ͱ →
9 2019.11.13ɹɹʛɹɹDatadog Summit Tokyo 2019ɹɹʛɹ ɹɹʛɹɹ© 2019 PLAID Inc. Ϛϧνςφϯτ
ɹɹʛɹɹ© 2019 PLAID Inc. 10 2019.11.13ɹɹʛɹɹDatadog Summit Tokyo 2019ɹɹʛɹ ɾ༷ʑͳෛՙͷ͓٬͞Μ͕͍Δ
ɾΞΫηεύλʔϯ৭ʑ Ϛϧνςφϯτ
ɹɹʛɹɹ© 2019 PLAID Inc. 11 2019.11.13ɹɹʛɹɹDatadog Summit Tokyo 2019ɹɹʛɹ ɾෛՙ͕1Ͱ10ഒมԽ͢Δ
ɾηʔϧͰෛՙ͕ٸ૿͢Δ Ϛϧνςφϯτ 1ͷෛՙਪҠ
12 ɹɹʛɹɹ© 2019 PLAID Inc. 2019.11.13ɹɹʛɹɹDatadog Summit Tokyo 2019ɹɹʛɹ KARTEͷಛ
1.Ϛϧνςφϯτ 2.ϨΠςϯγʔ/ߴෛՙ 3.େنγεςϜ 4.ϚϧνΫϥυߏ ͢͜ͱ →
13 2019.11.13ɹɹʛɹɹDatadog Summit Tokyo 2019ɹɹʛɹ ɹɹʛɹɹ© 2019 PLAID Inc. ϨΠςϯγʔɺߴෛՙ
sec to real-time actions 0.x events per day 3 billion stored in BigQuery 2+ PB events (rows) / sec 65,000 streaming insert per month 1 PB analysis per month 60+ PB SRE 3 ϨΠςϯγʔ/ߴෛՙ
14 2019.11.13ɹɹʛɹɹDatadog Summit Tokyo 2019ɹɹʛɹ ɹɹʛɹɹ© 2019 PLAID Inc. ϨΠςϯγʔɺߴෛՙ
sec to real-time actions 0.x events per day 3 billion stored in BigQuery 2+ PB events (rows) / sec 65,000 streaming insert per month 1 PB analysis per month 60+ PB SRE 3 ϨΠςϯγʔ/ߴෛՙ
15 2019.11.13ɹɹʛɹɹDatadog Summit Tokyo 2019ɹɹʛɹ ɹɹʛɹɹ© 2019 PLAID Inc. ϨΠςϯγʔɺߴෛՙ
16 2019.11.13ɹɹʛɹɹDatadog Summit Tokyo 2019ɹɹʛɹ ɹɹʛɹɹ© 2019 PLAID Inc. ϨΠςϯγʔɺߴෛՙ
sec to real-time actions 0.x events per day 3 billion stored in BigQuery 2+ PB events (rows) / sec 65,000 streaming insert per month 1 PB analysis per month 60+ PB SRE 3 ϨΠςϯγʔ/ߴෛՙ
17 ɹɹʛɹɹ© 2019 PLAID Inc. 2019.11.13ɹɹʛɹɹDatadog Summit Tokyo 2019ɹɹʛɹ KARTEͷಛ
1.Ϛϧνςφϯτ 2.ϨΠςϯγʔ/ߴෛՙ 3.େنγεςϜ 4.ϚϧνΫϥυߏ ͢͜ͱ →
18 2019.11.13ɹɹʛɹɹDatadog Summit Tokyo 2019ɹɹʛɹ ɹɹʛɹɹ© 2019 PLAID Inc. େنγεςϜ
19 ɹɹʛɹɹ© 2019 PLAID Inc. 2019.11.13ɹɹʛɹɹDatadog Summit Tokyo 2019ɹɹʛɹ KARTEͷಛ
1.Ϛϧνςφϯτ 2.ϨΠςϯγʔ/ߴෛՙ 3.େنγεςϜ 4.ϚϧνΫϥυߏ ͢͜ͱ →
20 2019.11.13ɹɹʛɹɹDatadog Summit Tokyo 2019ɹɹʛɹ ɹɹʛɹɹ© 2019 PLAID Inc. ϚϧνΫϥυߏ
ɹɹʛɹɹ© 2019 PLAID Inc. 21 2019.11.13ɹɹʛɹɹDatadog Summit Tokyo 2019ɹɹʛɹ Pros
1. ֤ϓϩόΠμʔͷྑ͍ͱ͜औΓ͕Ͱ͖Δ 2. ϓϩόΠμʔͷোΛผϓϩόΠμʔͰΧ όʔͰ͖Δ Pros/Cons ϚϧνΫϥυߏ Cons 1. ӡ༻ͷख͕ؒ૿͑Δ
ɹɹʛɹɹ© 2019 PLAID Inc. 22 2019.11.13ɹɹʛɹɹDatadog Summit Tokyo 2019ɹɹʛɹ ৄࡉΛΓ͍ͨํ
ʮσϒαϛ 2018ʯͷࢿྉͰʂ ϚϧνΫϥυߏ https://speakerdeck.com/tikson/zhan-lue-topointonituite
23 2019.11.13ɹɹʛɹɹDatadog Summit Tokyo 2019ɹɹʛɹ ɹɹʛɹɹ© 2019 PLAID Inc. ͢͜ͱ
KARTEͷಛ KARTEΛࢧ͑ΔࢹͷϙΠϯτ ࠓޙΓ͍ͨ͜ͱ ✓
KARTEΛࢧ͑ΔࢹͷϙΠϯτ 24 ɹɹʛɹɹ© 2019 PLAID Inc. 2019.11.13ɹɹʛɹɹDatadog Summit Tokyo 2019ɹɹʛɹ
ɾϦΞϧλΠϜੑ ɾॊೈੑ ɾεέʔϥϏϦςΟ ɾ҆ఆੑ
25 ɹɹʛɹɹ© 2019 PLAID Inc. 2019.11.13ɹɹʛɹɹDatadog Summit Tokyo 2019ɹɹʛɹ KARTEͷಛ
→ 1.Ϛϧνςφϯτ 2.ϨΠςϯγʔ/ߴෛՙ 3.େنγεςϜ 4.ϚϧνΫϥυߏ
26 ɹɹʛɹɹ© 2019 PLAID Inc. 2019.11.13ɹɹʛɹɹDatadog Summit Tokyo 2019ɹɹʛɹ KARTEΛࢧ͑Δ
ࢹͷϙΠϯτ 1.Datadogʹશͯू 2.Custom metricsͷੵۃ׆༻ 3.LayerΛ͚ͨDashboard ͢͜ͱ →
27 ɹɹʛɹɹ© 2019 PLAID Inc. 2019.11.13ɹɹʛɹɹDatadog Summit Tokyo 2019ɹɹʛɹ →
1.Datadogʹશͯू 2.Custom metricsͷੵۃ׆༻ 3.LayerΛ͚ͨDashboard 1.Ϛϧνςφϯτ 2.ϨΠςϯγʔ/ߴෛՙ 3.େنγεςϜ 4.ϚϧνΫϥυߏ
ɹɹʛɹɹ© 2019 PLAID Inc. 28 2019.11.13ɹɹʛɹɹDatadog Summit Tokyo 2019ɹɹʛɹ োରԠ࣌ʹDatadogΛݟΕ
͙͢ʹঢ়گΛѲͰ͖ΔΑ͏ʹ Datadogʹશͯू
29 ɹɹʛɹɹ© 2019 PLAID Inc. 2019.11.13ɹɹʛɹɹDatadog Summit Tokyo 2019ɹɹʛɹ KARTEΛࢧ͑Δ
ࢹͷϙΠϯτ ͢͜ͱ → 1.Datadogʹશͯू 2.Custom metricsͷੵۃ׆༻ 3.LayerΛ͚ͨDashboard
30 ɹɹʛɹɹ© 2019 PLAID Inc. 2019.11.13ɹɹʛɹɹDatadog Summit Tokyo 2019ɹɹʛɹ →
1.Datadogʹશͯू 2.Custom metricsͷੵۃ׆༻ 3.LayerΛ͚ͨDashboard 1.Ϛϧνςφϯτ 2.ϨΠςϯγʔ/ߴෛՙ 3.େنγεςϜ 4.ϚϧνΫϥυߏ
ɹɹʛɹɹ© 2019 PLAID Inc. 31 2019.11.13ɹɹʛɹɹDatadog Summit Tokyo 2019ɹɹʛɹ 32M/mon
΄Ͳར༻ Custom metricsͷੵۃ׆༻
Custom metricsͷੵۃ׆༻ 32 ɹɹʛɹɹ© 2019 PLAID Inc. 2019.11.13ɹɹʛɹɹDatadog Summit Tokyo
2019ɹɹʛɹ https://github.com/makinoy/libs-dogstatsd ɾlibs-dogstatsdͰɺ؆୯ʹॲཧͷ࣮ߦճ ࣮ߦ࣌ؒΛଌఆͰ͖ΔΑ͏ʹ ɾؔΑΓࡉ͔͍୯ҐͰlatencyΛଌఆ ɾϢʔβ୯ҐͰͷύϑΥʔϚϯεΛଌఆ 1 const dd = require('libs-dogstatsd'); 2 const stats = dd.start(); 3 // measure hoge 4 hoge(); 5 stats.tick('hoge', 1, 1, [user_id:1234]); Node.jsͷwrapper library
ɹɹʛɹɹ© 2019 PLAID Inc. 33 2019.11.13ɹɹʛɹɹDatadog Summit Tokyo 2019ɹɹʛɹ tagͰϢʔβࣝผ༻IDΛૹΔ
ࠓى͍ͬͯ͜Δ͕ ʮಛఆͷϢʔβىҼͰى͍ͬͯ͜Δʁʯ ʮKARTEશମͰى͍ͬͯ͜Δʁʯ ͔ΛΓ͚͍ͯ͘͢͠͠Δ Custom metricsͷੵۃ׆༻
34 ɹɹʛɹɹ© 2019 PLAID Inc. 2019.11.13ɹɹʛɹɹDatadog Summit Tokyo 2019ɹɹʛɹ KARTEΛࢧ͑Δ
ࢹͷϙΠϯτ ͢͜ͱ → 1.Datadogʹશͯू 2.Custom metricsͷੵۃ׆༻ 3.LayerΛ͚ͨDashboard
35 ɹɹʛɹɹ© 2019 PLAID Inc. 2019.11.13ɹɹʛɹɹDatadog Summit Tokyo 2019ɹɹʛɹ →
1.Datadogʹશͯू 2.Custom metricsͷੵۃ׆༻ 3.LayerΛ͚ͨDashboard 1.Ϛϧνςφϯτ 2.ϨΠςϯγʔ/ߴෛՙ 3.େنγεςϜ 4.ϚϧνΫϥυߏ
ɹɹʛɹɹ© 2019 PLAID Inc. 36 2019.11.13ɹɹʛɹɹDatadog Summit Tokyo 2019ɹɹʛɹ Layer
1 ɾKARTE͕ੜ͖͍ͯΔ͔Ͳ͏͔ LayerΛ͚ͨDashboard
ɹɹʛɹɹ© 2019 PLAID Inc. 37 2019.11.13ɹɹʛɹɹDatadog Summit Tokyo 2019ɹɹʛɹ Layer
2 ɾॏཁͳίϯϙʔωϯτ ɾॏཁͳΞϓϦέʔγϣϯͷϝτϦοΫ LayerΛ͚ͨDashboard
ɹɹʛɹɹ© 2019 PLAID Inc. 38 2019.11.13ɹɹʛɹɹDatadog Summit Tokyo 2019ɹɹʛɹ Layer
3 ɾ֤αʔό܈ͷࡉ͔͍ϝτϦοΫ LayerΛ͚ͨDashboard
39 2019.11.13ɹɹʛɹɹDatadog Summit Tokyo 2019ɹɹʛɹ ɹɹʛɹɹ© 2019 PLAID Inc. ͢͜ͱ
KARTEͷಛ KARTEΛࢧ͑ΔࢹͷϙΠϯτ ࠓޙΓ͍ͨ͜ͱ ✓ ✓
ɹࠓޙΓ͍ͨ͜ͱ 40 ɹɹʛɹɹ© 2019 PLAID Inc. 2019.11.13ɹɹʛɹɹDatadog Summit Tokyo 2019ɹɹʛɹ
શͯͷΤϯδχΞ͕ োͷҰ࣌ରԠΛͰ͖ΔΑ͏ʹ͢Δ ɾࢹ/োରԠͷϑϩʔͷཧ ɾDatadogΛ͍͜ͳͯ͠Β͏
ɹࠓޙΓ͍ͨ͜ͱ 41 ɹɹʛɹɹ© 2019 PLAID Inc. 2019.11.13ɹɹʛɹɹDatadog Summit Tokyo 2019ɹɹʛɹ
Custom metrics 1ԯԁͷղܾ ɾ͋·ΓʹศརͩͬͨͷͰ͍͢ ͗ͯ͠·ͬͨ… ɾCustom metricsͷཧΛߦ͏
42 2019.11.13ɹɹʛɹɹDatadog Summit Tokyo 2019ɹɹʛɹ ɹɹʛɹɹ© 2019 PLAID Inc. ·ͱΊ
·ͱΊ
43 ɹɹʛɹɹ© 2019 PLAID Inc. 2019.11.13ɹɹʛɹɹDatadog Summit Tokyo 2019ɹɹʛɹ KARTEΛࢧ͑Δ
ࢹͷϙΠϯτ 1.Datadogʹશͯू 2.Custom metricsͷੵۃ׆༻ 3.LayerΛ͚ͨDashboard →
None