Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
マルチクラウドな大規模サービスKARTEを支えるDatadog/multi-cloud mon...
Search
Daiki Matsui
November 13, 2019
Technology
0
2.5k
マルチクラウドな大規模サービスKARTEを支えるDatadog/multi-cloud monitoring with Datadog
Datadog Summit Tokyoでの登壇資料です。
動画は
https://www.youtube.com/watch?v=_Z4HBiwODbc
Daiki Matsui
November 13, 2019
Tweet
Share
More Decks by Daiki Matsui
See All by Daiki Matsui
The Fourth Age of SRE
ikemonn
0
530
Multicloud deploy with Spinnaker / builderscon tokyo 2018
ikemonn
2
4.6k
KARTE を支えるマルチプラットフォームインフラ監視 /karte-multi-platform-monitoring
ikemonn
2
16k
Node Apps Performance Tuning
ikemonn
0
140
Other Decks in Technology
See All in Technology
[SRE NEXT] ARR150億円_エンジニア140名_27チーム_17プロダクトから始めるSLO.pdf
satos
5
2.1k
CDKコード品質UP!ナイスな自作コンストラクタを作るための便利インターフェース
harukasakihara
2
200
United airlines®️ USA Contact Numbers: Complete 2025 Support Guide
unitedflyhelp
0
340
freeeのアクセシビリティの現在地 / freee's Current Position on Accessibility
ymrl
2
280
SEQUENCE object comparison - db tech showcase 2025 LT2
nori_shinoda
0
280
ゼロからはじめる採用広報
yutadayo
4
1k
microCMSではじめるAIライティング
himaratsu
0
120
VGGT: Visual Geometry Grounded Transformer
peisuke
1
620
助けて! XからWaylandに移行しないと新しいGNOMEが使えなくなっちゃう 2025-07-12
nobutomurata
2
140
NewSQLや分散データベースを支えるRaftの仕組み - 仕組みを理解して知る得意不得意
hacomono
PRO
3
230
Delta airlines®️ USA Contact Numbers: Complete 2025 Support Guide
airtravelguide
0
350
Operating Operator
shhnjk
1
650
Featured
See All Featured
It's Worth the Effort
3n
185
28k
Making Projects Easy
brettharned
116
6.3k
RailsConf & Balkan Ruby 2019: The Past, Present, and Future of Rails at GitHub
eileencodes
138
34k
How To Stay Up To Date on Web Technology
chriscoyier
790
250k
Building Applications with DynamoDB
mza
95
6.5k
The Art of Delivering Value - GDevCon NA Keynote
reverentgeek
15
1.6k
The Illustrated Children's Guide to Kubernetes
chrisshort
48
50k
Fashionably flexible responsive web design (full day workshop)
malarkey
407
66k
Balancing Empowerment & Direction
lara
1
440
Agile that works and the tools we love
rasmusluckow
329
21k
The Cost Of JavaScript in 2023
addyosmani
51
8.5k
Statistics for Hackers
jakevdp
799
220k
Transcript
1 ϚϧνΫϥυͳେنαʔϏε KARTEΛࢧ͑ΔDatadog ɹɹʛɹɹ© 2019 PLAID Inc. 2019.11.13ɹɹʛɹɹDatadog Summit Tokyo
2019ɹɹʛɹ PLAID, Inc. Daiki Matsui
2 2019.11.13ɹɹʛɹɹDatadog Summit Tokyo 2019ɹɹʛɹ ɹɹʛɹɹ© 2019 PLAID Inc. ࣗݾհ
দҪ େٍ גࣜձࣾϓϨΠυ SRE Senior Architect 2012-2015·ͰגࣜձࣾαΠόʔυʹͯήʔϜ։ൃʹैࣄ 2015ΑΓϓϨΠυʹࢀը͠ɺSREͱͯ͠ϚϧνΫϥυΠϯϑϥͷߏஙɾӡ༻Λ୲ 2015ೖࣾ
ɹɹʛɹɹ© 2019 PLAID Inc. 3 2019.11.13ɹɹʛɹɹDatadog Summit Tokyo 2019ɹɹʛɹ גࣜձࣾϓϨΠυʹ͍ͭͯ
גࣜձࣾϓϨΠυ ౦ژதԝ۠ۜ࠲6-10-1 GINZA SIX 10F ઃཱɿ201110݄ ैۀһɿ130໊ ࢿຊۚɿ149,987,878ԁ ※ࢿຊ४උؚۚΉ
4 2019.11.13ɹɹʛɹɹDatadog Summit Tokyo 2019ɹɹʛɹ ɹɹʛɹɹ© 2019 PLAID Inc. KARTEʹ͍ͭͯ
֬ೝ͢Δ جຊαʔϏε͝ར༻ํ๏Λ͝հ͠·͢ɻ ͓ಘͳಛయཱͭใ͕ຬࡌͰ͢ɻ ॳΊͯͷํ ְתּׅ然钠ׅ 6*رؠ؎ش٦ֽוזַזַ♳麦׃זְծ➙ ״إٔؗ،حف׃ְהְֲ倯䗳铣דׅկ UIرؠ؎ش٦ָ濼גֶֻץֹ 7אךرؠ؎ٕٝ٦ٕ DESIGN RULES » CHECK ׆ה⢪ִ،؎ذيֽ䲧ִת׃կ ֿך堣⠓ֶ鋅鷕׃זֻկ 窫㼎ծ 妜׃ְ ౙͷओɺ Ξλʔ COLLECTION OUTER 嗚稊勴⟝㼰׃㢌刿ֽׅדծ֮זך椚 䟝ך暟⟝ח⳿⠓ִ〳腉䚍ָ넝ֻזתׅկ 勴⟝㢌ִג嗚稊׃גגְֻׁկ ׀䋞劄ך暟⟝כ 鋅אַתׇד׃ַ ꟗׄ ҰਓҰਓʹ߹Θͤͨ ސ٬ମݧΛఏڙ WebαΠτͷ๚ऀͷߦಈΛ ސ٬͝ͱʹϦΞϧλΠϜʹղੳ $9 ސ٬ମݧ ϓϥοτϑΥʔϜ
5 2019.11.13ɹɹʛɹɹDatadog Summit Tokyo 2019ɹɹʛɹ ɹɹʛɹɹ© 2019 PLAID Inc. ͢͜ͱ
KARTEͷಛ KARTEΛࢧ͑ΔࢹͷϙΠϯτ ࠓޙΓ͍ͨ͜ͱ 1 2 3
6 2019.11.13ɹɹʛɹɹDatadog Summit Tokyo 2019ɹɹʛɹ ɹɹʛɹɹ© 2019 PLAID Inc. ͢͜ͱ
KARTEͷಛ KARTEΛࢧ͑ΔࢹͷϙΠϯτ ࠓޙΓ͍ͨ͜ͱ
7 ɹɹʛɹɹ© 2019 PLAID Inc. 2019.11.13ɹɹʛɹɹDatadog Summit Tokyo 2019ɹɹʛɹ KARTEͷಛ
1.Ϛϧνςφϯτ 2.ϨΠςϯγʔ/ߴෛՙ 3.େنγεςϜ 4.ϚϧνΫϥυߏ ͢͜ͱ →
8 ɹɹʛɹɹ© 2019 PLAID Inc. 2019.11.13ɹɹʛɹɹDatadog Summit Tokyo 2019ɹɹʛɹ KARTEͷಛ
1.Ϛϧνςφϯτ 2.ϨΠςϯγʔ/ߴෛՙ 3.େنγεςϜ 4.ϚϧνΫϥυߏ ͢͜ͱ →
9 2019.11.13ɹɹʛɹɹDatadog Summit Tokyo 2019ɹɹʛɹ ɹɹʛɹɹ© 2019 PLAID Inc. Ϛϧνςφϯτ
ɹɹʛɹɹ© 2019 PLAID Inc. 10 2019.11.13ɹɹʛɹɹDatadog Summit Tokyo 2019ɹɹʛɹ ɾ༷ʑͳෛՙͷ͓٬͞Μ͕͍Δ
ɾΞΫηεύλʔϯ৭ʑ Ϛϧνςφϯτ
ɹɹʛɹɹ© 2019 PLAID Inc. 11 2019.11.13ɹɹʛɹɹDatadog Summit Tokyo 2019ɹɹʛɹ ɾෛՙ͕1Ͱ10ഒมԽ͢Δ
ɾηʔϧͰෛՙ͕ٸ૿͢Δ Ϛϧνςφϯτ 1ͷෛՙਪҠ
12 ɹɹʛɹɹ© 2019 PLAID Inc. 2019.11.13ɹɹʛɹɹDatadog Summit Tokyo 2019ɹɹʛɹ KARTEͷಛ
1.Ϛϧνςφϯτ 2.ϨΠςϯγʔ/ߴෛՙ 3.େنγεςϜ 4.ϚϧνΫϥυߏ ͢͜ͱ →
13 2019.11.13ɹɹʛɹɹDatadog Summit Tokyo 2019ɹɹʛɹ ɹɹʛɹɹ© 2019 PLAID Inc. ϨΠςϯγʔɺߴෛՙ
sec to real-time actions 0.x events per day 3 billion stored in BigQuery 2+ PB events (rows) / sec 65,000 streaming insert per month 1 PB analysis per month 60+ PB SRE 3 ϨΠςϯγʔ/ߴෛՙ
14 2019.11.13ɹɹʛɹɹDatadog Summit Tokyo 2019ɹɹʛɹ ɹɹʛɹɹ© 2019 PLAID Inc. ϨΠςϯγʔɺߴෛՙ
sec to real-time actions 0.x events per day 3 billion stored in BigQuery 2+ PB events (rows) / sec 65,000 streaming insert per month 1 PB analysis per month 60+ PB SRE 3 ϨΠςϯγʔ/ߴෛՙ
15 2019.11.13ɹɹʛɹɹDatadog Summit Tokyo 2019ɹɹʛɹ ɹɹʛɹɹ© 2019 PLAID Inc. ϨΠςϯγʔɺߴෛՙ
16 2019.11.13ɹɹʛɹɹDatadog Summit Tokyo 2019ɹɹʛɹ ɹɹʛɹɹ© 2019 PLAID Inc. ϨΠςϯγʔɺߴෛՙ
sec to real-time actions 0.x events per day 3 billion stored in BigQuery 2+ PB events (rows) / sec 65,000 streaming insert per month 1 PB analysis per month 60+ PB SRE 3 ϨΠςϯγʔ/ߴෛՙ
17 ɹɹʛɹɹ© 2019 PLAID Inc. 2019.11.13ɹɹʛɹɹDatadog Summit Tokyo 2019ɹɹʛɹ KARTEͷಛ
1.Ϛϧνςφϯτ 2.ϨΠςϯγʔ/ߴෛՙ 3.େنγεςϜ 4.ϚϧνΫϥυߏ ͢͜ͱ →
18 2019.11.13ɹɹʛɹɹDatadog Summit Tokyo 2019ɹɹʛɹ ɹɹʛɹɹ© 2019 PLAID Inc. େنγεςϜ
19 ɹɹʛɹɹ© 2019 PLAID Inc. 2019.11.13ɹɹʛɹɹDatadog Summit Tokyo 2019ɹɹʛɹ KARTEͷಛ
1.Ϛϧνςφϯτ 2.ϨΠςϯγʔ/ߴෛՙ 3.େنγεςϜ 4.ϚϧνΫϥυߏ ͢͜ͱ →
20 2019.11.13ɹɹʛɹɹDatadog Summit Tokyo 2019ɹɹʛɹ ɹɹʛɹɹ© 2019 PLAID Inc. ϚϧνΫϥυߏ
ɹɹʛɹɹ© 2019 PLAID Inc. 21 2019.11.13ɹɹʛɹɹDatadog Summit Tokyo 2019ɹɹʛɹ Pros
1. ֤ϓϩόΠμʔͷྑ͍ͱ͜औΓ͕Ͱ͖Δ 2. ϓϩόΠμʔͷোΛผϓϩόΠμʔͰΧ όʔͰ͖Δ Pros/Cons ϚϧνΫϥυߏ Cons 1. ӡ༻ͷख͕ؒ૿͑Δ
ɹɹʛɹɹ© 2019 PLAID Inc. 22 2019.11.13ɹɹʛɹɹDatadog Summit Tokyo 2019ɹɹʛɹ ৄࡉΛΓ͍ͨํ
ʮσϒαϛ 2018ʯͷࢿྉͰʂ ϚϧνΫϥυߏ https://speakerdeck.com/tikson/zhan-lue-topointonituite
23 2019.11.13ɹɹʛɹɹDatadog Summit Tokyo 2019ɹɹʛɹ ɹɹʛɹɹ© 2019 PLAID Inc. ͢͜ͱ
KARTEͷಛ KARTEΛࢧ͑ΔࢹͷϙΠϯτ ࠓޙΓ͍ͨ͜ͱ ✓
KARTEΛࢧ͑ΔࢹͷϙΠϯτ 24 ɹɹʛɹɹ© 2019 PLAID Inc. 2019.11.13ɹɹʛɹɹDatadog Summit Tokyo 2019ɹɹʛɹ
ɾϦΞϧλΠϜੑ ɾॊೈੑ ɾεέʔϥϏϦςΟ ɾ҆ఆੑ
25 ɹɹʛɹɹ© 2019 PLAID Inc. 2019.11.13ɹɹʛɹɹDatadog Summit Tokyo 2019ɹɹʛɹ KARTEͷಛ
→ 1.Ϛϧνςφϯτ 2.ϨΠςϯγʔ/ߴෛՙ 3.େنγεςϜ 4.ϚϧνΫϥυߏ
26 ɹɹʛɹɹ© 2019 PLAID Inc. 2019.11.13ɹɹʛɹɹDatadog Summit Tokyo 2019ɹɹʛɹ KARTEΛࢧ͑Δ
ࢹͷϙΠϯτ 1.Datadogʹશͯू 2.Custom metricsͷੵۃ׆༻ 3.LayerΛ͚ͨDashboard ͢͜ͱ →
27 ɹɹʛɹɹ© 2019 PLAID Inc. 2019.11.13ɹɹʛɹɹDatadog Summit Tokyo 2019ɹɹʛɹ →
1.Datadogʹશͯू 2.Custom metricsͷੵۃ׆༻ 3.LayerΛ͚ͨDashboard 1.Ϛϧνςφϯτ 2.ϨΠςϯγʔ/ߴෛՙ 3.େنγεςϜ 4.ϚϧνΫϥυߏ
ɹɹʛɹɹ© 2019 PLAID Inc. 28 2019.11.13ɹɹʛɹɹDatadog Summit Tokyo 2019ɹɹʛɹ োରԠ࣌ʹDatadogΛݟΕ
͙͢ʹঢ়گΛѲͰ͖ΔΑ͏ʹ Datadogʹશͯू
29 ɹɹʛɹɹ© 2019 PLAID Inc. 2019.11.13ɹɹʛɹɹDatadog Summit Tokyo 2019ɹɹʛɹ KARTEΛࢧ͑Δ
ࢹͷϙΠϯτ ͢͜ͱ → 1.Datadogʹશͯू 2.Custom metricsͷੵۃ׆༻ 3.LayerΛ͚ͨDashboard
30 ɹɹʛɹɹ© 2019 PLAID Inc. 2019.11.13ɹɹʛɹɹDatadog Summit Tokyo 2019ɹɹʛɹ →
1.Datadogʹશͯू 2.Custom metricsͷੵۃ׆༻ 3.LayerΛ͚ͨDashboard 1.Ϛϧνςφϯτ 2.ϨΠςϯγʔ/ߴෛՙ 3.େنγεςϜ 4.ϚϧνΫϥυߏ
ɹɹʛɹɹ© 2019 PLAID Inc. 31 2019.11.13ɹɹʛɹɹDatadog Summit Tokyo 2019ɹɹʛɹ 32M/mon
΄Ͳར༻ Custom metricsͷੵۃ׆༻
Custom metricsͷੵۃ׆༻ 32 ɹɹʛɹɹ© 2019 PLAID Inc. 2019.11.13ɹɹʛɹɹDatadog Summit Tokyo
2019ɹɹʛɹ https://github.com/makinoy/libs-dogstatsd ɾlibs-dogstatsdͰɺ؆୯ʹॲཧͷ࣮ߦճ ࣮ߦ࣌ؒΛଌఆͰ͖ΔΑ͏ʹ ɾؔΑΓࡉ͔͍୯ҐͰlatencyΛଌఆ ɾϢʔβ୯ҐͰͷύϑΥʔϚϯεΛଌఆ 1 const dd = require('libs-dogstatsd'); 2 const stats = dd.start(); 3 // measure hoge 4 hoge(); 5 stats.tick('hoge', 1, 1, [user_id:1234]); Node.jsͷwrapper library
ɹɹʛɹɹ© 2019 PLAID Inc. 33 2019.11.13ɹɹʛɹɹDatadog Summit Tokyo 2019ɹɹʛɹ tagͰϢʔβࣝผ༻IDΛૹΔ
ࠓى͍ͬͯ͜Δ͕ ʮಛఆͷϢʔβىҼͰى͍ͬͯ͜Δʁʯ ʮKARTEશମͰى͍ͬͯ͜Δʁʯ ͔ΛΓ͚͍ͯ͘͢͠͠Δ Custom metricsͷੵۃ׆༻
34 ɹɹʛɹɹ© 2019 PLAID Inc. 2019.11.13ɹɹʛɹɹDatadog Summit Tokyo 2019ɹɹʛɹ KARTEΛࢧ͑Δ
ࢹͷϙΠϯτ ͢͜ͱ → 1.Datadogʹશͯू 2.Custom metricsͷੵۃ׆༻ 3.LayerΛ͚ͨDashboard
35 ɹɹʛɹɹ© 2019 PLAID Inc. 2019.11.13ɹɹʛɹɹDatadog Summit Tokyo 2019ɹɹʛɹ →
1.Datadogʹશͯू 2.Custom metricsͷੵۃ׆༻ 3.LayerΛ͚ͨDashboard 1.Ϛϧνςφϯτ 2.ϨΠςϯγʔ/ߴෛՙ 3.େنγεςϜ 4.ϚϧνΫϥυߏ
ɹɹʛɹɹ© 2019 PLAID Inc. 36 2019.11.13ɹɹʛɹɹDatadog Summit Tokyo 2019ɹɹʛɹ Layer
1 ɾKARTE͕ੜ͖͍ͯΔ͔Ͳ͏͔ LayerΛ͚ͨDashboard
ɹɹʛɹɹ© 2019 PLAID Inc. 37 2019.11.13ɹɹʛɹɹDatadog Summit Tokyo 2019ɹɹʛɹ Layer
2 ɾॏཁͳίϯϙʔωϯτ ɾॏཁͳΞϓϦέʔγϣϯͷϝτϦοΫ LayerΛ͚ͨDashboard
ɹɹʛɹɹ© 2019 PLAID Inc. 38 2019.11.13ɹɹʛɹɹDatadog Summit Tokyo 2019ɹɹʛɹ Layer
3 ɾ֤αʔό܈ͷࡉ͔͍ϝτϦοΫ LayerΛ͚ͨDashboard
39 2019.11.13ɹɹʛɹɹDatadog Summit Tokyo 2019ɹɹʛɹ ɹɹʛɹɹ© 2019 PLAID Inc. ͢͜ͱ
KARTEͷಛ KARTEΛࢧ͑ΔࢹͷϙΠϯτ ࠓޙΓ͍ͨ͜ͱ ✓ ✓
ɹࠓޙΓ͍ͨ͜ͱ 40 ɹɹʛɹɹ© 2019 PLAID Inc. 2019.11.13ɹɹʛɹɹDatadog Summit Tokyo 2019ɹɹʛɹ
શͯͷΤϯδχΞ͕ োͷҰ࣌ରԠΛͰ͖ΔΑ͏ʹ͢Δ ɾࢹ/োରԠͷϑϩʔͷཧ ɾDatadogΛ͍͜ͳͯ͠Β͏
ɹࠓޙΓ͍ͨ͜ͱ 41 ɹɹʛɹɹ© 2019 PLAID Inc. 2019.11.13ɹɹʛɹɹDatadog Summit Tokyo 2019ɹɹʛɹ
Custom metrics 1ԯԁͷղܾ ɾ͋·ΓʹศརͩͬͨͷͰ͍͢ ͗ͯ͠·ͬͨ… ɾCustom metricsͷཧΛߦ͏
42 2019.11.13ɹɹʛɹɹDatadog Summit Tokyo 2019ɹɹʛɹ ɹɹʛɹɹ© 2019 PLAID Inc. ·ͱΊ
·ͱΊ
43 ɹɹʛɹɹ© 2019 PLAID Inc. 2019.11.13ɹɹʛɹɹDatadog Summit Tokyo 2019ɹɹʛɹ KARTEΛࢧ͑Δ
ࢹͷϙΠϯτ 1.Datadogʹશͯू 2.Custom metricsͷੵۃ׆༻ 3.LayerΛ͚ͨDashboard →
None