Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
マルチクラウドな大規模サービスKARTEを支えるDatadog/multi-cloud monitoring with Datadog
Search
Daiki Matsui
November 13, 2019
Technology
0
1.9k
マルチクラウドな大規模サービスKARTEを支えるDatadog/multi-cloud monitoring with Datadog
Datadog Summit Tokyoでの登壇資料です。
動画は
https://www.youtube.com/watch?v=_Z4HBiwODbc
Daiki Matsui
November 13, 2019
Tweet
Share
More Decks by Daiki Matsui
See All by Daiki Matsui
The Fourth Age of SRE
ikemonn
0
190
Multicloud deploy with Spinnaker / builderscon tokyo 2018
ikemonn
2
4.2k
KARTE を支えるマルチプラットフォームインフラ監視 /karte-multi-platform-monitoring
ikemonn
2
15k
Node Apps Performance Tuning
ikemonn
0
130
Other Decks in Technology
See All in Technology
ゼロから始めるVue.jsコミュニティ貢献 / first-vuejs-community-contribution-link-and-motivation
lmi
1
130
推しは推せるときに推せ! プロダクトにフィードバックしていこう
nakasho
0
380
どうするコスト最適化のトレードオフ
tetsuyaooooo
1
590
地理空間データ可視化・解析・活用ソリューション Pacific Spatial Solutions (PSS)
pacificspatialsolutions
0
300
いつか使うかも貯金してたらめちゃめちゃ機能が増えてた話
riyaamemiya
0
410
TechFeed Experts Night#27 〜 フロントエンドフレームワーク最前線 (Svelte)
baseballyama
1
540
オーナーシップを持つ領域を明確にする
konifar
13
3.2k
Delivering Millions of Messages within seconds @ Duolingo
pelelgrino
0
350
ルーターでプレゼンする
puhitaku
0
720
require(ESM)とECMAScript仕様
uhyo
3
830
Building a RAG-poweredAI chat appwith Python and VS Code
pamelafox
0
110
GraphQL 成熟度モデルの紹介と、プロダクトに当てはめた事例 / GraphQL maturity model
mh4gf
7
1.4k
Featured
See All Featured
Done Done
chrislema
178
15k
StorybookのUI Testing Handbookを読んだ
zakiyama
13
4.6k
Building Better People: How to give real-time feedback that sticks.
wjessup
355
18k
Rebuilding a faster, lazier Slack
samanthasiow
73
8.2k
Making Projects Easy
brettharned
108
5.5k
RailsConf 2023
tenderlove
4
540
WebSockets: Embracing the real-time Web
robhawkes
59
7k
Building an army of robots
kneath
300
41k
I Don’t Have Time: Getting Over the Fear to Launch Your Podcast
jcasabona
21
1.6k
Happy Clients
brianwarren
92
6.4k
Pencils Down: Stop Designing & Start Developing
hursman
117
11k
Embracing the Ebb and Flow
colly
80
4.1k
Transcript
1 ϚϧνΫϥυͳେنαʔϏε KARTEΛࢧ͑ΔDatadog ɹɹʛɹɹ© 2019 PLAID Inc. 2019.11.13ɹɹʛɹɹDatadog Summit Tokyo
2019ɹɹʛɹ PLAID, Inc. Daiki Matsui
2 2019.11.13ɹɹʛɹɹDatadog Summit Tokyo 2019ɹɹʛɹ ɹɹʛɹɹ© 2019 PLAID Inc. ࣗݾհ
দҪ େٍ גࣜձࣾϓϨΠυ SRE Senior Architect 2012-2015·ͰגࣜձࣾαΠόʔυʹͯήʔϜ։ൃʹैࣄ 2015ΑΓϓϨΠυʹࢀը͠ɺSREͱͯ͠ϚϧνΫϥυΠϯϑϥͷߏஙɾӡ༻Λ୲ 2015ೖࣾ
ɹɹʛɹɹ© 2019 PLAID Inc. 3 2019.11.13ɹɹʛɹɹDatadog Summit Tokyo 2019ɹɹʛɹ גࣜձࣾϓϨΠυʹ͍ͭͯ
גࣜձࣾϓϨΠυ ౦ژதԝ۠ۜ࠲6-10-1 GINZA SIX 10F ઃཱɿ201110݄ ैۀһɿ130໊ ࢿຊۚɿ149,987,878ԁ ※ࢿຊ४උؚۚΉ
4 2019.11.13ɹɹʛɹɹDatadog Summit Tokyo 2019ɹɹʛɹ ɹɹʛɹɹ© 2019 PLAID Inc. KARTEʹ͍ͭͯ
֬ೝ͢Δ جຊαʔϏε͝ར༻ํ๏Λ͝հ͠·͢ɻ ͓ಘͳಛయཱͭใ͕ຬࡌͰ͢ɻ ॳΊͯͷํ ְתּׅ然钠ׅ 6*رؠ؎ش٦ֽוזַזַ♳麦׃זְծ➙ ״إٔؗ،حف׃ְהְֲ倯䗳铣דׅկ UIرؠ؎ش٦ָ濼גֶֻץֹ 7אךرؠ؎ٕٝ٦ٕ DESIGN RULES » CHECK ׆ה⢪ִ،؎ذيֽ䲧ִת׃կ ֿך堣⠓ֶ鋅鷕׃זֻկ 窫㼎ծ 妜׃ְ ౙͷओɺ Ξλʔ COLLECTION OUTER 嗚稊勴⟝㼰׃㢌刿ֽׅדծ֮זך椚 䟝ך暟⟝ח⳿⠓ִ〳腉䚍ָ넝ֻזתׅկ 勴⟝㢌ִג嗚稊׃גגְֻׁկ ׀䋞劄ך暟⟝כ 鋅אַתׇד׃ַ ꟗׄ ҰਓҰਓʹ߹Θͤͨ ސ٬ମݧΛఏڙ WebαΠτͷ๚ऀͷߦಈΛ ސ٬͝ͱʹϦΞϧλΠϜʹղੳ $9 ސ٬ମݧ ϓϥοτϑΥʔϜ
5 2019.11.13ɹɹʛɹɹDatadog Summit Tokyo 2019ɹɹʛɹ ɹɹʛɹɹ© 2019 PLAID Inc. ͢͜ͱ
KARTEͷಛ KARTEΛࢧ͑ΔࢹͷϙΠϯτ ࠓޙΓ͍ͨ͜ͱ 1 2 3
6 2019.11.13ɹɹʛɹɹDatadog Summit Tokyo 2019ɹɹʛɹ ɹɹʛɹɹ© 2019 PLAID Inc. ͢͜ͱ
KARTEͷಛ KARTEΛࢧ͑ΔࢹͷϙΠϯτ ࠓޙΓ͍ͨ͜ͱ
7 ɹɹʛɹɹ© 2019 PLAID Inc. 2019.11.13ɹɹʛɹɹDatadog Summit Tokyo 2019ɹɹʛɹ KARTEͷಛ
1.Ϛϧνςφϯτ 2.ϨΠςϯγʔ/ߴෛՙ 3.େنγεςϜ 4.ϚϧνΫϥυߏ ͢͜ͱ →
8 ɹɹʛɹɹ© 2019 PLAID Inc. 2019.11.13ɹɹʛɹɹDatadog Summit Tokyo 2019ɹɹʛɹ KARTEͷಛ
1.Ϛϧνςφϯτ 2.ϨΠςϯγʔ/ߴෛՙ 3.େنγεςϜ 4.ϚϧνΫϥυߏ ͢͜ͱ →
9 2019.11.13ɹɹʛɹɹDatadog Summit Tokyo 2019ɹɹʛɹ ɹɹʛɹɹ© 2019 PLAID Inc. Ϛϧνςφϯτ
ɹɹʛɹɹ© 2019 PLAID Inc. 10 2019.11.13ɹɹʛɹɹDatadog Summit Tokyo 2019ɹɹʛɹ ɾ༷ʑͳෛՙͷ͓٬͞Μ͕͍Δ
ɾΞΫηεύλʔϯ৭ʑ Ϛϧνςφϯτ
ɹɹʛɹɹ© 2019 PLAID Inc. 11 2019.11.13ɹɹʛɹɹDatadog Summit Tokyo 2019ɹɹʛɹ ɾෛՙ͕1Ͱ10ഒมԽ͢Δ
ɾηʔϧͰෛՙ͕ٸ૿͢Δ Ϛϧνςφϯτ 1ͷෛՙਪҠ
12 ɹɹʛɹɹ© 2019 PLAID Inc. 2019.11.13ɹɹʛɹɹDatadog Summit Tokyo 2019ɹɹʛɹ KARTEͷಛ
1.Ϛϧνςφϯτ 2.ϨΠςϯγʔ/ߴෛՙ 3.େنγεςϜ 4.ϚϧνΫϥυߏ ͢͜ͱ →
13 2019.11.13ɹɹʛɹɹDatadog Summit Tokyo 2019ɹɹʛɹ ɹɹʛɹɹ© 2019 PLAID Inc. ϨΠςϯγʔɺߴෛՙ
sec to real-time actions 0.x events per day 3 billion stored in BigQuery 2+ PB events (rows) / sec 65,000 streaming insert per month 1 PB analysis per month 60+ PB SRE 3 ϨΠςϯγʔ/ߴෛՙ
14 2019.11.13ɹɹʛɹɹDatadog Summit Tokyo 2019ɹɹʛɹ ɹɹʛɹɹ© 2019 PLAID Inc. ϨΠςϯγʔɺߴෛՙ
sec to real-time actions 0.x events per day 3 billion stored in BigQuery 2+ PB events (rows) / sec 65,000 streaming insert per month 1 PB analysis per month 60+ PB SRE 3 ϨΠςϯγʔ/ߴෛՙ
15 2019.11.13ɹɹʛɹɹDatadog Summit Tokyo 2019ɹɹʛɹ ɹɹʛɹɹ© 2019 PLAID Inc. ϨΠςϯγʔɺߴෛՙ
16 2019.11.13ɹɹʛɹɹDatadog Summit Tokyo 2019ɹɹʛɹ ɹɹʛɹɹ© 2019 PLAID Inc. ϨΠςϯγʔɺߴෛՙ
sec to real-time actions 0.x events per day 3 billion stored in BigQuery 2+ PB events (rows) / sec 65,000 streaming insert per month 1 PB analysis per month 60+ PB SRE 3 ϨΠςϯγʔ/ߴෛՙ
17 ɹɹʛɹɹ© 2019 PLAID Inc. 2019.11.13ɹɹʛɹɹDatadog Summit Tokyo 2019ɹɹʛɹ KARTEͷಛ
1.Ϛϧνςφϯτ 2.ϨΠςϯγʔ/ߴෛՙ 3.େنγεςϜ 4.ϚϧνΫϥυߏ ͢͜ͱ →
18 2019.11.13ɹɹʛɹɹDatadog Summit Tokyo 2019ɹɹʛɹ ɹɹʛɹɹ© 2019 PLAID Inc. େنγεςϜ
19 ɹɹʛɹɹ© 2019 PLAID Inc. 2019.11.13ɹɹʛɹɹDatadog Summit Tokyo 2019ɹɹʛɹ KARTEͷಛ
1.Ϛϧνςφϯτ 2.ϨΠςϯγʔ/ߴෛՙ 3.େنγεςϜ 4.ϚϧνΫϥυߏ ͢͜ͱ →
20 2019.11.13ɹɹʛɹɹDatadog Summit Tokyo 2019ɹɹʛɹ ɹɹʛɹɹ© 2019 PLAID Inc. ϚϧνΫϥυߏ
ɹɹʛɹɹ© 2019 PLAID Inc. 21 2019.11.13ɹɹʛɹɹDatadog Summit Tokyo 2019ɹɹʛɹ Pros
1. ֤ϓϩόΠμʔͷྑ͍ͱ͜औΓ͕Ͱ͖Δ 2. ϓϩόΠμʔͷোΛผϓϩόΠμʔͰΧ όʔͰ͖Δ Pros/Cons ϚϧνΫϥυߏ Cons 1. ӡ༻ͷख͕ؒ૿͑Δ
ɹɹʛɹɹ© 2019 PLAID Inc. 22 2019.11.13ɹɹʛɹɹDatadog Summit Tokyo 2019ɹɹʛɹ ৄࡉΛΓ͍ͨํ
ʮσϒαϛ 2018ʯͷࢿྉͰʂ ϚϧνΫϥυߏ https://speakerdeck.com/tikson/zhan-lue-topointonituite
23 2019.11.13ɹɹʛɹɹDatadog Summit Tokyo 2019ɹɹʛɹ ɹɹʛɹɹ© 2019 PLAID Inc. ͢͜ͱ
KARTEͷಛ KARTEΛࢧ͑ΔࢹͷϙΠϯτ ࠓޙΓ͍ͨ͜ͱ ✓
KARTEΛࢧ͑ΔࢹͷϙΠϯτ 24 ɹɹʛɹɹ© 2019 PLAID Inc. 2019.11.13ɹɹʛɹɹDatadog Summit Tokyo 2019ɹɹʛɹ
ɾϦΞϧλΠϜੑ ɾॊೈੑ ɾεέʔϥϏϦςΟ ɾ҆ఆੑ
25 ɹɹʛɹɹ© 2019 PLAID Inc. 2019.11.13ɹɹʛɹɹDatadog Summit Tokyo 2019ɹɹʛɹ KARTEͷಛ
→ 1.Ϛϧνςφϯτ 2.ϨΠςϯγʔ/ߴෛՙ 3.େنγεςϜ 4.ϚϧνΫϥυߏ
26 ɹɹʛɹɹ© 2019 PLAID Inc. 2019.11.13ɹɹʛɹɹDatadog Summit Tokyo 2019ɹɹʛɹ KARTEΛࢧ͑Δ
ࢹͷϙΠϯτ 1.Datadogʹશͯू 2.Custom metricsͷੵۃ׆༻ 3.LayerΛ͚ͨDashboard ͢͜ͱ →
27 ɹɹʛɹɹ© 2019 PLAID Inc. 2019.11.13ɹɹʛɹɹDatadog Summit Tokyo 2019ɹɹʛɹ →
1.Datadogʹશͯू 2.Custom metricsͷੵۃ׆༻ 3.LayerΛ͚ͨDashboard 1.Ϛϧνςφϯτ 2.ϨΠςϯγʔ/ߴෛՙ 3.େنγεςϜ 4.ϚϧνΫϥυߏ
ɹɹʛɹɹ© 2019 PLAID Inc. 28 2019.11.13ɹɹʛɹɹDatadog Summit Tokyo 2019ɹɹʛɹ োରԠ࣌ʹDatadogΛݟΕ
͙͢ʹঢ়گΛѲͰ͖ΔΑ͏ʹ Datadogʹશͯू
29 ɹɹʛɹɹ© 2019 PLAID Inc. 2019.11.13ɹɹʛɹɹDatadog Summit Tokyo 2019ɹɹʛɹ KARTEΛࢧ͑Δ
ࢹͷϙΠϯτ ͢͜ͱ → 1.Datadogʹશͯू 2.Custom metricsͷੵۃ׆༻ 3.LayerΛ͚ͨDashboard
30 ɹɹʛɹɹ© 2019 PLAID Inc. 2019.11.13ɹɹʛɹɹDatadog Summit Tokyo 2019ɹɹʛɹ →
1.Datadogʹશͯू 2.Custom metricsͷੵۃ׆༻ 3.LayerΛ͚ͨDashboard 1.Ϛϧνςφϯτ 2.ϨΠςϯγʔ/ߴෛՙ 3.େنγεςϜ 4.ϚϧνΫϥυߏ
ɹɹʛɹɹ© 2019 PLAID Inc. 31 2019.11.13ɹɹʛɹɹDatadog Summit Tokyo 2019ɹɹʛɹ 32M/mon
΄Ͳར༻ Custom metricsͷੵۃ׆༻
Custom metricsͷੵۃ׆༻ 32 ɹɹʛɹɹ© 2019 PLAID Inc. 2019.11.13ɹɹʛɹɹDatadog Summit Tokyo
2019ɹɹʛɹ https://github.com/makinoy/libs-dogstatsd ɾlibs-dogstatsdͰɺ؆୯ʹॲཧͷ࣮ߦճ ࣮ߦ࣌ؒΛଌఆͰ͖ΔΑ͏ʹ ɾؔΑΓࡉ͔͍୯ҐͰlatencyΛଌఆ ɾϢʔβ୯ҐͰͷύϑΥʔϚϯεΛଌఆ 1 const dd = require('libs-dogstatsd'); 2 const stats = dd.start(); 3 // measure hoge 4 hoge(); 5 stats.tick('hoge', 1, 1, [user_id:1234]); Node.jsͷwrapper library
ɹɹʛɹɹ© 2019 PLAID Inc. 33 2019.11.13ɹɹʛɹɹDatadog Summit Tokyo 2019ɹɹʛɹ tagͰϢʔβࣝผ༻IDΛૹΔ
ࠓى͍ͬͯ͜Δ͕ ʮಛఆͷϢʔβىҼͰى͍ͬͯ͜Δʁʯ ʮKARTEશମͰى͍ͬͯ͜Δʁʯ ͔ΛΓ͚͍ͯ͘͢͠͠Δ Custom metricsͷੵۃ׆༻
34 ɹɹʛɹɹ© 2019 PLAID Inc. 2019.11.13ɹɹʛɹɹDatadog Summit Tokyo 2019ɹɹʛɹ KARTEΛࢧ͑Δ
ࢹͷϙΠϯτ ͢͜ͱ → 1.Datadogʹશͯू 2.Custom metricsͷੵۃ׆༻ 3.LayerΛ͚ͨDashboard
35 ɹɹʛɹɹ© 2019 PLAID Inc. 2019.11.13ɹɹʛɹɹDatadog Summit Tokyo 2019ɹɹʛɹ →
1.Datadogʹશͯू 2.Custom metricsͷੵۃ׆༻ 3.LayerΛ͚ͨDashboard 1.Ϛϧνςφϯτ 2.ϨΠςϯγʔ/ߴෛՙ 3.େنγεςϜ 4.ϚϧνΫϥυߏ
ɹɹʛɹɹ© 2019 PLAID Inc. 36 2019.11.13ɹɹʛɹɹDatadog Summit Tokyo 2019ɹɹʛɹ Layer
1 ɾKARTE͕ੜ͖͍ͯΔ͔Ͳ͏͔ LayerΛ͚ͨDashboard
ɹɹʛɹɹ© 2019 PLAID Inc. 37 2019.11.13ɹɹʛɹɹDatadog Summit Tokyo 2019ɹɹʛɹ Layer
2 ɾॏཁͳίϯϙʔωϯτ ɾॏཁͳΞϓϦέʔγϣϯͷϝτϦοΫ LayerΛ͚ͨDashboard
ɹɹʛɹɹ© 2019 PLAID Inc. 38 2019.11.13ɹɹʛɹɹDatadog Summit Tokyo 2019ɹɹʛɹ Layer
3 ɾ֤αʔό܈ͷࡉ͔͍ϝτϦοΫ LayerΛ͚ͨDashboard
39 2019.11.13ɹɹʛɹɹDatadog Summit Tokyo 2019ɹɹʛɹ ɹɹʛɹɹ© 2019 PLAID Inc. ͢͜ͱ
KARTEͷಛ KARTEΛࢧ͑ΔࢹͷϙΠϯτ ࠓޙΓ͍ͨ͜ͱ ✓ ✓
ɹࠓޙΓ͍ͨ͜ͱ 40 ɹɹʛɹɹ© 2019 PLAID Inc. 2019.11.13ɹɹʛɹɹDatadog Summit Tokyo 2019ɹɹʛɹ
શͯͷΤϯδχΞ͕ োͷҰ࣌ରԠΛͰ͖ΔΑ͏ʹ͢Δ ɾࢹ/োରԠͷϑϩʔͷཧ ɾDatadogΛ͍͜ͳͯ͠Β͏
ɹࠓޙΓ͍ͨ͜ͱ 41 ɹɹʛɹɹ© 2019 PLAID Inc. 2019.11.13ɹɹʛɹɹDatadog Summit Tokyo 2019ɹɹʛɹ
Custom metrics 1ԯԁͷղܾ ɾ͋·ΓʹศརͩͬͨͷͰ͍͢ ͗ͯ͠·ͬͨ… ɾCustom metricsͷཧΛߦ͏
42 2019.11.13ɹɹʛɹɹDatadog Summit Tokyo 2019ɹɹʛɹ ɹɹʛɹɹ© 2019 PLAID Inc. ·ͱΊ
·ͱΊ
43 ɹɹʛɹɹ© 2019 PLAID Inc. 2019.11.13ɹɹʛɹɹDatadog Summit Tokyo 2019ɹɹʛɹ KARTEΛࢧ͑Δ
ࢹͷϙΠϯτ 1.Datadogʹશͯू 2.Custom metricsͷੵۃ׆༻ 3.LayerΛ͚ͨDashboard →
None