Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
マルチクラウドな大規模サービスKARTEを支えるDatadog/multi-cloud mon...
Search
Daiki Matsui
November 13, 2019
Technology
0
2.5k
マルチクラウドな大規模サービスKARTEを支えるDatadog/multi-cloud monitoring with Datadog
Datadog Summit Tokyoでの登壇資料です。
動画は
https://www.youtube.com/watch?v=_Z4HBiwODbc
Daiki Matsui
November 13, 2019
Tweet
Share
More Decks by Daiki Matsui
See All by Daiki Matsui
The Fourth Age of SRE
ikemonn
0
530
Multicloud deploy with Spinnaker / builderscon tokyo 2018
ikemonn
2
4.6k
KARTE を支えるマルチプラットフォームインフラ監視 /karte-multi-platform-monitoring
ikemonn
2
16k
Node Apps Performance Tuning
ikemonn
0
140
Other Decks in Technology
See All in Technology
開発者を支える Internal Developer Portal のイマとコレカラ / To-day and To-morrow of Internal Developer Portals: Supporting Developers
aoto
PRO
1
480
サラリーマンの小遣いで作るtoCサービス - Cloudflare Workersでスケールする開発戦略
shinaps
2
470
CDK CLIで使ってたあの機能、CDK Toolkit Libraryではどうやるの?
smt7174
4
190
Codeful Serverless / 一人運用でもやり抜く力
_kensh
7
450
Claude Code でアプリ開発をオートパイロットにするためのTips集 Zennの場合 / Claude Code Tips in Zenn
wadayusuke
5
1.3k
MagicPod導入から半年、オープンロジQAチームで実際にやったこと
tjoko
0
110
複数サービスを支えるマルチテナント型Batch MLプラットフォーム
lycorptech_jp
PRO
1
940
Snowflake×dbtを用いたテレシーのデータ基盤のこれまでとこれから
sagara
0
120
Snowflake Intelligence × Document AIで“使いにくいデータ”を“使えるデータ”に
kevinrobot34
1
120
S3アクセス制御の設計ポイント
tommy0124
3
200
Rustから学ぶ 非同期処理の仕組み
skanehira
1
150
LLMを搭載したプロダクトの品質保証の模索と学び
qa
0
1.1k
Featured
See All Featured
JavaScript: Past, Present, and Future - NDC Porto 2020
reverentgeek
52
5.6k
The Myth of the Modular Monolith - Day 2 Keynote - Rails World 2024
eileencodes
26
3k
Rebuilding a faster, lazier Slack
samanthasiow
83
9.2k
Dealing with People You Can't Stand - Big Design 2015
cassininazir
367
27k
Learning to Love Humans: Emotional Interface Design
aarron
273
40k
Rails Girls Zürich Keynote
gr2m
95
14k
RailsConf & Balkan Ruby 2019: The Past, Present, and Future of Rails at GitHub
eileencodes
139
34k
GraphQLの誤解/rethinking-graphql
sonatard
72
11k
CSS Pre-Processors: Stylus, Less & Sass
bermonpainter
358
30k
Refactoring Trust on Your Teams (GOTO; Chicago 2020)
rmw
35
3.1k
The Art of Programming - Codeland 2020
erikaheidi
56
13k
"I'm Feeling Lucky" - Building Great Search Experiences for Today's Users (#IAC19)
danielanewman
229
22k
Transcript
1 ϚϧνΫϥυͳେنαʔϏε KARTEΛࢧ͑ΔDatadog ɹɹʛɹɹ© 2019 PLAID Inc. 2019.11.13ɹɹʛɹɹDatadog Summit Tokyo
2019ɹɹʛɹ PLAID, Inc. Daiki Matsui
2 2019.11.13ɹɹʛɹɹDatadog Summit Tokyo 2019ɹɹʛɹ ɹɹʛɹɹ© 2019 PLAID Inc. ࣗݾհ
দҪ େٍ גࣜձࣾϓϨΠυ SRE Senior Architect 2012-2015·ͰגࣜձࣾαΠόʔυʹͯήʔϜ։ൃʹैࣄ 2015ΑΓϓϨΠυʹࢀը͠ɺSREͱͯ͠ϚϧνΫϥυΠϯϑϥͷߏஙɾӡ༻Λ୲ 2015ೖࣾ
ɹɹʛɹɹ© 2019 PLAID Inc. 3 2019.11.13ɹɹʛɹɹDatadog Summit Tokyo 2019ɹɹʛɹ גࣜձࣾϓϨΠυʹ͍ͭͯ
גࣜձࣾϓϨΠυ ౦ژதԝ۠ۜ࠲6-10-1 GINZA SIX 10F ઃཱɿ201110݄ ैۀһɿ130໊ ࢿຊۚɿ149,987,878ԁ ※ࢿຊ४උؚۚΉ
4 2019.11.13ɹɹʛɹɹDatadog Summit Tokyo 2019ɹɹʛɹ ɹɹʛɹɹ© 2019 PLAID Inc. KARTEʹ͍ͭͯ
֬ೝ͢Δ جຊαʔϏε͝ར༻ํ๏Λ͝հ͠·͢ɻ ͓ಘͳಛయཱͭใ͕ຬࡌͰ͢ɻ ॳΊͯͷํ ְתּׅ然钠ׅ 6*رؠ؎ش٦ֽוזַזַ♳麦׃זְծ➙ ״إٔؗ،حف׃ְהְֲ倯䗳铣דׅկ UIرؠ؎ش٦ָ濼גֶֻץֹ 7אךرؠ؎ٕٝ٦ٕ DESIGN RULES » CHECK ׆ה⢪ִ،؎ذيֽ䲧ִת׃կ ֿך堣⠓ֶ鋅鷕׃זֻկ 窫㼎ծ 妜׃ְ ౙͷओɺ Ξλʔ COLLECTION OUTER 嗚稊勴⟝㼰׃㢌刿ֽׅדծ֮זך椚 䟝ך暟⟝ח⳿⠓ִ〳腉䚍ָ넝ֻזתׅկ 勴⟝㢌ִג嗚稊׃גגְֻׁկ ׀䋞劄ך暟⟝כ 鋅אַתׇד׃ַ ꟗׄ ҰਓҰਓʹ߹Θͤͨ ސ٬ମݧΛఏڙ WebαΠτͷ๚ऀͷߦಈΛ ސ٬͝ͱʹϦΞϧλΠϜʹղੳ $9 ސ٬ମݧ ϓϥοτϑΥʔϜ
5 2019.11.13ɹɹʛɹɹDatadog Summit Tokyo 2019ɹɹʛɹ ɹɹʛɹɹ© 2019 PLAID Inc. ͢͜ͱ
KARTEͷಛ KARTEΛࢧ͑ΔࢹͷϙΠϯτ ࠓޙΓ͍ͨ͜ͱ 1 2 3
6 2019.11.13ɹɹʛɹɹDatadog Summit Tokyo 2019ɹɹʛɹ ɹɹʛɹɹ© 2019 PLAID Inc. ͢͜ͱ
KARTEͷಛ KARTEΛࢧ͑ΔࢹͷϙΠϯτ ࠓޙΓ͍ͨ͜ͱ
7 ɹɹʛɹɹ© 2019 PLAID Inc. 2019.11.13ɹɹʛɹɹDatadog Summit Tokyo 2019ɹɹʛɹ KARTEͷಛ
1.Ϛϧνςφϯτ 2.ϨΠςϯγʔ/ߴෛՙ 3.େنγεςϜ 4.ϚϧνΫϥυߏ ͢͜ͱ →
8 ɹɹʛɹɹ© 2019 PLAID Inc. 2019.11.13ɹɹʛɹɹDatadog Summit Tokyo 2019ɹɹʛɹ KARTEͷಛ
1.Ϛϧνςφϯτ 2.ϨΠςϯγʔ/ߴෛՙ 3.େنγεςϜ 4.ϚϧνΫϥυߏ ͢͜ͱ →
9 2019.11.13ɹɹʛɹɹDatadog Summit Tokyo 2019ɹɹʛɹ ɹɹʛɹɹ© 2019 PLAID Inc. Ϛϧνςφϯτ
ɹɹʛɹɹ© 2019 PLAID Inc. 10 2019.11.13ɹɹʛɹɹDatadog Summit Tokyo 2019ɹɹʛɹ ɾ༷ʑͳෛՙͷ͓٬͞Μ͕͍Δ
ɾΞΫηεύλʔϯ৭ʑ Ϛϧνςφϯτ
ɹɹʛɹɹ© 2019 PLAID Inc. 11 2019.11.13ɹɹʛɹɹDatadog Summit Tokyo 2019ɹɹʛɹ ɾෛՙ͕1Ͱ10ഒมԽ͢Δ
ɾηʔϧͰෛՙ͕ٸ૿͢Δ Ϛϧνςφϯτ 1ͷෛՙਪҠ
12 ɹɹʛɹɹ© 2019 PLAID Inc. 2019.11.13ɹɹʛɹɹDatadog Summit Tokyo 2019ɹɹʛɹ KARTEͷಛ
1.Ϛϧνςφϯτ 2.ϨΠςϯγʔ/ߴෛՙ 3.େنγεςϜ 4.ϚϧνΫϥυߏ ͢͜ͱ →
13 2019.11.13ɹɹʛɹɹDatadog Summit Tokyo 2019ɹɹʛɹ ɹɹʛɹɹ© 2019 PLAID Inc. ϨΠςϯγʔɺߴෛՙ
sec to real-time actions 0.x events per day 3 billion stored in BigQuery 2+ PB events (rows) / sec 65,000 streaming insert per month 1 PB analysis per month 60+ PB SRE 3 ϨΠςϯγʔ/ߴෛՙ
14 2019.11.13ɹɹʛɹɹDatadog Summit Tokyo 2019ɹɹʛɹ ɹɹʛɹɹ© 2019 PLAID Inc. ϨΠςϯγʔɺߴෛՙ
sec to real-time actions 0.x events per day 3 billion stored in BigQuery 2+ PB events (rows) / sec 65,000 streaming insert per month 1 PB analysis per month 60+ PB SRE 3 ϨΠςϯγʔ/ߴෛՙ
15 2019.11.13ɹɹʛɹɹDatadog Summit Tokyo 2019ɹɹʛɹ ɹɹʛɹɹ© 2019 PLAID Inc. ϨΠςϯγʔɺߴෛՙ
16 2019.11.13ɹɹʛɹɹDatadog Summit Tokyo 2019ɹɹʛɹ ɹɹʛɹɹ© 2019 PLAID Inc. ϨΠςϯγʔɺߴෛՙ
sec to real-time actions 0.x events per day 3 billion stored in BigQuery 2+ PB events (rows) / sec 65,000 streaming insert per month 1 PB analysis per month 60+ PB SRE 3 ϨΠςϯγʔ/ߴෛՙ
17 ɹɹʛɹɹ© 2019 PLAID Inc. 2019.11.13ɹɹʛɹɹDatadog Summit Tokyo 2019ɹɹʛɹ KARTEͷಛ
1.Ϛϧνςφϯτ 2.ϨΠςϯγʔ/ߴෛՙ 3.େنγεςϜ 4.ϚϧνΫϥυߏ ͢͜ͱ →
18 2019.11.13ɹɹʛɹɹDatadog Summit Tokyo 2019ɹɹʛɹ ɹɹʛɹɹ© 2019 PLAID Inc. େنγεςϜ
19 ɹɹʛɹɹ© 2019 PLAID Inc. 2019.11.13ɹɹʛɹɹDatadog Summit Tokyo 2019ɹɹʛɹ KARTEͷಛ
1.Ϛϧνςφϯτ 2.ϨΠςϯγʔ/ߴෛՙ 3.େنγεςϜ 4.ϚϧνΫϥυߏ ͢͜ͱ →
20 2019.11.13ɹɹʛɹɹDatadog Summit Tokyo 2019ɹɹʛɹ ɹɹʛɹɹ© 2019 PLAID Inc. ϚϧνΫϥυߏ
ɹɹʛɹɹ© 2019 PLAID Inc. 21 2019.11.13ɹɹʛɹɹDatadog Summit Tokyo 2019ɹɹʛɹ Pros
1. ֤ϓϩόΠμʔͷྑ͍ͱ͜औΓ͕Ͱ͖Δ 2. ϓϩόΠμʔͷোΛผϓϩόΠμʔͰΧ όʔͰ͖Δ Pros/Cons ϚϧνΫϥυߏ Cons 1. ӡ༻ͷख͕ؒ૿͑Δ
ɹɹʛɹɹ© 2019 PLAID Inc. 22 2019.11.13ɹɹʛɹɹDatadog Summit Tokyo 2019ɹɹʛɹ ৄࡉΛΓ͍ͨํ
ʮσϒαϛ 2018ʯͷࢿྉͰʂ ϚϧνΫϥυߏ https://speakerdeck.com/tikson/zhan-lue-topointonituite
23 2019.11.13ɹɹʛɹɹDatadog Summit Tokyo 2019ɹɹʛɹ ɹɹʛɹɹ© 2019 PLAID Inc. ͢͜ͱ
KARTEͷಛ KARTEΛࢧ͑ΔࢹͷϙΠϯτ ࠓޙΓ͍ͨ͜ͱ ✓
KARTEΛࢧ͑ΔࢹͷϙΠϯτ 24 ɹɹʛɹɹ© 2019 PLAID Inc. 2019.11.13ɹɹʛɹɹDatadog Summit Tokyo 2019ɹɹʛɹ
ɾϦΞϧλΠϜੑ ɾॊೈੑ ɾεέʔϥϏϦςΟ ɾ҆ఆੑ
25 ɹɹʛɹɹ© 2019 PLAID Inc. 2019.11.13ɹɹʛɹɹDatadog Summit Tokyo 2019ɹɹʛɹ KARTEͷಛ
→ 1.Ϛϧνςφϯτ 2.ϨΠςϯγʔ/ߴෛՙ 3.େنγεςϜ 4.ϚϧνΫϥυߏ
26 ɹɹʛɹɹ© 2019 PLAID Inc. 2019.11.13ɹɹʛɹɹDatadog Summit Tokyo 2019ɹɹʛɹ KARTEΛࢧ͑Δ
ࢹͷϙΠϯτ 1.Datadogʹશͯू 2.Custom metricsͷੵۃ׆༻ 3.LayerΛ͚ͨDashboard ͢͜ͱ →
27 ɹɹʛɹɹ© 2019 PLAID Inc. 2019.11.13ɹɹʛɹɹDatadog Summit Tokyo 2019ɹɹʛɹ →
1.Datadogʹશͯू 2.Custom metricsͷੵۃ׆༻ 3.LayerΛ͚ͨDashboard 1.Ϛϧνςφϯτ 2.ϨΠςϯγʔ/ߴෛՙ 3.େنγεςϜ 4.ϚϧνΫϥυߏ
ɹɹʛɹɹ© 2019 PLAID Inc. 28 2019.11.13ɹɹʛɹɹDatadog Summit Tokyo 2019ɹɹʛɹ োରԠ࣌ʹDatadogΛݟΕ
͙͢ʹঢ়گΛѲͰ͖ΔΑ͏ʹ Datadogʹશͯू
29 ɹɹʛɹɹ© 2019 PLAID Inc. 2019.11.13ɹɹʛɹɹDatadog Summit Tokyo 2019ɹɹʛɹ KARTEΛࢧ͑Δ
ࢹͷϙΠϯτ ͢͜ͱ → 1.Datadogʹશͯू 2.Custom metricsͷੵۃ׆༻ 3.LayerΛ͚ͨDashboard
30 ɹɹʛɹɹ© 2019 PLAID Inc. 2019.11.13ɹɹʛɹɹDatadog Summit Tokyo 2019ɹɹʛɹ →
1.Datadogʹશͯू 2.Custom metricsͷੵۃ׆༻ 3.LayerΛ͚ͨDashboard 1.Ϛϧνςφϯτ 2.ϨΠςϯγʔ/ߴෛՙ 3.େنγεςϜ 4.ϚϧνΫϥυߏ
ɹɹʛɹɹ© 2019 PLAID Inc. 31 2019.11.13ɹɹʛɹɹDatadog Summit Tokyo 2019ɹɹʛɹ 32M/mon
΄Ͳར༻ Custom metricsͷੵۃ׆༻
Custom metricsͷੵۃ׆༻ 32 ɹɹʛɹɹ© 2019 PLAID Inc. 2019.11.13ɹɹʛɹɹDatadog Summit Tokyo
2019ɹɹʛɹ https://github.com/makinoy/libs-dogstatsd ɾlibs-dogstatsdͰɺ؆୯ʹॲཧͷ࣮ߦճ ࣮ߦ࣌ؒΛଌఆͰ͖ΔΑ͏ʹ ɾؔΑΓࡉ͔͍୯ҐͰlatencyΛଌఆ ɾϢʔβ୯ҐͰͷύϑΥʔϚϯεΛଌఆ 1 const dd = require('libs-dogstatsd'); 2 const stats = dd.start(); 3 // measure hoge 4 hoge(); 5 stats.tick('hoge', 1, 1, [user_id:1234]); Node.jsͷwrapper library
ɹɹʛɹɹ© 2019 PLAID Inc. 33 2019.11.13ɹɹʛɹɹDatadog Summit Tokyo 2019ɹɹʛɹ tagͰϢʔβࣝผ༻IDΛૹΔ
ࠓى͍ͬͯ͜Δ͕ ʮಛఆͷϢʔβىҼͰى͍ͬͯ͜Δʁʯ ʮKARTEશମͰى͍ͬͯ͜Δʁʯ ͔ΛΓ͚͍ͯ͘͢͠͠Δ Custom metricsͷੵۃ׆༻
34 ɹɹʛɹɹ© 2019 PLAID Inc. 2019.11.13ɹɹʛɹɹDatadog Summit Tokyo 2019ɹɹʛɹ KARTEΛࢧ͑Δ
ࢹͷϙΠϯτ ͢͜ͱ → 1.Datadogʹશͯू 2.Custom metricsͷੵۃ׆༻ 3.LayerΛ͚ͨDashboard
35 ɹɹʛɹɹ© 2019 PLAID Inc. 2019.11.13ɹɹʛɹɹDatadog Summit Tokyo 2019ɹɹʛɹ →
1.Datadogʹશͯू 2.Custom metricsͷੵۃ׆༻ 3.LayerΛ͚ͨDashboard 1.Ϛϧνςφϯτ 2.ϨΠςϯγʔ/ߴෛՙ 3.େنγεςϜ 4.ϚϧνΫϥυߏ
ɹɹʛɹɹ© 2019 PLAID Inc. 36 2019.11.13ɹɹʛɹɹDatadog Summit Tokyo 2019ɹɹʛɹ Layer
1 ɾKARTE͕ੜ͖͍ͯΔ͔Ͳ͏͔ LayerΛ͚ͨDashboard
ɹɹʛɹɹ© 2019 PLAID Inc. 37 2019.11.13ɹɹʛɹɹDatadog Summit Tokyo 2019ɹɹʛɹ Layer
2 ɾॏཁͳίϯϙʔωϯτ ɾॏཁͳΞϓϦέʔγϣϯͷϝτϦοΫ LayerΛ͚ͨDashboard
ɹɹʛɹɹ© 2019 PLAID Inc. 38 2019.11.13ɹɹʛɹɹDatadog Summit Tokyo 2019ɹɹʛɹ Layer
3 ɾ֤αʔό܈ͷࡉ͔͍ϝτϦοΫ LayerΛ͚ͨDashboard
39 2019.11.13ɹɹʛɹɹDatadog Summit Tokyo 2019ɹɹʛɹ ɹɹʛɹɹ© 2019 PLAID Inc. ͢͜ͱ
KARTEͷಛ KARTEΛࢧ͑ΔࢹͷϙΠϯτ ࠓޙΓ͍ͨ͜ͱ ✓ ✓
ɹࠓޙΓ͍ͨ͜ͱ 40 ɹɹʛɹɹ© 2019 PLAID Inc. 2019.11.13ɹɹʛɹɹDatadog Summit Tokyo 2019ɹɹʛɹ
શͯͷΤϯδχΞ͕ োͷҰ࣌ରԠΛͰ͖ΔΑ͏ʹ͢Δ ɾࢹ/োରԠͷϑϩʔͷཧ ɾDatadogΛ͍͜ͳͯ͠Β͏
ɹࠓޙΓ͍ͨ͜ͱ 41 ɹɹʛɹɹ© 2019 PLAID Inc. 2019.11.13ɹɹʛɹɹDatadog Summit Tokyo 2019ɹɹʛɹ
Custom metrics 1ԯԁͷղܾ ɾ͋·ΓʹศརͩͬͨͷͰ͍͢ ͗ͯ͠·ͬͨ… ɾCustom metricsͷཧΛߦ͏
42 2019.11.13ɹɹʛɹɹDatadog Summit Tokyo 2019ɹɹʛɹ ɹɹʛɹɹ© 2019 PLAID Inc. ·ͱΊ
·ͱΊ
43 ɹɹʛɹɹ© 2019 PLAID Inc. 2019.11.13ɹɹʛɹɹDatadog Summit Tokyo 2019ɹɹʛɹ KARTEΛࢧ͑Δ
ࢹͷϙΠϯτ 1.Datadogʹશͯू 2.Custom metricsͷੵۃ׆༻ 3.LayerΛ͚ͨDashboard →
None