Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
measuring api performance using druid
Search
Ananth Packkildurai
November 28, 2017
Programming
0
1.6k
measuring api performance using druid
Druid with auto scale, monitoring metrics to build trust with our clients and wishlist from Druid.
Ananth Packkildurai
November 28, 2017
Tweet
Share
More Decks by Ananth Packkildurai
See All by Ananth Packkildurai
Data Contracts & Domain Ownership
vananth22
0
83
Data Catalogs - Rebuild the Broken Promise
vananth22
0
73
Functional Data Engineering - A Blueprint for adopting functional principles in data pipeline
vananth22
0
460
Back To The Future: Emerging Trends in Data Engineering
vananth22
0
1.2k
Murron: A Reliable Monitoring Pipeline
vananth22
0
360
The_journey_towards_Pinot.pdf
vananth22
0
210
Reliable_Event_Pipeline___scale.pdf
vananth22
0
150
Operating Data Pipeline with Airflow @ Slack
vananth22
1
2.3k
Streaming data pipelines @ Slack
vananth22
2
2.2k
Other Decks in Programming
See All in Programming
feature環境をGitHub ActionsとCloudFormationでいい感じに管理する
nealle
2
310
データカタログ運用物語 〜令和6年夏の理想と現実〜
kuro_kurorrr
0
110
Composing an API the *right* way (Droidcon Berlin 2024)
zsmb
1
450
CSC307 Lecture 13
javiergs
PRO
0
150
CSC307 Lecture 12
javiergs
PRO
0
220
AWS CDKにおける「再利用性」を考える / aws-cdk-reusability
gotok365
6
1.3k
3 Effective Rules for Success with Signals in Angular
manfredsteyer
PRO
0
120
AWS初心者ってどうやってAWSを学ぶ?〜アプリエンジニアがやってよかったアーキテクチャ学習方法〜
yamanashi_ren01
0
190
APIのない大学ログインWebサービスをWKWebViewとJavaScriptでアプリ化した話
akidon0000
1
330
入社1ヶ月でここまでやった!Findy Toolsインフラ支援の最適化
rvirus0817
6
1.4k
開発部に不満を持っていたCSがエンジニアにジョブチェンしてわかった「勝手に諦めない」ことの大切さ
sakuraikotone
28
16k
初心者がおさえておきたいAWS CDKのベストプラクティス 2024
konokenj
15
7.3k
Featured
See All Featured
Side Projects
sachag
451
42k
A Philosophy of Restraint
colly
200
16k
The Mythical Team-Month
searls
217
43k
In The Pink: A Labor of Love
frogandcode
139
22k
Building Applications with DynamoDB
mza
89
5.8k
Design and Strategy: How to Deal with People Who Don’t "Get" Design
morganepeng
121
18k
BBQ
matthewcrist
82
9k
Statistics for Hackers
jakevdp
792
220k
The Power of CSS Pseudo Elements
geoffreycrofte
64
5.2k
The Brand Is Dead. Long Live the Brand.
mthomps
52
36k
5 minutes of I Can Smell Your CMS
philhawksworth
200
19k
The Cult of Friendly URLs
andyhume
75
5.9k
Transcript
Ananth Packkildurai November 28, 2017 1 Measuring Slack API performance
using Druid
Public launch: 2014 800+ employees across 7 countries worldwide HQ
in San Francisco Diverse set of industries including software/technology, retail, media, telecom and professional services. About Slack
An unprecedented adoption rate
Agenda 1. A bit history. 2. Druid infrastructure & usecases
3. Challenges.
A bit history
March 2016 5 350+ 2M Data Engineers Slack employees Active
users
October 2017 10 800+ 6M Data Engineers Slack employees Active
users
Data usage 1 in 3 per week 500+ tables 400k
access data warehouse Tables Events per sec
It is all about Slogs
Well, not exactly
Slog
Slog
Druid infrastructure & usecases
What can go wrong?
We want more...
Performance & Experimentation • Engineering & CE team should be
able to detect the performance bottleneck proactively. • Engineers should be able to see their experimentation performance in near real-time.
Near Real time Pipeline
Keep the load in DW Kafka predictable. More comfortable to
upgrade and verify newer Kafka version. Smaller Kafka cluster is relatively more straightforward to operate. Why Analytics Kafka
Druid Architecture
Middle manager Autoscale based on number of running tasks. Historical
node autoscale based on the segment size. Fault tolerance deployment for overlord & Coordinator Brokers autoscale and load balanced by ELB. Druid Architecture
Challenges
Cascading failures
Forward Index fields
SQL
Bridge the gap between batch and realtime tables.
Thank You! 26