Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Speaker Deck
PRO
Sign in
Sign up for free
measuring api performance using druid
Ananth Packkildurai
November 28, 2017
Programming
0
1.4k
measuring api performance using druid
Druid with auto scale, monitoring metrics to build trust with our clients and wishlist from Druid.
Ananth Packkildurai
November 28, 2017
Tweet
Share
More Decks by Ananth Packkildurai
See All by Ananth Packkildurai
Functional Data Engineering - A Blueprint for adopting functional principles in data pipeline
vananth22
0
150
Back To The Future: Emerging Trends in Data Engineering
vananth22
0
1.1k
Murron: A Reliable Monitoring Pipeline
vananth22
0
260
The_journey_towards_Pinot.pdf
vananth22
0
140
Reliable_Event_Pipeline___scale.pdf
vananth22
0
91
Operating Data Pipeline with Airflow @ Slack
vananth22
1
1.9k
Streaming data pipelines @ Slack
vananth22
2
1.8k
Search Infrastructure using Lambda Architecture
vananth22
1
260
Other Decks in Programming
See All in Programming
CDKでValidationする本当の方法 / cdk-validation
gotok365
1
140
KubeClarityで始めるSBOM管理 @3-shake SRE Tech Talk / 3-shake-sre-teck-talk-202212
masayaaoyama
0
290
Kyvernoを利用したKubernetesのポリシー制御
kisokazu
0
170
Circuit⚡
monaapk
0
200
AWS App Runnerがそろそろ本番環境でも使い物になりそう
n1215
PRO
0
820
Domain-Driven Design (Tutorial)
hschwentner
11
15k
Cloudflare Workersと状態管理
chimame
2
430
(新米)エンジニアリングマネージャーのしごと #RSGT2023
murabayashi
9
5.3k
kakutanitalk2022_opening_act
shirotamaki
0
100
An Advanced Introduction to R
nicetak
0
1.5k
監視せなあかんし、五大紙だけにオオカミってな🐺🐺🐺🐺🐺
sadnessojisan
0
350
OSSから学んだPR Descriptionの書き方
fugakkbn
4
120
Featured
See All Featured
YesSQL, Process and Tooling at Scale
rocio
159
12k
Building Flexible Design Systems
yeseniaperezcruz
314
35k
個人開発の失敗を避けるイケてる考え方 / tips for indie hackers
panda_program
29
7.7k
VelocityConf: Rendering Performance Case Studies
addyosmani
317
22k
Art Directing for the Web. Five minutes with CSS Template Areas
malarkey
196
9.8k
Faster Mobile Websites
deanohume
295
29k
RailsConf & Balkan Ruby 2019: The Past, Present, and Future of Rails at GitHub
eileencodes
120
29k
Stop Working from a Prison Cell
hatefulcrawdad
263
18k
実際に使うSQLの書き方 徹底解説 / pgcon21j-tutorial
soudai
44
14k
Building a Modern Day E-commerce SEO Strategy
aleyda
6
4.5k
A Tale of Four Properties
chriscoyier
149
21k
The Straight Up "How To Draw Better" Workshop
denniskardys
225
130k
Transcript
Ananth Packkildurai November 28, 2017 1 Measuring Slack API performance
using Druid
Public launch: 2014 800+ employees across 7 countries worldwide HQ
in San Francisco Diverse set of industries including software/technology, retail, media, telecom and professional services. About Slack
An unprecedented adoption rate
Agenda 1. A bit history. 2. Druid infrastructure & usecases
3. Challenges.
A bit history
March 2016 5 350+ 2M Data Engineers Slack employees Active
users
October 2017 10 800+ 6M Data Engineers Slack employees Active
users
Data usage 1 in 3 per week 500+ tables 400k
access data warehouse Tables Events per sec
It is all about Slogs
Well, not exactly
Slog
Slog
Druid infrastructure & usecases
What can go wrong?
We want more...
Performance & Experimentation • Engineering & CE team should be
able to detect the performance bottleneck proactively. • Engineers should be able to see their experimentation performance in near real-time.
Near Real time Pipeline
Keep the load in DW Kafka predictable. More comfortable to
upgrade and verify newer Kafka version. Smaller Kafka cluster is relatively more straightforward to operate. Why Analytics Kafka
Druid Architecture
Middle manager Autoscale based on number of running tasks. Historical
node autoscale based on the segment size. Fault tolerance deployment for overlord & Coordinator Brokers autoscale and load balanced by ELB. Druid Architecture
Challenges
Cascading failures
Forward Index fields
SQL
Bridge the gap between batch and realtime tables.
Thank You! 26