Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Speaker Deck
PRO
Sign in
Sign up
for free
measuring api performance using druid
Ananth Packkildurai
November 28, 2017
Programming
0
1.1k
measuring api performance using druid
Druid with auto scale, monitoring metrics to build trust with our clients and wishlist from Druid.
Ananth Packkildurai
November 28, 2017
Tweet
Share
More Decks by Ananth Packkildurai
See All by Ananth Packkildurai
vananth22
0
150
vananth22
0
64
vananth22
0
52
vananth22
1
1.4k
vananth22
2
1.2k
vananth22
1
170
Other Decks in Programming
See All in Programming
akito0107
0
210
coa00
2
170
martysuzuki
1
570
manfredsteyer
PRO
0
160
heistak
2
130
koheisakata
0
170
panini
1
160
tooppoo
0
210
joergneumann
0
130
gernotstarke
0
390
konstantin_diener
0
130
makicamel
0
200
Featured
See All Featured
bkeepers
408
58k
cherdarchuk
71
260k
jcasabona
8
560
pedronauck
652
110k
andyhume
63
3.7k
sachag
267
17k
tmm1
61
9.4k
matthewcrist
73
7.5k
holman
288
130k
philnash
9
590
notwaldorf
16
1.8k
jponch
103
5.1k
Transcript
Ananth Packkildurai November 28, 2017 1 Measuring Slack API performance
using Druid
Public launch: 2014 800+ employees across 7 countries worldwide HQ
in San Francisco Diverse set of industries including software/technology, retail, media, telecom and professional services. About Slack
An unprecedented adoption rate
Agenda 1. A bit history. 2. Druid infrastructure & usecases
3. Challenges.
A bit history
March 2016 5 350+ 2M Data Engineers Slack employees Active
users
October 2017 10 800+ 6M Data Engineers Slack employees Active
users
Data usage 1 in 3 per week 500+ tables 400k
access data warehouse Tables Events per sec
It is all about Slogs
Well, not exactly
Slog
Slog
Druid infrastructure & usecases
What can go wrong?
We want more...
Performance & Experimentation • Engineering & CE team should be
able to detect the performance bottleneck proactively. • Engineers should be able to see their experimentation performance in near real-time.
Near Real time Pipeline
Keep the load in DW Kafka predictable. More comfortable to
upgrade and verify newer Kafka version. Smaller Kafka cluster is relatively more straightforward to operate. Why Analytics Kafka
Druid Architecture
Middle manager Autoscale based on number of running tasks. Historical
node autoscale based on the segment size. Fault tolerance deployment for overlord & Coordinator Brokers autoscale and load balanced by ELB. Druid Architecture
Challenges
Cascading failures
Forward Index fields
SQL
Bridge the gap between batch and realtime tables.
Thank You! 26