Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
measuring api performance using druid
Search
Ananth Packkildurai
November 28, 2017
Programming
0
1.6k
measuring api performance using druid
Druid with auto scale, monitoring metrics to build trust with our clients and wishlist from Druid.
Ananth Packkildurai
November 28, 2017
Tweet
Share
More Decks by Ananth Packkildurai
See All by Ananth Packkildurai
Data Contracts & Domain Ownership
vananth22
0
93
Data Catalogs - Rebuild the Broken Promise
vananth22
0
79
Functional Data Engineering - A Blueprint for adopting functional principles in data pipeline
vananth22
0
490
Back To The Future: Emerging Trends in Data Engineering
vananth22
0
1.2k
Murron: A Reliable Monitoring Pipeline
vananth22
0
380
The_journey_towards_Pinot.pdf
vananth22
0
210
Reliable_Event_Pipeline___scale.pdf
vananth22
0
180
Operating Data Pipeline with Airflow @ Slack
vananth22
1
2.4k
Streaming data pipelines @ Slack
vananth22
2
2.3k
Other Decks in Programming
See All in Programming
QA環境で誰でも自由自在に現在時刻を操って検証できるようにした話
kalibora
1
110
php-conference-japan-2024
tasuku43
0
400
競技プログラミングへのお誘い@阪大BOOSTセミナー
kotamanegi
0
390
Оптимизируем производительность блока Казначейство
lamodatech
0
860
今年一番支援させていただいたのは認証系サービスでした
satoshi256kbyte
1
280
良いユニットテストを書こう
mototakatsu
11
3.5k
オニオンアーキテクチャを使って、 Unityと.NETでコードを共有する
soi013
0
340
Fibonacci Function Gallery - Part 2
philipschwarz
PRO
0
200
CloudflareStack でRAGに入門
asahiiwm
0
140
rails newと同時に型を書く
aki19035vc
4
570
Go の GC の不得意な部分を克服したい
taiyow
3
960
情報漏洩させないための設計
kubotak
5
1.2k
Featured
See All Featured
Large-scale JavaScript Application Architecture
addyosmani
510
110k
Distributed Sagas: A Protocol for Coordinating Microservices
caitiem20
330
21k
Building Better People: How to give real-time feedback that sticks.
wjessup
366
19k
Helping Users Find Their Own Way: Creating Modern Search Experiences
danielanewman
29
2.4k
Why Our Code Smells
bkeepers
PRO
335
57k
Writing Fast Ruby
sferik
628
61k
A better future with KSS
kneath
238
17k
The Art of Delivering Value - GDevCon NA Keynote
reverentgeek
8
1.2k
ReactJS: Keep Simple. Everything can be a component!
pedronauck
666
120k
Music & Morning Musume
bryan
46
6.3k
Intergalactic Javascript Robots from Outer Space
tanoku
270
27k
Building an army of robots
kneath
302
44k
Transcript
Ananth Packkildurai November 28, 2017 1 Measuring Slack API performance
using Druid
Public launch: 2014 800+ employees across 7 countries worldwide HQ
in San Francisco Diverse set of industries including software/technology, retail, media, telecom and professional services. About Slack
An unprecedented adoption rate
Agenda 1. A bit history. 2. Druid infrastructure & usecases
3. Challenges.
A bit history
March 2016 5 350+ 2M Data Engineers Slack employees Active
users
October 2017 10 800+ 6M Data Engineers Slack employees Active
users
Data usage 1 in 3 per week 500+ tables 400k
access data warehouse Tables Events per sec
It is all about Slogs
Well, not exactly
Slog
Slog
Druid infrastructure & usecases
What can go wrong?
We want more...
Performance & Experimentation • Engineering & CE team should be
able to detect the performance bottleneck proactively. • Engineers should be able to see their experimentation performance in near real-time.
Near Real time Pipeline
Keep the load in DW Kafka predictable. More comfortable to
upgrade and verify newer Kafka version. Smaller Kafka cluster is relatively more straightforward to operate. Why Analytics Kafka
Druid Architecture
Middle manager Autoscale based on number of running tasks. Historical
node autoscale based on the segment size. Fault tolerance deployment for overlord & Coordinator Brokers autoscale and load balanced by ELB. Druid Architecture
Challenges
Cascading failures
Forward Index fields
SQL
Bridge the gap between batch and realtime tables.
Thank You! 26