Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
measuring api performance using druid
Search
Ananth Packkildurai
November 28, 2017
Programming
0
1.7k
measuring api performance using druid
Druid with auto scale, monitoring metrics to build trust with our clients and wishlist from Druid.
Ananth Packkildurai
November 28, 2017
Tweet
Share
More Decks by Ananth Packkildurai
See All by Ananth Packkildurai
Data Contracts & Domain Ownership
vananth22
0
110
Data Catalogs - Rebuild the Broken Promise
vananth22
0
84
Functional Data Engineering - A Blueprint for adopting functional principles in data pipeline
vananth22
0
540
Back To The Future: Emerging Trends in Data Engineering
vananth22
0
1.2k
Murron: A Reliable Monitoring Pipeline
vananth22
0
390
The_journey_towards_Pinot.pdf
vananth22
0
220
Reliable_Event_Pipeline___scale.pdf
vananth22
0
200
Operating Data Pipeline with Airflow @ Slack
vananth22
1
2.5k
Streaming data pipelines @ Slack
vananth22
2
2.4k
Other Decks in Programming
See All in Programming
おやつのお供はお決まりですか?@WWDC25 Recap -Japan-\(region).swift
shingangan
0
140
「テストは愚直&&網羅的に書くほどよい」という誤解 / Test Smarter, Not Harder
munetoshi
0
200
商品比較サービス「マイベスト」における パーソナライズレコメンドの第一歩
ucchiii43
0
180
ソフトウェア品質を数字で捉える技術。事業成長を支えるシステム品質の マネジメント
takuya542
2
15k
スタートアップの急成長を支えるプラットフォームエンジニアリングと組織戦略
sutochin26
1
7.3k
生成AI時代のコンポーネントライブラリの作り方
touyou
1
290
脱Riverpod?fqueryで考える、TanStack Queryライクなアーキテクチャの可能性
ostk0069
0
500
AI駆動のマルチエージェントによる業務フロー自動化の設計と実践
h_okkah
0
230
The Modern View Layer Rails Deserves: A Vision For 2025 And Beyond @ RailsConf 2025, Philadelphia, PA
marcoroth
2
730
レベル1の開発生産性向上に取り組む − 日々の作業の効率化・自動化を通じた改善活動
kesoji
0
300
Advanced Micro Frontends: Multi Version/ Framework Scenarios @WAD 2025, Berlin
manfredsteyer
PRO
0
390
#QiitaBash MCPのセキュリティ
ryosukedtomita
1
1.5k
Featured
See All Featured
Build your cross-platform service in a week with App Engine
jlugia
231
18k
Done Done
chrislema
184
16k
Distributed Sagas: A Protocol for Coordinating Microservices
caitiem20
331
22k
Design and Strategy: How to Deal with People Who Don’t "Get" Design
morganepeng
130
19k
Building Adaptive Systems
keathley
43
2.7k
Evolution of real-time – Irina Nazarova, EuRuKo, 2024
irinanazarova
8
830
Performance Is Good for Brains [We Love Speed 2024]
tammyeverts
10
970
Practical Tips for Bootstrapping Information Extraction Pipelines
honnibal
PRO
21
1.3k
Bootstrapping a Software Product
garrettdimon
PRO
307
110k
jQuery: Nuts, Bolts and Bling
dougneiner
63
7.8k
Product Roadmaps are Hard
iamctodd
PRO
54
11k
Unsuck your backbone
ammeep
671
58k
Transcript
Ananth Packkildurai November 28, 2017 1 Measuring Slack API performance
using Druid
Public launch: 2014 800+ employees across 7 countries worldwide HQ
in San Francisco Diverse set of industries including software/technology, retail, media, telecom and professional services. About Slack
An unprecedented adoption rate
Agenda 1. A bit history. 2. Druid infrastructure & usecases
3. Challenges.
A bit history
March 2016 5 350+ 2M Data Engineers Slack employees Active
users
October 2017 10 800+ 6M Data Engineers Slack employees Active
users
Data usage 1 in 3 per week 500+ tables 400k
access data warehouse Tables Events per sec
It is all about Slogs
Well, not exactly
Slog
Slog
Druid infrastructure & usecases
What can go wrong?
We want more...
Performance & Experimentation • Engineering & CE team should be
able to detect the performance bottleneck proactively. • Engineers should be able to see their experimentation performance in near real-time.
Near Real time Pipeline
Keep the load in DW Kafka predictable. More comfortable to
upgrade and verify newer Kafka version. Smaller Kafka cluster is relatively more straightforward to operate. Why Analytics Kafka
Druid Architecture
Middle manager Autoscale based on number of running tasks. Historical
node autoscale based on the segment size. Fault tolerance deployment for overlord & Coordinator Brokers autoscale and load balanced by ELB. Druid Architecture
Challenges
Cascading failures
Forward Index fields
SQL
Bridge the gap between batch and realtime tables.
Thank You! 26