Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
measuring api performance using druid
Search
Ananth Packkildurai
November 28, 2017
Programming
0
1.7k
measuring api performance using druid
Druid with auto scale, monitoring metrics to build trust with our clients and wishlist from Druid.
Ananth Packkildurai
November 28, 2017
Tweet
Share
More Decks by Ananth Packkildurai
See All by Ananth Packkildurai
Data Contracts & Domain Ownership
vananth22
0
110
Data Catalogs - Rebuild the Broken Promise
vananth22
0
84
Functional Data Engineering - A Blueprint for adopting functional principles in data pipeline
vananth22
0
540
Back To The Future: Emerging Trends in Data Engineering
vananth22
0
1.2k
Murron: A Reliable Monitoring Pipeline
vananth22
0
390
The_journey_towards_Pinot.pdf
vananth22
0
220
Reliable_Event_Pipeline___scale.pdf
vananth22
0
200
Operating Data Pipeline with Airflow @ Slack
vananth22
1
2.5k
Streaming data pipelines @ Slack
vananth22
2
2.4k
Other Decks in Programming
See All in Programming
Rubyでやりたい駆動開発 / Ruby driven development
chobishiba
1
650
『自分のデータだけ見せたい!』を叶える──Laravel × Casbin で複雑権限をスッキリ解きほぐす 25 分
akitotsukahara
2
620
ruby.wasmで多人数リアルタイム通信ゲームを作ろう
lnit
3
440
猫と暮らす Google Nest Cam生活🐈 / WebRTC with Google Nest Cam
yutailang0119
0
110
Azure AI Foundryではじめてのマルチエージェントワークフロー
seosoft
0
160
GitHub Copilot and GitHub Codespaces Hands-on
ymd65536
2
140
Node-RED を(HTTP で)つなげる MCP サーバーを作ってみた
highu
0
120
XP, Testing and ninja testing
m_seki
3
230
スタートアップの急成長を支えるプラットフォームエンジニアリングと組織戦略
sutochin26
0
4.2k
プロダクト志向ってなんなんだろうね
righttouch
PRO
0
180
20250704_教育事業におけるアジャイルなデータ基盤構築
hanon52_
5
680
datadog dash 2025 LLM observability for reliability and stability
ivry_presentationmaterials
0
450
Featured
See All Featured
It's Worth the Effort
3n
185
28k
Thoughts on Productivity
jonyablonski
69
4.7k
The Invisible Side of Design
smashingmag
301
51k
個人開発の失敗を避けるイケてる考え方 / tips for indie hackers
panda_program
107
19k
Product Roadmaps are Hard
iamctodd
PRO
54
11k
Templates, Plugins, & Blocks: Oh My! Creating the theme that thinks of everything
marktimemedia
31
2.4k
Improving Core Web Vitals using Speculation Rules API
sergeychernyshev
18
960
Fantastic passwords and where to find them - at NoRuKo
philnash
51
3.3k
How GitHub (no longer) Works
holman
314
140k
Keith and Marios Guide to Fast Websites
keithpitt
411
22k
Exploring the Power of Turbo Streams & Action Cable | RailsConf2023
kevinliebholz
34
5.9k
Mobile First: as difficult as doing things right
swwweet
223
9.7k
Transcript
Ananth Packkildurai November 28, 2017 1 Measuring Slack API performance
using Druid
Public launch: 2014 800+ employees across 7 countries worldwide HQ
in San Francisco Diverse set of industries including software/technology, retail, media, telecom and professional services. About Slack
An unprecedented adoption rate
Agenda 1. A bit history. 2. Druid infrastructure & usecases
3. Challenges.
A bit history
March 2016 5 350+ 2M Data Engineers Slack employees Active
users
October 2017 10 800+ 6M Data Engineers Slack employees Active
users
Data usage 1 in 3 per week 500+ tables 400k
access data warehouse Tables Events per sec
It is all about Slogs
Well, not exactly
Slog
Slog
Druid infrastructure & usecases
What can go wrong?
We want more...
Performance & Experimentation • Engineering & CE team should be
able to detect the performance bottleneck proactively. • Engineers should be able to see their experimentation performance in near real-time.
Near Real time Pipeline
Keep the load in DW Kafka predictable. More comfortable to
upgrade and verify newer Kafka version. Smaller Kafka cluster is relatively more straightforward to operate. Why Analytics Kafka
Druid Architecture
Middle manager Autoscale based on number of running tasks. Historical
node autoscale based on the segment size. Fault tolerance deployment for overlord & Coordinator Brokers autoscale and load balanced by ELB. Druid Architecture
Challenges
Cascading failures
Forward Index fields
SQL
Bridge the gap between batch and realtime tables.
Thank You! 26