Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Speaker Deck
PRO
Sign in
Sign up for free
measuring api performance using druid
Ananth Packkildurai
November 28, 2017
Programming
0
1.3k
measuring api performance using druid
Druid with auto scale, monitoring metrics to build trust with our clients and wishlist from Druid.
Ananth Packkildurai
November 28, 2017
Tweet
Share
More Decks by Ananth Packkildurai
See All by Ananth Packkildurai
Back To The Future: Emerging Trends in Data Engineering
vananth22
0
1k
Murron: A Reliable Monitoring Pipeline
vananth22
0
240
The_journey_towards_Pinot.pdf
vananth22
0
120
Reliable_Event_Pipeline___scale.pdf
vananth22
0
85
Operating Data Pipeline with Airflow @ Slack
vananth22
1
1.8k
Streaming data pipelines @ Slack
vananth22
2
1.7k
Search Infrastructure using Lambda Architecture
vananth22
1
240
Other Decks in Programming
See All in Programming
ベストプラクティス・ドリフト
sssssssssssshhhhhhhhhh
1
210
料理の注文メニューの3D化への挑戦
hideg
0
290
Automating Gradle benchmarks at N26
ubiratansoares
PRO
1
140
SAM × Dockerでサーバーレス開発が超捗った話
yu_yukk_y
1
360
段階的な技術的負債の解消方法.pdf
ko2ic
2
930
モデルの定義に基づくバリデーションを実現するためのpydantic入門
daikikatsuragawa
0
120
クラウド KMS の活用 / TOKYO BLOCKCHAIN TECH MEETUP 2022
odanado
PRO
0
190
Better Angular Architectures: Architectures with Standalone Components @DWX2022
manfredsteyer
PRO
1
410
How to Test Your Compose UI (Droidcon Berlin 2022)
stewemetal
1
130
それ全部エラーメッセージに書いてあるよ!〜独学でPHPプログラミングが上達するたった一つの方法〜
77web
1
150
ESM移行は無理だけどおれもSindreのライブラリが使いたい!
sosukesuzuki
2
550
リーダブルテストコード / #vstat
jnchito
48
36k
Featured
See All Featured
The Success of Rails: Ensuring Growth for the Next 100 Years
eileencodes
14
3.8k
Design and Strategy: How to Deal with People Who Don’t "Get" Design
morganepeng
107
16k
Reflections from 52 weeks, 52 projects
jeffersonlam
337
17k
Intergalactic Javascript Robots from Outer Space
tanoku
260
25k
BBQ
matthewcrist
74
7.9k
Bash Introduction
62gerente
598
210k
Atom: Resistance is Futile
akmur
255
20k
Responsive Adventures: Dirty Tricks From The Dark Corners of Front-End
smashingmag
237
19k
Build The Right Thing And Hit Your Dates
maggiecrowley
19
1.2k
Templates, Plugins, & Blocks: Oh My! Creating the theme that thinks of everything
marktimemedia
15
980
Creating an realtime collaboration tool: Agile Flush - .NET Oxford
marcduiker
6
570
Adopting Sorbet at Scale
ufuk
63
7.6k
Transcript
Ananth Packkildurai November 28, 2017 1 Measuring Slack API performance
using Druid
Public launch: 2014 800+ employees across 7 countries worldwide HQ
in San Francisco Diverse set of industries including software/technology, retail, media, telecom and professional services. About Slack
An unprecedented adoption rate
Agenda 1. A bit history. 2. Druid infrastructure & usecases
3. Challenges.
A bit history
March 2016 5 350+ 2M Data Engineers Slack employees Active
users
October 2017 10 800+ 6M Data Engineers Slack employees Active
users
Data usage 1 in 3 per week 500+ tables 400k
access data warehouse Tables Events per sec
It is all about Slogs
Well, not exactly
Slog
Slog
Druid infrastructure & usecases
What can go wrong?
We want more...
Performance & Experimentation • Engineering & CE team should be
able to detect the performance bottleneck proactively. • Engineers should be able to see their experimentation performance in near real-time.
Near Real time Pipeline
Keep the load in DW Kafka predictable. More comfortable to
upgrade and verify newer Kafka version. Smaller Kafka cluster is relatively more straightforward to operate. Why Analytics Kafka
Druid Architecture
Middle manager Autoscale based on number of running tasks. Historical
node autoscale based on the segment size. Fault tolerance deployment for overlord & Coordinator Brokers autoscale and load balanced by ELB. Druid Architecture
Challenges
Cascading failures
Forward Index fields
SQL
Bridge the gap between batch and realtime tables.
Thank You! 26