Upgrade to Pro — share decks privately, control downloads, hide ads and more …

measuring api performance using druid

2c4b23630d3e6ee69efb4db16186d266?s=47 Ananth Packkildurai
November 28, 2017
Programming
0
1.2k

measuring api performance using druid

Druid with auto scale, monitoring metrics to build trust with our clients and wishlist from Druid.

2c4b23630d3e6ee69efb4db16186d266?s=128

Ananth Packkildurai

November 28, 2017
Tweet

More Decks by Ananth Packkildurai

See All by Ananth Packkildurai
2c4b23630d3e6ee69efb4db16186d266?s=48 vananth22
0
170
2c4b23630d3e6ee69efb4db16186d266?s=48 vananth22
0
77
2c4b23630d3e6ee69efb4db16186d266?s=48 vananth22
0
64
2c4b23630d3e6ee69efb4db16186d266?s=48 vananth22
1
1.5k
2c4b23630d3e6ee69efb4db16186d266?s=48 vananth22
2
1.4k
2c4b23630d3e6ee69efb4db16186d266?s=48 vananth22
1
180

Other Decks in Programming

See All in Programming
Ff0453fcbf27a84de92b9cc1f2aed9af?s=48 koike91
1
310
Fe1f65a4116d7d80691072061e0ad560?s=48 keiichiwatanabe
0
130
29dd31db419a2e56cf6a8a11f9de98ee?s=48 masakazoo
2
510
042b7c0e45c53de46667f07de2fb2614?s=48 vixentael
0
410
D76231a2114896dfcc7b79ac69558b79?s=48 yosuke_furukawa
PRO
35
9.7k
E586b46d5b09f8109beb66bb146e4f20?s=48 tamappe
2
190
F301c0275838d562c72ea9177e7ddc68?s=48 fkino
1
140
5b3338b3676d4f7091a575e584f3a4f4?s=48 coe401_
PRO
2
660
5059d3f370ad7e1f4d8de4be79ae1a2c?s=48 rvirus0817
0
660
4fd283672253efa184ec8f63fb1a8d02?s=48 hexarf
2
220
0b26590a0a8b0f1da140ed5de9b68379?s=48 usamik26
3
220
43aec61f9ef926b809d6f16e1f2c97bf?s=48 italocegatta
0
110

Featured

See All Featured
D8fc2580cfaca035f666d9e4ee79a7f7?s=48 mongodb
25
4.1k
5b6a2bd86ef643786a381a212e0ef2f1?s=48 geoffreycrofte
41
1.5k
4394ceb8b277f35ee3da7030384efb5d?s=48 davidbonilla
75
3.8k
3d4519fd34b76fb265fc0237f3792bd4?s=48 danielanewman
6
740
E6c6e133e74c3b83f04d2861deaa1c20?s=48 searls
207
38k
11c84dff30266b907714802088852677?s=48 jasonvnalue
87
8.3k
44b54d2ffcb7857004071cc6795dde11?s=48 cassininazir
350
20k
7cbd3f5f922d798cc312eaf596de7ae6?s=48 iamctodd
28
2.8k
A16eb159db895e3b01d3dc95767ad595?s=48 jonyablonski
37
1.7k
Af6a12710770ad9a7cfd08c97efd40d3?s=48 pedronauck
654
120k
56c4053438af8e8b90d6f53cbb7573be?s=48 jakevdp
778
200k
Eb8975af8e49e19e3dd6b6b84a542e26?s=48 qrush
288
19k

Transcript

  1. Ananth Packkildurai November 28, 2017 1 Measuring Slack API performance

    using Druid

  2. Public launch: 2014 800+ employees across 7 countries worldwide HQ

    in San Francisco Diverse set of industries including software/technology, retail, media, telecom and professional services. About Slack

  3. An unprecedented adoption rate

  4. Agenda 1. A bit history. 2. Druid infrastructure & usecases

    3. Challenges.

  5. A bit history

  6. March 2016 5 350+ 2M Data Engineers Slack employees Active

    users

  7. October 2017 10 800+ 6M Data Engineers Slack employees Active

    users

  8. Data usage 1 in 3 per week 500+ tables 400k

    access data warehouse Tables Events per sec

  9. It is all about Slogs

  10. Well, not exactly

  11. Slog

  12. Slog

  13. Druid infrastructure & usecases

  14. What can go wrong?

  15. We want more...

  16. Performance & Experimentation • Engineering & CE team should be

    able to detect the performance bottleneck proactively. • Engineers should be able to see their experimentation performance in near real-time.

  17. Near Real time Pipeline

  18. Keep the load in DW Kafka predictable. More comfortable to

    upgrade and verify newer Kafka version. Smaller Kafka cluster is relatively more straightforward to operate. Why Analytics Kafka

  19. Druid Architecture

  20. Middle manager Autoscale based on number of running tasks. Historical

    node autoscale based on the segment size. Fault tolerance deployment for overlord & Coordinator Brokers autoscale and load balanced by ELB. Druid Architecture

  21. Challenges

  22. Cascading failures

  23. Forward Index fields

  24. SQL

  25. Bridge the gap between batch and realtime tables.

  26. Thank You! 26