Upgrade to Pro — share decks privately, control downloads, hide ads and more …

measuring api performance using druid

2c4b23630d3e6ee69efb4db16186d266?s=47 Ananth Packkildurai
November 28, 2017
Programming
0
1.1k

measuring api performance using druid

Druid with auto scale, monitoring metrics to build trust with our clients and wishlist from Druid.

2c4b23630d3e6ee69efb4db16186d266?s=128

Ananth Packkildurai

November 28, 2017
Tweet

More Decks by Ananth Packkildurai

See All by Ananth Packkildurai
2c4b23630d3e6ee69efb4db16186d266?s=48 vananth22
0
150
2c4b23630d3e6ee69efb4db16186d266?s=48 vananth22
0
64
2c4b23630d3e6ee69efb4db16186d266?s=48 vananth22
0
52
2c4b23630d3e6ee69efb4db16186d266?s=48 vananth22
1
1.4k
2c4b23630d3e6ee69efb4db16186d266?s=48 vananth22
2
1.2k
2c4b23630d3e6ee69efb4db16186d266?s=48 vananth22
1
170

Other Decks in Programming

See All in Programming
9a3693c0c3d12f2fbae51792dd6a1344?s=48 akito0107
0
210
74253b7e85d0a315a23d74db50abd8e4?s=48 coa00
2
170
Ae276805027a01983503c3edafbdb6b2?s=48 martysuzuki
1
570
15934fa2aa7b2ce21f091e9b7cffa856?s=48 manfredsteyer
PRO
0
160
843e2b37af4a84f89438478aa939a324?s=48 heistak
2
130
97a42cc1630ebdfdaaa8d50078a9fd3d?s=48 koheisakata
0
170
3295d80faee461db1093982361256fe4?s=48 panini
1
160
B8403d102456248570005ee7fb2ba0f7?s=48 tooppoo
0
210
32ae0ae04b5c4f1b29ee9e8826823fbc?s=48 joergneumann
0
130
Fb469a72f1b12726aea391f479dc1b6d?s=48 gernotstarke
0
390
75cf8176bf14811428f77d8fe737f0d5?s=48 konstantin_diener
0
130
Cfbe23392787cb3ad4689c5f72463fcc?s=48 makicamel
0
200

Featured

See All Featured
20bfe76b3d6105641f879fe45cfc9272?s=48 bkeepers
408
58k
Ef32e0b1ac5082450dc8c1fca3a26b5c?s=48 cherdarchuk
71
260k
B3ace07412a5178ea778e7eec161fc0a?s=48 jcasabona
8
560
Af6a12710770ad9a7cfd08c97efd40d3?s=48 pedronauck
652
110k
Ded01cdab114abe4ec13511e4c25b9bb?s=48 andyhume
63
3.7k
027d1ebf66cc039a0bd3b55eeadbe75d?s=48 sachag
267
17k
D47656e20ff5e42f125fc5ea0fd636c6?s=48 tmm1
61
9.4k
761be20b5ebd271008bcee1244fc5b52?s=48 matthewcrist
73
7.5k
78b475797a14c84799063c7cd073962f?s=48 holman
288
130k
8ec1383b240b5ba15ffb9743fceb3c0e?s=48 philnash
9
590
053e75a5b48b44d6dd0612795dfb326d?s=48 notwaldorf
16
1.8k
7653c7c449918ff6542880a8c284f5e7?s=48 jponch
103
5.1k

Transcript

  1. Ananth Packkildurai November 28, 2017 1 Measuring Slack API performance

    using Druid

  2. Public launch: 2014 800+ employees across 7 countries worldwide HQ

    in San Francisco Diverse set of industries including software/technology, retail, media, telecom and professional services. About Slack

  3. An unprecedented adoption rate

  4. Agenda 1. A bit history. 2. Druid infrastructure & usecases

    3. Challenges.

  5. A bit history

  6. March 2016 5 350+ 2M Data Engineers Slack employees Active

    users

  7. October 2017 10 800+ 6M Data Engineers Slack employees Active

    users

  8. Data usage 1 in 3 per week 500+ tables 400k

    access data warehouse Tables Events per sec

  9. It is all about Slogs

  10. Well, not exactly

  11. Slog

  12. Slog

  13. Druid infrastructure & usecases

  14. What can go wrong?

  15. We want more...

  16. Performance & Experimentation • Engineering & CE team should be

    able to detect the performance bottleneck proactively. • Engineers should be able to see their experimentation performance in near real-time.

  17. Near Real time Pipeline

  18. Keep the load in DW Kafka predictable. More comfortable to

    upgrade and verify newer Kafka version. Smaller Kafka cluster is relatively more straightforward to operate. Why Analytics Kafka

  19. Druid Architecture

  20. Middle manager Autoscale based on number of running tasks. Historical

    node autoscale based on the segment size. Fault tolerance deployment for overlord & Coordinator Brokers autoscale and load balanced by ELB. Druid Architecture

  21. Challenges

  22. Cascading failures

  23. Forward Index fields

  24. SQL

  25. Bridge the gap between batch and realtime tables.

  26. Thank You! 26