Upgrade to Pro — share decks privately, control downloads, hide ads and more …

measuring api performance using druid

2c4b23630d3e6ee69efb4db16186d266?s=47 Ananth Packkildurai
November 28, 2017
Programming
0
1.3k

measuring api performance using druid

Druid with auto scale, monitoring metrics to build trust with our clients and wishlist from Druid.

2c4b23630d3e6ee69efb4db16186d266?s=128

Ananth Packkildurai

November 28, 2017
Tweet

More Decks by Ananth Packkildurai

See All by Ananth Packkildurai
2c4b23630d3e6ee69efb4db16186d266?s=48 vananth22
0
720
2c4b23630d3e6ee69efb4db16186d266?s=48 vananth22
0
210
2c4b23630d3e6ee69efb4db16186d266?s=48 vananth22
0
97
2c4b23630d3e6ee69efb4db16186d266?s=48 vananth22
0
75
2c4b23630d3e6ee69efb4db16186d266?s=48 vananth22
1
1.6k
2c4b23630d3e6ee69efb4db16186d266?s=48 vananth22
2
1.6k
2c4b23630d3e6ee69efb4db16186d266?s=48 vananth22
1
210

Other Decks in Programming

See All in Programming
49ae3cdb8b1a20b56aac93648f202238?s=48 futabato
0
830
D5744e72b6a5e6673f991cab987a31ab?s=48 siketyan
0
160
646a01801bac1886ddf86aee2de913ed?s=48 miyukki
0
130
96466365a9bf7e66990f2b9ba547b02f?s=48 yumu19
1
740
84931a5617d844ecfda982721de3a9e9?s=48 grace2riku
0
180
6a1ff829a4a5333eb16f33ffc99542fd?s=48 torifukukaiou
0
130
3cfad86677c5966ecfe9b81955fa3489?s=48 inoue2002
0
710
C9237e9aa6c8fe028a484c9621991423?s=48 arahabica
0
420
1b031ccbf968811f157cf7a892dddfed?s=48 geshan
0
310
A45d7e872a00eb4478c45fa0f7595a3d?s=48 rilmayer
3
810
7166bc2cbc462ab5fd1987a76d0fe709?s=48 takahirom
1
740
6aaaf03c4f5c681d89c3078bd6070a6f?s=48 rumi_yamaguchi
0
290

Featured

See All Featured
C7393b7ba7ec9c8890dd77d209fbb3c9?s=48 maltzj
505
37k
5f83ef806155b403c3393160ce51f955?s=48 jlugia
218
16k
3d4519fd34b76fb265fc0237f3792bd4?s=48 danielanewman
6
830
282d599b414022eba5096510f675b6e2?s=48 chrislema
174
14k
96270e4c3e5e9806cf7245475c00b275?s=48 addyosmani
1352
200k
Ded01cdab114abe4ec13511e4c25b9bb?s=48 andyhume
68
4.6k
245cee81a9c424266e5e401d844ea881?s=48 lara
175
11k
E837d2996f52008ace055a08fbfff992?s=48 frogandcode
130
20k
E4f5d494ebdc9e7cce1aecf3ce3e8bc1?s=48 shpigford
167
20k
06f8b41980eb4c577fa40c41d5030c19?s=48 keathley
25
990
9cde37f47e4a800ea081ea42de8d749a?s=48 dougneiner
123
8.3k
C9b18d9dff88a9dd2393364c2b3b21bd?s=48 colly
71
3.2k

Transcript

  1. Ananth Packkildurai November 28, 2017 1 Measuring Slack API performance

    using Druid

  2. Public launch: 2014 800+ employees across 7 countries worldwide HQ

    in San Francisco Diverse set of industries including software/technology, retail, media, telecom and professional services. About Slack

  3. An unprecedented adoption rate

  4. Agenda 1. A bit history. 2. Druid infrastructure & usecases

    3. Challenges.

  5. A bit history

  6. March 2016 5 350+ 2M Data Engineers Slack employees Active

    users

  7. October 2017 10 800+ 6M Data Engineers Slack employees Active

    users

  8. Data usage 1 in 3 per week 500+ tables 400k

    access data warehouse Tables Events per sec

  9. It is all about Slogs

  10. Well, not exactly

  11. Slog

  12. Slog

  13. Druid infrastructure & usecases

  14. What can go wrong?

  15. We want more...

  16. Performance & Experimentation • Engineering & CE team should be

    able to detect the performance bottleneck proactively. • Engineers should be able to see their experimentation performance in near real-time.

  17. Near Real time Pipeline

  18. Keep the load in DW Kafka predictable. More comfortable to

    upgrade and verify newer Kafka version. Smaller Kafka cluster is relatively more straightforward to operate. Why Analytics Kafka

  19. Druid Architecture

  20. Middle manager Autoscale based on number of running tasks. Historical

    node autoscale based on the segment size. Fault tolerance deployment for overlord & Coordinator Brokers autoscale and load balanced by ELB. Druid Architecture

  21. Challenges

  22. Cascading failures

  23. Forward Index fields

  24. SQL

  25. Bridge the gap between batch and realtime tables.

  26. Thank You! 26