Pro Yearly is on sale from $80 to $50! »

measuring api performance using druid

2c4b23630d3e6ee69efb4db16186d266?s=47 Ananth Packkildurai
November 28, 2017
Programming
0
1k

measuring api performance using druid

Druid with auto scale, monitoring metrics to build trust with our clients and wishlist from Druid.

2c4b23630d3e6ee69efb4db16186d266?s=128

Ananth Packkildurai

November 28, 2017
Tweet

More Decks by Ananth Packkildurai

See All by Ananth Packkildurai
2c4b23630d3e6ee69efb4db16186d266?s=48 vananth22
0
130
2c4b23630d3e6ee69efb4db16186d266?s=48 vananth22
0
54
2c4b23630d3e6ee69efb4db16186d266?s=48 vananth22
0
47
2c4b23630d3e6ee69efb4db16186d266?s=48 vananth22
1
1.3k
2c4b23630d3e6ee69efb4db16186d266?s=48 vananth22
2
1.1k
2c4b23630d3e6ee69efb4db16186d266?s=48 vananth22
1
160

Other Decks in Programming

See All in Programming
36631e35786e16256423db3c9de70a78?s=48 extendwings
0
290
C0e29dade6ec33081689ac135a269b45?s=48 crgarridos
1
150
Fc815be82d09a2e922d4000b65b9f83b?s=48 74th
1
440
6170c1d1970baf2a36a9ae2955e47ff3?s=48 jimhester
0
260
911650d8fcd455d179e45b4bf6cb7219?s=48 masaykisano
0
200
Cfd5ecaceb5f1479df1f836b4b98f77e?s=48 riscurred
0
180
75486cbfd37125f121cf4a6c5614601c?s=48 hideokamoto
1
1.4k
1dfa2bb34977b1db41d0d46d4505183f?s=48 osuke
2
330
198d69383210fbec8c6bea4a35863c9d?s=48 tippy
3
650
271fad8d53cd1f12f2b4b6d38e3d7bd3?s=48 uzulla
1
190
Eb44761e0fb3a5ec8e23ec28048dd7a5?s=48 stoitsev
1
120
A665efd264afb82cc7d725d7e4847495?s=48 arimura6
0
230

Featured

See All Featured
0bce06bd06ee7a2c4f5500f25dcf7f18?s=48 paulrobertlloyd
71
3.5k
8081b26e05bb4354f7d65ffc34cbbd67?s=48 chriscoyier
683
180k
7cbd3f5f922d798cc312eaf596de7ae6?s=48 iamctodd
13
1.7k
3d4519fd34b76fb265fc0237f3792bd4?s=48 danielanewman
199
19k
Ae14cc4491ac334f9cd23f9f93b4305e?s=48 orderedlist
PRO
327
35k
Af6a12710770ad9a7cfd08c97efd40d3?s=48 pedronauck
651
110k
E190023b66e2b8aa73a842b106920c93?s=48 zenorocha
296
39k
1f74b13f1e5c6c69cb5d7fbaabb1e2cb?s=48 sferik
609
54k
F57e2136eb9895eb95ff5dc8a1c02677?s=48 zakiwarfel
87
3.3k
Bfd6b681ec6e8c9bef82ff0521c364f7?s=48 kastner
54
1.9k
6804f1775cb4babfcc3851298566fbce?s=48 tanoku
85
8.4k
B3d6434763caa0ef5dc4b792662c49f7?s=48 smashingmag
228
17k

Transcript

  1. Ananth Packkildurai November 28, 2017 1 Measuring Slack API performance

    using Druid

  2. Public launch: 2014 800+ employees across 7 countries worldwide HQ

    in San Francisco Diverse set of industries including software/technology, retail, media, telecom and professional services. About Slack

  3. An unprecedented adoption rate

  4. Agenda 1. A bit history. 2. Druid infrastructure & usecases

    3. Challenges.

  5. A bit history

  6. March 2016 5 350+ 2M Data Engineers Slack employees Active

    users

  7. October 2017 10 800+ 6M Data Engineers Slack employees Active

    users

  8. Data usage 1 in 3 per week 500+ tables 400k

    access data warehouse Tables Events per sec

  9. It is all about Slogs

  10. Well, not exactly

  11. Slog

  12. Slog

  13. Druid infrastructure & usecases

  14. What can go wrong?

  15. We want more...

  16. Performance & Experimentation • Engineering & CE team should be

    able to detect the performance bottleneck proactively. • Engineers should be able to see their experimentation performance in near real-time.

  17. Near Real time Pipeline

  18. Keep the load in DW Kafka predictable. More comfortable to

    upgrade and verify newer Kafka version. Smaller Kafka cluster is relatively more straightforward to operate. Why Analytics Kafka

  19. Druid Architecture

  20. Middle manager Autoscale based on number of running tasks. Historical

    node autoscale based on the segment size. Fault tolerance deployment for overlord & Coordinator Brokers autoscale and load balanced by ELB. Druid Architecture

  21. Challenges

  22. Cascading failures

  23. Forward Index fields

  24. SQL

  25. Bridge the gap between batch and realtime tables.

  26. Thank You! 26