measuring api performance using druid

2c4b23630d3e6ee69efb4db16186d266?s=47 Ananth Packkildurai
November 28, 2017
Programming
0
1k

measuring api performance using druid

Druid with auto scale, monitoring metrics to build trust with our clients and wishlist from Druid.

2c4b23630d3e6ee69efb4db16186d266?s=128

Ananth Packkildurai

November 28, 2017
Tweet

More Decks by Ananth Packkildurai

See All by Ananth Packkildurai
2c4b23630d3e6ee69efb4db16186d266?s=48 vananth22
0
120
2c4b23630d3e6ee69efb4db16186d266?s=48 vananth22
0
48
2c4b23630d3e6ee69efb4db16186d266?s=48 vananth22
0
42
2c4b23630d3e6ee69efb4db16186d266?s=48 vananth22
1
1.3k
2c4b23630d3e6ee69efb4db16186d266?s=48 vananth22
2
1.1k
2c4b23630d3e6ee69efb4db16186d266?s=48 vananth22
1
160

Other Decks in Programming

See All in Programming
7d1521d950d9bc5697868830cb04ff2f?s=48 yimajo
3
450
C95cdabc569dc4c7750d776687f66037?s=48 yutannihilation
0
120
B49933741d74e122bc1314b2975e9fc9?s=48 mrtc0
2
410
Da1305e5281b5208da85d14a356d01f3?s=48 taka1068
5
230
0d04162c7a3c2907c059a06dbcd23cf4?s=48 tokorom
6
250
127f2573f14072eba29ccb60506663eb?s=48 madai0517
2
270
F3a547c97985b117a1453a85720626fb?s=48 19920930
0
120
52711e2157a6fed933b0361cc06a6953?s=48 marcelgsantos
2
140
6d64a4d4ab6201f2ce57e3befc687785?s=48 yoshi0202
0
130
7e0087a19318ded4ba2203187694740f?s=48 christianweyer
0
110
Cc8ec9594b83b5dc3c7eef345c05dc8d?s=48 shiz
9
960
44df43ecb9d8ae00cafee6b804db3fcd?s=48 kentatada
2
200

Featured

See All Featured
462dad0dd24c568a32dcdb0ae895ae1b?s=48 rasmusluckow
315
18k
78b475797a14c84799063c7cd073962f?s=48 holman
448
130k
433acaea1012b25d97ae66da9b998dad?s=48 malarkey
190
8.2k
A60068bce2e73de3a37ca9d2dbe36092?s=48 bryan
100
11k
Aff5641764408271f7bc398f2097edd0?s=48 denniskardys
219
110k
027d1ebf66cc039a0bd3b55eeadbe75d?s=48 sachag
448
35k
D36d2c1821af9249b69ff7f5ed60529b?s=48 tammielis
236
23k
E76911cbe088e5b850d966de3fc7435b?s=48 robhawkes
52
2.7k
170b760e0147792d0140e59ec78e9893?s=48 eitanlees
106
9.5k
7559f6cff1f5efc2d210965febd4d71c?s=48 bermonpainter
340
26k
766b9746b59e6f09e280cd33cf4ed419?s=48 skipperchong
4
490
0bce06bd06ee7a2c4f5500f25dcf7f18?s=48 paulrobertlloyd
70
3.5k

Transcript

  1. Ananth Packkildurai November 28, 2017 1 Measuring Slack API performance

    using Druid

  2. Public launch: 2014 800+ employees across 7 countries worldwide HQ

    in San Francisco Diverse set of industries including software/technology, retail, media, telecom and professional services. About Slack

  3. An unprecedented adoption rate

  4. Agenda 1. A bit history. 2. Druid infrastructure & usecases

    3. Challenges.

  5. A bit history

  6. March 2016 5 350+ 2M Data Engineers Slack employees Active

    users

  7. October 2017 10 800+ 6M Data Engineers Slack employees Active

    users

  8. Data usage 1 in 3 per week 500+ tables 400k

    access data warehouse Tables Events per sec

  9. It is all about Slogs

  10. Well, not exactly

  11. Slog

  12. Slog

  13. Druid infrastructure & usecases

  14. What can go wrong?

  15. We want more...

  16. Performance & Experimentation • Engineering & CE team should be

    able to detect the performance bottleneck proactively. • Engineers should be able to see their experimentation performance in near real-time.

  17. Near Real time Pipeline

  18. Keep the load in DW Kafka predictable. More comfortable to

    upgrade and verify newer Kafka version. Smaller Kafka cluster is relatively more straightforward to operate. Why Analytics Kafka

  19. Druid Architecture

  20. Middle manager Autoscale based on number of running tasks. Historical

    node autoscale based on the segment size. Fault tolerance deployment for overlord & Coordinator Brokers autoscale and load balanced by ELB. Druid Architecture

  21. Challenges

  22. Cascading failures

  23. Forward Index fields

  24. SQL

  25. Bridge the gap between batch and realtime tables.

  26. Thank You! 26