Upgrade to Pro — share decks privately, control downloads, hide ads and more …

measuring api performance using druid

2c4b23630d3e6ee69efb4db16186d266?s=47 Ananth Packkildurai
November 28, 2017
Programming
0
1.1k

measuring api performance using druid

Druid with auto scale, monitoring metrics to build trust with our clients and wishlist from Druid.

2c4b23630d3e6ee69efb4db16186d266?s=128

Ananth Packkildurai

November 28, 2017
Tweet

More Decks by Ananth Packkildurai

See All by Ananth Packkildurai
2c4b23630d3e6ee69efb4db16186d266?s=48 vananth22
0
170
2c4b23630d3e6ee69efb4db16186d266?s=48 vananth22
0
74
2c4b23630d3e6ee69efb4db16186d266?s=48 vananth22
0
60
2c4b23630d3e6ee69efb4db16186d266?s=48 vananth22
1
1.5k
2c4b23630d3e6ee69efb4db16186d266?s=48 vananth22
2
1.3k
2c4b23630d3e6ee69efb4db16186d266?s=48 vananth22
1
180

Other Decks in Programming

See All in Programming
4047c64e3a1e2f81addd4ba675ddc451?s=48 zsmb
0
110
9bf923e39671cde83584e3e926296c13?s=48 kishikawakatsumi
1
530
1fab9d01b25e99522f3dfd01e3d4cb51?s=48 soutaro
3
580
8e8410eb785755ae2cb485f85dbab6e8?s=48 hiroga
1
170
A45681c58220b58ca83a926b1be52e65?s=48 moriatsushi
1
430
218473ff7abad0fcd72865da2b23bb5b?s=48 memory1994
6
630
102398563fd1bf878b35dc2d5ee67cbe?s=48 kamilahsantos
0
160
57c0317c344c1cc5265a78ee0b57a94f?s=48 fukazawashun
0
1.2k
Bd593f2def2a91327c2845cb47dfd359?s=48 hiji
1
250
3135a28c51dfa27396bb525e9972db95?s=48 clown0082
1
140
102398563fd1bf878b35dc2d5ee67cbe?s=48 kamilahsantos
1
440
700eacfd79780dfbd6453c971cadafb8?s=48 martinisoft
1
170

Featured

See All Featured
B83d5b590577969ee79a5ad845411a7d?s=48 ammeep
657
54k
Fde6c3a50ca518a909ef35c22c0ee0e4?s=48 destraynor
222
47k
C51b39fa3c79ccb99dcb8a076bc98287?s=48 brianwarren
82
4.9k
D83926c323d4f9289f947b4b4e76b939?s=48 jensimmons
208
10k
5f83ef806155b403c3393160ce51f955?s=48 jlugia
217
16k
D8fc2580cfaca035f666d9e4ee79a7f7?s=48 mongodb
23
4k
Dafc4723e9a1c067796c0fec6f509774?s=48 lemiorhan
631
48k
Be9caeb9d4ef9944d151af909063ed6e?s=48 samlambert
237
10k
56c4053438af8e8b90d6f53cbb7573be?s=48 jakevdp
776
200k
6bb82b13cda86d3b821fa44100335ddf?s=48 thoeni
4
720
282d599b414022eba5096510f675b6e2?s=48 chrislema
173
14k
5b9fe87ec1faa67a4599782930f45ec9?s=48 sstephenson
147
12k

Transcript

  1. Ananth Packkildurai November 28, 2017 1 Measuring Slack API performance

    using Druid

  2. Public launch: 2014 800+ employees across 7 countries worldwide HQ

    in San Francisco Diverse set of industries including software/technology, retail, media, telecom and professional services. About Slack

  3. An unprecedented adoption rate

  4. Agenda 1. A bit history. 2. Druid infrastructure & usecases

    3. Challenges.

  5. A bit history

  6. March 2016 5 350+ 2M Data Engineers Slack employees Active

    users

  7. October 2017 10 800+ 6M Data Engineers Slack employees Active

    users

  8. Data usage 1 in 3 per week 500+ tables 400k

    access data warehouse Tables Events per sec

  9. It is all about Slogs

  10. Well, not exactly

  11. Slog

  12. Slog

  13. Druid infrastructure & usecases

  14. What can go wrong?

  15. We want more...

  16. Performance & Experimentation • Engineering & CE team should be

    able to detect the performance bottleneck proactively. • Engineers should be able to see their experimentation performance in near real-time.

  17. Near Real time Pipeline

  18. Keep the load in DW Kafka predictable. More comfortable to

    upgrade and verify newer Kafka version. Smaller Kafka cluster is relatively more straightforward to operate. Why Analytics Kafka

  19. Druid Architecture

  20. Middle manager Autoscale based on number of running tasks. Historical

    node autoscale based on the segment size. Fault tolerance deployment for overlord & Coordinator Brokers autoscale and load balanced by ELB. Druid Architecture

  21. Challenges

  22. Cascading failures

  23. Forward Index fields

  24. SQL

  25. Bridge the gap between batch and realtime tables.

  26. Thank You! 26