measuring api performance using druid

Druid with auto scale, monitoring metrics to build trust with our clients and wishlist from Druid.

Transcript

1. Ananth Packkildurai November 28, 2017 1 Measuring Slack API performance using Druid
2. Public launch: 2014 800+ employees across 7 countries worldwide HQ in San Francisco Diverse set of industries including software/technology, retail, media, telecom and professional services. About Slack
3. An unprecedented adoption rate
4. Agenda 1. A bit history. 2. Druid infrastructure & usecases 3. Challenges.
5. A bit history
6. March 2016 5 350+ 2M Data Engineers Slack employees Active users
7. October 2017 10 800+ 6M Data Engineers Slack employees Active users
8. Data usage 1 in 3 per week 500+ tables 400k access data warehouse Tables Events per sec
9. It is all about Slogs
10. Well, not exactly
11. Slog
12. Slog
13. Druid infrastructure & usecases
14. What can go wrong?
15. We want more...
16. Performance & Experimentation • Engineering & CE team should be able to detect the performance bottleneck proactively. • Engineers should be able to see their experimentation performance in near real-time.
17. Near Real time Pipeline
18. Keep the load in DW Kafka predictable. More comfortable to upgrade and verify newer Kafka version. Smaller Kafka cluster is relatively more straightforward to operate. Why Analytics Kafka
19. Druid Architecture
20. Middle manager Autoscale based on number of running tasks. Historical node autoscale based on the segment size. Fault tolerance deployment for overlord & Coordinator Brokers autoscale and load balanced by ELB. Druid Architecture
21. Challenges
22. Cascading failures
23. Forward Index fields
24. SQL
25. Bridge the gap between batch and realtime tables.
26. Thank You! 26