measuring api performance using druid

Transcript

1. 1.

   Ananth Packkildurai November 28, 2017 1 Measuring Slack API performance

   using Druid
2. 2.

   Public launch: 2014 800+ employees across 7 countries worldwide HQ

   in San Francisco Diverse set of industries including software/technology, retail, media, telecom and professional services. About Slack
3. 3.

   An unprecedented adoption rate

4. 4.

   Agenda 1. A bit history. 2. Druid infrastructure & usecases

   3. Challenges.
5. 5.

   A bit history

6. 6.

   March 2016 5 350+ 2M Data Engineers Slack employees Active

   users
7. 7.

   October 2017 10 800+ 6M Data Engineers Slack employees Active

   users
8. 8.

   Data usage 1 in 3 per week 500+ tables 400k

   access data warehouse Tables Events per sec
9. 9.

   It is all about Slogs

10. 10.

    Well, not exactly

11. 11.

    Slog

12. 12.

    Slog

13. 13.

    Druid infrastructure & usecases

14. 14.

    What can go wrong?

15. 15.

    We want more...

16. 16.

    Performance & Experimentation • Engineering & CE team should be

    able to detect the performance bottleneck proactively. • Engineers should be able to see their experimentation performance in near real-time.
17. 17.

    Near Real time Pipeline

18. 18.

    Keep the load in DW Kafka predictable. More comfortable to

    upgrade and verify newer Kafka version. Smaller Kafka cluster is relatively more straightforward to operate. Why Analytics Kafka
19. 19.

    Druid Architecture

20. 20.

    Middle manager Autoscale based on number of running tasks. Historical

    node autoscale based on the segment size. Fault tolerance deployment for overlord & Coordinator Brokers autoscale and load balanced by ELB. Druid Architecture
21. 21.

    Challenges

22. 22.

    Cascading failures

23. 23.

    Forward Index fields

24. 24.

    SQL

25. 25.

    Bridge the gap between batch and realtime tables.

26. 26.

    Thank You! 26