Upgrade to Pro — share decks privately, control downloads, hide ads and more …

measuring api performance using druid

measuring api performance using druid

Druid with auto scale, monitoring metrics to build trust with our clients and wishlist from Druid.

Ananth Packkildurai

November 28, 2017

More Decks by Ananth Packkildurai

Other Decks in Programming


  1. Ananth Packkildurai November 28, 2017 1 Measuring Slack API performance

    using Druid
  2. Public launch: 2014 800+ employees across 7 countries worldwide HQ

    in San Francisco Diverse set of industries including software/technology, retail, media, telecom and professional services. About Slack
  3. An unprecedented adoption rate

  4. Agenda 1. A bit history. 2. Druid infrastructure & usecases

    3. Challenges.
  5. A bit history

  6. March 2016 5 350+ 2M Data Engineers Slack employees Active

  7. October 2017 10 800+ 6M Data Engineers Slack employees Active

  8. Data usage 1 in 3 per week 500+ tables 400k

    access data warehouse Tables Events per sec
  9. It is all about Slogs

  10. Well, not exactly

  11. Slog

  12. Slog

  13. Druid infrastructure & usecases

  14. What can go wrong?

  15. We want more...

  16. Performance & Experimentation • Engineering & CE team should be

    able to detect the performance bottleneck proactively. • Engineers should be able to see their experimentation performance in near real-time.
  17. Near Real time Pipeline

  18. Keep the load in DW Kafka predictable. More comfortable to

    upgrade and verify newer Kafka version. Smaller Kafka cluster is relatively more straightforward to operate. Why Analytics Kafka
  19. Druid Architecture

  20. Middle manager Autoscale based on number of running tasks. Historical

    node autoscale based on the segment size. Fault tolerance deployment for overlord & Coordinator Brokers autoscale and load balanced by ELB. Druid Architecture
  21. Challenges

  22. Cascading failures

  23. Forward Index fields

  24. SQL

  25. Bridge the gap between batch and realtime tables.

  26. Thank You! 26