Sometimes, Druid is not the best solution for a business use case

The presentation is about few use cases when you should use druid and when not.

Transcript

1. Sometimes, Druid is not the best solution for a business use case Yulia Trakhtenberg
2. Real Time Dashboard • Optimizing campaigns based on user performance analytics • 8 billion events daily • 12-15 dimensions • Data starting 2011 • Extendable
3. Time based data is mutable! - Life Time Value Session started - date1 Install date purchase - date2 uninstall - date3
4. Previous Solution - Toku (Mongo) KAFKA Toku writers Toku master Toku slaves Dashboard
5. Toku Problems • Failures on weekly basis – master lost – no ability to select a new master • Bad modeling - slow writes • Data loss was possible • No ability to recover
6. Dashboard - DB abstraction level KAFKA Toku writers Toku master Toku slaves Dashboard Middleware (Vishnu)
7. 1. Cassandra 2. mongo 3 3. proprietary DB 4. redis labs + indices DB 5. Pinot 6. Redshift 7. druid We tried...
8. Druid DB • Storage optimized for analytics • Lambda architecture inside • JSON-based query language • Developed by analytics SAAS company • Free and open source • Scalable to petabytes... • Optimized for time based data
9. Druid - take 1 - Batch and Realtime combination KAFKA Druid sink Dashboard Middleware (Vishnu) Daily Batch process In Memory Historical node Daily backup S3 files Druid
10. Druid - take 1 - Going to production • biggest customers cannot open a dashboard • Busy weekend to give Druid a shot • On Sunday morning - dashboard opens and works well!!! • Customer was happy!!! • We moved more and more customers to Druid
11. Druid - problems • Not time based data - LTV • Solution: – Timeseries on event time – Secondary index on event install date • Realtime data - in memory, no secondary indices - too slow
12. Current solution - MemSQL • In Memory DB • Rowstore and Columnstore • Aggregators and Leaves
13. MemSQL Architecture KAFKA MemSQL writers Memsql Cluster Dashboard Middleware (Vishnu) MemSQL writers Memsql Cluster (Slave)
14. MemSQL - why is it a solution • Fast • Extendable • Better modeling • Recoverable solution • Possibility to return to 0 point
15. MemSQL - why is it a solution • Data - 450 GB x 2 clusters • Query Latency - 1-3 seconds • Machines x 2 clusters – 2 aggregators - m4.4xlarge – 4 leaves - r3.4xlarge • Cost reduction -$20K monthly
16. Recovery KAFKA (24h) MemSQL writers Master Memsql Cluster Dashboard Middleware (Vishnu) Yesterday snapshot Recovery Memsql Cluster MemSQL writers - only current day
17. Yesterday S3 files Daily backup S3 files Daily Incremental Batch - Yesterday Snapshot KAFKA Daily Batch Incremental process Past S3 files
18. Next steps - Architecture KAFKA writers - only new data Memsql Rowstore Cluster 1-2 weeks Dashboard Middleware (Vishnu) Daily Batch process S3 files Memsql Columnstore History Cluster Daily
19. When Druid?
20. Thank you Join the Data Team! yulia@appsflyer.com