Druid: A High Performance, Column-oriented, Distributed Data Store

Transcript

1. Druid High performance, column-oriented, distributed data store Fangjin Yang Cofounder @ Imply
2. Overview History & Motivation Demo Alternative Architectures Druid Architecture
3. History & Motivation First lines of Druid started in 2011 Initial use case: power ad-tech analytics product Requirements: - Scalable (trillions of events/day, petabytes of data) - Multi-tenant (thousands of current users) - Interactive (low latency queries) - “Real-time” (low latency data ingestion)
4. History & Motivation Druid went open source in late 2012 - GPL license initially - Part-time development until early 2014 Community growth - Apache v2 licensed in early 2015 - 150+ contributors from 100+ organizations In production at many different companies across many verticals - Ad-tech, network traffic, security, finance, gaming, operations, activity streams, etc.
5. Use Cases Powering user-facing analytic applications Unify historical and real-time events Business intelligence/OLAP queries (slice and dice and drill into data) Behavioral analysis (measuring distinct counts, retention analysis, funnel analysis, A/B testing) Exploratory analytics/root cause analysis
6. Demo In case the internet didn’t work, pretend you saw something cool
7. Business Intelligence Queries Event data - Time, dimensions (attributes), measures Business intelligence/OLAP queries - “How much revenue did product X generate last quarter in SF”? - “How many of my users that visited last week returned this week?” - Not dumping entire data set - Not examining single events - Filtering, grouping, and aggregating data - Result set < input set (aggregations)
8. Solution Space Relational databases (MySQL, Postgres) Key/value stores (HBase, Cassandra) General Compute Engines (Hadoop, Spark) Column stores
9. Relational Database Traditional Data Warehouse - Row oriented - Star schema - Aggregates tables & query caches Fast becoming outdated Slow!
10. Key/Value Stores Very fast writes Very fast lookups Timeseries databases often have K/V storage engines
11. Key/Value Stores Pre-computation - Pre-compute every possible query - Pre-compute a set of queries - Exponential scaling costs
12. Key/Value Stores Range scans - Primary key: dimensions/attributes - Value: measures/metrics (things to aggregate) - Still too slow!
13. Key/Value Stores
14. SQL-on-Hadoop Enable ad-hoc queries on different input formats Examples: Impala, Hive, Spark SQL, Drill, Presto
15. SQL-on-Hadoop
16. Column stores Load/scan exactly what you need for a query Different compression algorithms for different columns - Encoding for string columns - Compression for measure columns Different indexes for different columns
17. Druid
18. Druid Custom column format optimized for event data and BI queries Supports lots of concurrent reads Streaming data ingestion Supports extremely fast filters Ideal for powering user-facing analytic applications
19. Storage Format
20. Raw data timestamp domain gender clicked 2011-01-01T00:01:35Z bieber.com Female 1 2011-01-01T00:03:03Z bieber.com Female 0 2011-01-01T00:04:51Z ultra.com Male 1 2011-01-01T00:05:33Z ultra.com Male 1 2011-01-01T00:05:53Z ultra.com Female 0 2011-01-01T00:06:17Z ultra.com Female 1 2011-01-01T00:23:15Z bieber.com Female 0 2011-01-01T00:38:51Z ultra.com Male 1 2011-01-01T00:49:33Z bieber.com Female 1 2011-01-01T00:49:53Z ultra.com Female 0
21. Summarization timestamp domain gender clicked 2011-01-01T00:00:00Z bieber.com Female 1 2011-01-01T00:00:00Z ultra.com Female 2 2011-01-01T00:00:00Z ultra.com Male 3 timestamp domain gender clicked 2011-01-01T00:01:35Z bieber.com Female 1 2011-01-01T00:03:03Z bieber.com Female 0 2011-01-01T00:04:51Z ultra.com Male 1 2011-01-01T00:05:33Z ultra.com Male 1 2011-01-01T00:05:53Z ultra.com Female 0 2011-01-01T00:06:17Z ultra.com Female 1 2011-01-01T00:23:15Z bieber.com Female 0 2011-01-01T00:38:51Z ultra.com Male 1 2011-01-01T00:49:33Z bieber.com Female 1 2011-01-01T00:49:53Z ultra.com Female 0
22. Segmentation
23. Immutable Segments Fundamental storage unit in Druid No contention between reads and writes One thread scans one segment
24. Columnar Storage Create IDs • Justin Bieber -> 0, Ke$ha -> 1 Store • page → [0 0 0 1 1 1] • language → [0 0 0 0 0 0]
25. Columnar Storage Justin Bieber → [0, 1, 2] → [111000] Ke$ha → [3, 4, 5] → [000111] Justin Bieber OR Ke$ha → [111111] Compression!
26. Plugin Architecture Write your own plugins for different computations and components Often used for approximate algorithms - Count distinct (Hyperloglog) - Approximate Histograms - Funnel/behavioral analysis (theta sketches) Approximate algorithms are very powerful for fast queries
27. Approximate Algorithms timestamp domain gender clicked 2011-01-01T00:00:00Z bieber.com Female 1 2011-01-01T00:00:00Z ultra.com Female 2 2011-01-01T00:00:00Z ultra.com Male 3 timestamp domain gender clicked 2011-01-01T00:01:35Z bieber.com Female 1 2011-01-01T00:03:03Z bieber.com Female 0 2011-01-01T00:04:51Z ultra.com Male 1 2011-01-01T00:05:33Z ultra.com Male 1 2011-01-01T00:05:53Z ultra.com Female 0 2011-01-01T00:06:17Z ultra.com Female 1 2011-01-01T00:23:15Z bieber.com Female 0 2011-01-01T00:38:51Z ultra.com Male 1 2011-01-01T00:49:33Z bieber.com Female 1 2011-01-01T00:49:53Z ultra.com Female 0
28. Approximate Algorithms timestamp domain user gender clicked 2011-01-01T00:01:35Z bieber.com 4312345532 Female 1 2011-01-01T00:03:03Z bieber.com 3484920241 Female 0 2011-01-01T00:04:51Z ultra.com 9530174728 Male 1 2011-01-01T00:05:33Z ultra.com 4098310573 Male 1 2011-01-01T00:05:53Z ultra.com 5832058870 Female 0 2011-01-01T00:06:17Z ultra.com 5789283478 Female 1 2011-01-01T00:23:15Z bieber.com 4730093842 Female 0 2011-01-01T00:38:51Z ultra.com 9530174728 Male 1 2011-01-01T00:49:33Z bieber.com 4930097162 Female 1 2011-01-01T00:49:53Z ultra.com 3081837193 Female 0
29. Approximate Algorithms timestamp domain user gender clicked 2011-01-01T00:01:35Z bieber.com 4312345532 Female 1 2011-01-01T00:03:03Z bieber.com 3484920241 Female 0 2011-01-01T00:04:51Z ultra.com 9530174728 Male 1 2011-01-01T00:05:33Z ultra.com 4098310573 Male 1 2011-01-01T00:05:53Z ultra.com 5832058870 Female 0 2011-01-01T00:06:17Z ultra.com 5789283478 Female 1 2011-01-01T00:23:15Z bieber.com 4730093842 Female 0 2011-01-01T00:38:51Z ultra.com 9530174728 Male 1 2011-01-01T00:49:33Z bieber.com 4930097162 Female 1 2011-01-01T00:49:53Z ultra.com 3081837193 Female 0 timestamp domain gender clicked users 2011-01-01T00:00:00Z bieber.com Female 1 {sketch_data structure} 2011-01-01T00:00:00Z ultra.com Female 2 {sketch_data_structure} 2011-01-01T00:00:00Z ultra.com Male 3 {sketch_data_structure}
30. Architecture
31. Architecture (Batch Ingestion)
32. Architecture (Batch Ingestion)
33. Real-time Nodes Write-optimized data structure: hash map in heap Convert write optimized -> read optimized Read-optimized data structure: Druid segments Query data immediately
34. Architecture (Streaming Ingestion)
35. Architecture (Lambda)
36. Querying Query libraries: - JSON over HTTP - SQL - R - Python - Ruby Open source UIs - Pivot - Grafana - Caravel
37. Druid in Production
38. Ingestion >3M events / second sustained (200B+ events/day) 10 – 100k events / second / core
39. Volume Largest known cluster - >500 TB of segments (>50 trillion raw events, >50 PB raw data) Extremely cost effective at scale
40. Queries 500ms average query latency 90% < 1s, 95% < 2S, 99% < 10s
41. Multi-tenancy Several Hundred queries / second Variety of group by & top-K queries
42. Druid & the Data Space
43. End-to-end Data Stack Druid Stream Processor Batch Processor Message bus Events Apps
44. Integration Druid is complementary to many solutions - SQL-on-Hadoop (Hive, Impala, Spark SQL, Drill, Presto) - Stream processors (Storm, Spark streaming, Flink, Samza) - Batch processors (Spark, Hadoop, Flink) - Messages buses (Kafka, RabbitMQ)
45. Takeaway Druid is pretty good for analytic applications Druid is pretty good at fast OLAP queries Druid is pretty good at streaming ingestion Druid works well with existing data infrastructure systems
46. Thanks! http://imply.io/docs/latest/quickstart