Inside Apache Druid : Built for High-Performance Real-time Analytics

Transcript

1. Inside Apache Druid Built for High-Performance Real-time Analytics Surekha Saharan surekha.saharan@imply.io
2. Who am I? Surekha Saharan Committer on Software Engineer at 2
3. Agenda • Background • The problem • Enter the Druid • Under the hood • What’s coming up in future • Try it yourself 3
4. Background
5. Data warehouses Tightly coupled architecture with limited flexibility. 5 Data Data Data Data Sources ETL Data warehouse Processing Store and Compute Analytics Reporting Data mining Querying
6. Data lakes Modern data architectures are more application-centric. 6 Data Data Data Data Sources MapReduce, Spark Apps ETL SQL ML/AI TSDB Data lake Storage
7. Data rivers 7 Data Data Data Data Sources Stream processors Data lake / stream hub Real-time analytics Stream processing ETL Storage Apps Archive to data lake Streaming architectures are true-to-life and enable faster decision cycles.
8. The Problem
9. The Problem Data Exploration Data Ingestion Data Availability
10. The problem 10
11. The problem • Slice-and-dice for big data • Interactive exploration • Look under the hood of reports and dashboards • And we want our data fresh, too 11
12. Challenges • Scale: when data is large, we need a lot of servers • Speed: aiming for sub-second response time • Complexity: too much fine grain to precompute • High dimensionality: 10s or 100s of dimensions • Concurrency: many users and tenants • Freshness: load from streams 12
13. Motivation • Sub-second responses allow dialogue with data • Rapid iteration on questions • Remove barriers to understanding 13
14. high performance analytics data store for event-driven data
15. What is Druid? • “high performance”: low query latency, high ingest rates • “analytics”: counting, ranking, groupBy, time trend • “data store”: the cluster stores a copy of your data • “event-driven data”: fact data like clickstream, network flows, user behavior, digital marketing, server metrics, IoT 15
16. Key features • Column oriented • Scalable to 100s of servers, millions of messages/sec • Time-based partitioning • Indexes for quick filtering • Automatic summarization at ingest time • Query through SQL • Rapid queries on denormalized data 16
17. Use cases • Clickstreams, user behavior • Application performance metrics • Network flows • IoT • Digital advertising • OLAP / business intelligence 17
18. Powered by Druid 18 Source: http://druid.io/druid-powered.html + many more!
19. Where Druid fits in 19 Data lakes Message buses Raw data Storage Analyze Application
20. Why this works • Computers are fast these days • Indexes help save work and cost • But don’t be afraid to scan tables — it can be done efficiently 20
21. 21 How does it work?
22. “Bad programmers worry about the code. Good programmers worry about data structures and their relationships.” ― Linus Torvalds
23. Tricks of the trade • Druid stores data in immutable segments • Global time index • Secondary indexes on individual columns • Memory-mappable compressed columns • Late tuple materialization • Query engines operate directly on compressed data • Vectorization • Rollup (partial aggregation)
24. Druid’s logical data model Timestamp Dimensions Metrics
25. Druid segments 2011-01-01T00:01:35Z Justin Bieber SF 10 5 2011-01-01T00:03:45Z Justin Bieber LA 25 37 2011-01-01T00:05:62Z Justin Bieber SF 15 19 2011-01-01T01:06:33Z Ke$ha LA 30 45 2011-01-01T01:08:51Z Ke$ha LA 16 8 2011-01-01T01:09:17Z Miley Cyrus DC 75 10 2011-01-01T02:23:30Z Miley Cyrus DC 22 12 2011-01-01T02:49:33Z Miley Cyrus DC 90 41 Segment 2011-01-01T00/2011-01-01T01 Segment 2011-01-01T01/2011-01-01T02 Segment 2011-01-01T02/2011-01-01T03 timestamp page city added deleted
26. Columnar storage 26 Dictionary • 0 → Justin Bieber, 1 → Ke$ha Data • page → [0 0 0 1 1 1] • language → [0 0 0 0 0 0] Compression • Dictionary • Bit packing (future: PFOR, RLE, …) • LZ4 (optional)
27. Indexing 27 Query • Justin Bieber OR Ke$ha → [111111] Index • Justin Bieber → [0, 1, 2] → [111000] • Ke$ha → [3, 4, 5] → [000111] Compression • Roaring (https://roaringbitmap.org/) • CONCISE
28. page (STRING) Anatomy of a Druid segment Physical storage format removed (LONG) __time (LONG) 1293840000000 1293840000000 1293840000000 1293840000000 1293840000000 1293840000000 1293840000000 1293840000000 DATA DICT INDEX 0 0 0 1 1 2 2 2 Justin = 0 Ke$ha = 1 Miley = 2 [0,1,2](11100000) [3,4] (00011000) [5,6,7](0000111) 25 42 17 170 112 67 53 94 DATA 2 1 2 1 1 0 0 0 IND [0,2] (10100000) [1,3,4](01011000) [5,6,7](00000111) DICT DC = 0 LA=1 SF = 2 INDEX city (STRING) added (LONG) 1800 2912 1953 3194 5690 1100 8423 9080 Dict encoded (sorted) Bitmap index (stored compressed)
29. Secondary indexing: Roaring bitmaps Divides the possible integers into 16 bit chunks 0, 216-1 216, 232-1 232, 248-1 Inside each chunk one of 3 possible formats(hybrid model) Sorted array {2, 52, 112 ...} Bitset {01001100...} List of runs ([1,10],[16, 22]) https://github.com/RoaringBitmap/RoaringBitmap
30. Compressed column internals 10001 00 1001 1002 1003 1004 1005 1006 1007 1008 1009 1010 1011 1012 1013 1014 1015 1000100 00 00 00 00 00 00 03 E9 00 00 00 00 00 00 03 EA 00 00 00 00 00 00 03 EB 00 00 00 00 00 00 03 EC 00 00 00 00 00 00 03 ED 00 00 00 00 00 00 03 EE 00 00 00 00 00 00 03 EF 00 00 00 00 00 00 03 F0 00 00 00 00 00 00 03 F1 00 00 00 00 00 00 03 F2 00 00 00 00 00 00 03 F3 00 00 00 00 00 00 03 F4 00 00 00 00 00 00 03 F5 00 00 00 00 00 00 03 F6 00 00 00 00 00 00 03 F7 (LONG) 64 bit long representation 0000 0001 0002 0003 0004 0005 0006 0007 0008 0009 0010 0011 0012 0013 0014 Delta encoding Delta encoding LZ4 Compress LZ4 compress block#1 block#2 4 bit encoded representation
31. Query execution pipeline 31 Scan Filter Project Aggregate Filter Project Sort Druid’s query execution pipeline
32. Late materialization 32 Scan Filter Aggregate Scan is a no-op! Load column data necessary for filtering, cache it in case it’s used for later operators too Load column data necessary for aggregation
33. Operating on compressed data 33 • Recall that columnar data is dictionary-encoded as a form of compression • Aggregation operators read dictionary codes and use them as array slots or keys in hashtable • Only when merging inter-segment results (different dictionaries) is a dictionary lookup done Aggregate
34. Operating on compressed data : groupby query Country (Dict ID) rev 100 …. 90 …. 85 …. Country rev USA UK France China ... Segment Aggregate Buffer Segment1 (100K rows) Segment2 (100K rows) Merging inter-segment results eg. SELECT country, SUM(revenue) AS rev FROM tbl GROUP BY country Country (Dict ID) rev 45 …. 67 …. 12 …. 1 billion rows Operate on Dict ID Operate on Dict ID Resolve dict IDs (Use hashtable with string keys) Segment (150K rows)
35. Operating on compressed data: order preserving dicts eg. SELECT SUM(revenue) as total_rev FROM tbl WHERE country LIKE ‘US%’ Find dictID for lower bound (lowerID) Find dictID for upper bound (upperID) Compare dict ids (lowerId <= ID <= upperId) 33 → France 34 → UK 44 → USA comparing ids is >> faster than comparing strings
36. Vectorization 36 public interface SumAggregator { void aggregate(int input); int get(); } public interface VectorAggregator { void aggregate(int[] input); int get(); } vs. Aggregate
37. Vectorization 37 public interface SumAggregator { void aggregate(int input); int get(); } public interface VectorAggregator { void aggregate(int[] input); int get(); } vs. • Minimize # of JVM function calls and related overhead • Improve CPU cache locality • Opens up possibility of using vectorized CPU instructions
38. Vectorization : Query execution time Source: https://static.imply.io/gianm/vb.html
39. Vectorization : Query execution rates Source: https://static.imply.io/gianm/vb.html
40. Rollup • Pre-aggregation at ingestion time • Saves space, better compression • Query performance boost
41. Rollup timestamp page city count sum_added sum_deleted 2011-01-01T00:00:00Z Justin Bieber SF 3 50 61 2011-01-01T00:00:00Z Ke$ha LA 2 46 53 2011-01-01T00:00:00Z Miley Cyrus DC 4 198 88 timestamp page city added deleted 2011-01-01T00:01:35Z Justin Bieber SF 10 5 2011-01-01T00:03:45Z Justin Bieber SF 25 37 2011-01-01T00:05:62Z Justin Bieber SF 15 19 2011-01-01T00:06:33Z Ke$ha LA 30 45 2011-01-01T00:08:51Z Ke$ha LA 16 8 2011-01-01T00:09:17Z Miley Cyrus DC 75 10 2011-01-01T00:11:25Z Miley Cyrus DC 11 25 2011-01-01T00:23:30Z Miley Cyrus DC 22 12 2011-01-01T00:49:33Z Miley Cyrus DC 90 41
42. Roll-up vs no roll-up Do roll-up • Working with space constraint. • No need to retain high cardinality dimensions (like user id, precise location information). • Maximize price / performance. Don’t roll-up • Need the ability to retrieve individual events. • May need to group or filter on any column.
43. Approx Algorithms • Data sketches are lossy data structures • Tradeoff accuracy for reduced storage and improved performance. • Summarize data at ingestion time using sketches • Improves roll-up, reduce memory footprint
44. Summarize with data sketches timestamp page city count sum_ added sum_ deleted userid_sketch 2011-01-01T00:00:00Z Justin Bieber SF 3 50 61 sketch_obj 2011-01-01T00:00:00Z Ke$ha LA 2 46 53 sketch_obj 2011-01-01T00:00:00Z Miley Cyrus DC 4 198 88 sketch_obj timestamp page userid city added deleted 2011-01-01T00:01:3 5Z Justin Bieber user11 SF 10 5 2011-01-01T00:03:4 5Z Justin Bieber user22 SF 25 37 2011-01-01T00:05:6 2Z Justin Bieber user11 SF 15 19 2011-01-01T00:06:3 3Z Ke$ha user33 LA 30 45 2011-01-01T00:08:5 1Z Ke$ha user33 LA 16 8 2011-01-01T00:09:1 7Z Miley Cyrus user11 DC 75 10 2011-01-01T00:11:2 5Z Miley Cyrus user44 DC 11 25 2011-01-01T00:23:3 0Z Miley Cyrus user44 DC 22 12 2011-01-01T00:49:3 3Z Miley Cyrus user55 DC 90 41
45. Approx queries can reduce memory Count Distinct Query (100 million values of size 10 bytes each) Stores all distinct values in a set Stores a HLL sketch of results 1 GB memory Few KBs Exact solution Approx solution
47. Next release • Query Vectorization • Array based group by • Native parallel batch indexing with shuffle • Big indexer process • SQL improvements
49. Download Druid community site: https://druid.apache.org/ Druid community site (legacy): http://druid.io/ Imply distribution: https://imply.io/get-started 49
50. Contribute 50 https://github.com/apache/druid
51. Stay in touch 51 @druidio Join the community! https://druid.apache.org/community/ ASF slack: https://druid.apache.org/community/join-slack Follow Apache Druid on Twitter!