An Introduction to Druid

The history and motivations behind why we built Druid and our plans for the future of the project.

Transcript

1. DRUID
   REAL²TIME EXPLORATORY ANALYTICS ON LARGE DATASETS FANGJIN YANG · SOFTWARE ENGINEER · METAMARKETS
2. DEMO SEE SOME NEAT THINGS MOTIVATION WHY DRUID? ARCHITECTURE PICTURES WITH ARROWS OVERVIEW
3. DEMO IN CASE THE INTERNET DIDN’T WORK PRETEND YOU SAW SOMETHING COOL
4. PROBLEMS
5. 2013 DATA EXPLORATION
6. 2013 DATA INGESTION
7. 2013 AVAILABILITY
8. WHAT WE TRIED
9. 2013 I. RDBMS - Relational Database WHAT WE TRIED
10. 2013 ‣ Star Schema ‣ Aggregate Tables ‣ Query Caching I. RDBMS - THE SETUP
11. 2013 ‣ Queries that were cached • fast ‣ Queries against aggregate tables • fast to acceptable ‣ Queries against base fact table • generally unacceptable I. RDBMS - THE RESULTS
12. 2013 I. RDBMS - PERFORMANCE Naive benchmark scan rate ~5.5M rows / second / core 1 day of summarized aggregates 60M+ rows 1 query over 1 week, 16 cores ~5 seconds Page load with 20 queries over a week of data long time
13. 2013 I. RDBMS - Relational Database WHAT WE TRIED
14. 2013 I. RDMBS - Relational Database II.NoSQL - Key/Value Store WHAT WE TRIED
15. 2013 ‣ Pre-aggregate all dimensional combinations ‣ Store results in a NoSQL store II. NOSQL - THE SETUP ts gender age revenue 1 M 18 $0.15 1 F 25 $1.03 1 F 18 $0.01 Key Value 1 revenue=$1.19 1,M revenue=$0.15 1,F revenue=$1.04 1,18 revenue=$0.16 1,25 revenue=$1.03 1,M,18 revenue=$0.15 1,F,18 revenue=$0.01 1,F,25 revenue=$1.03
16. 2013 ‣ Queries were fast • range scan on primary key ‣ Inflexible • not aggregated, not available ‣ Not continuously updated • aggregate first, then display ‣ Processing scales exponentially II. NOSQL - THE RESULTS
17. 2013 ‣ Dimensional combinations => exponential increase ‣ Tried limiting dimensional depth • still expands exponentially ‣ Example: ~500k records • 11 dimensions, 5-deep • 4.5 hours on a 15-node Hadoop cluster • 14 dimensions, 5-deep • 9 hours on a 25-node Hadoop cluster II. NOSQL - PERFORMANCE
18. 2013 I. RDMBS - Relational Database II.NoSQL - Key/Value Store WHAT WE TRIED
19. 2013 I. RDMBS - Relational Database II.NoSQL - Key/Value Store III. ??? WHAT WE TRIED
20. 2013 ‣ Problem with RDBMS: scans are slow ‣ Problem with NoSQL: computationally intractable WHAT WE LEARNED
21. 2013 ‣ Problem with RDBMS: scans are slow ‣ Problem with NoSQL: computationally intractable ! ! ! ! ! ‣ Tackling the RDBMS issue seems easier WHAT WE LEARNED
22. “ ” INTRODUCING DRUID
24. 2013 KEY FEATURES FAST AGGREGATIONS ARBITRARY SLICE-N-DICE DATA HIGHLY AVAILABLE “REAL-TIME” DRUID
25. REAL-TIME · WHAT DOES IT MEAN?
26. “ ” “REAL-TIME” IS THE NEW “CLOUD” THE NEW ADJECTIVE THAT EVERYONE WANTS BUT NO ONE KNOWS WHAT IT MEANS
27. 2013 WHAT IT MEANS TO US RATE OF QUERY RETURN QUERY TIME < 5S ! INGESTION LATENCY
 EVENT OCCURRENCE TO VISIBILITY < 10S
28. ARCHITECTURE
29. 2013 ARCHITECTURE
30. 2013 ARCHITECTURE Realtime Nodes Query API
31. 2013 ‣ Ingest Data streams ‣ Query data as soon as it is ingested ‣ Buffer data in memory ‣ Periodically “hand off” data REAL-TIME NODES
32. 2013 ARCHITECTURE Realtime Nodes Query API Query API Historical Nodes a Hand Off Data
33. 2013 ‣ Main workhorses of a Druid cluster ‣ Load historical data (most data in a cluster is historical data) ‣ Respond to queries HISTORICAL NODES
34. 2013 ARCHITECTURE Query API Historical Nodes Realtime Nodes Query API Broker Nodes Query API Query Rewrite Scatter/Gather a Hand Off Data
35. 2013 ‣ Knows which nodes hold what data ‣ Query scatter/gather (send requests to nodes and merge results) ‣ Caching BROKER NODES
36. DATA STORAGE
37. 2013 DATA! timestamp page language city country ... added deleted! 2011-01-01T00:01:35Z Justin Bieber en SF USA 10 65! 2011-01-01T00:03:63Z Justin Bieber en SF USA 15 62! 2011-01-01T00:04:51Z Justin Bieber en SF USA 32 45! 2011-01-01T01:00:00Z Ke$ha en Calgary CA 17 87! 2011-01-01T02:00:00Z Ke$ha en Calgary CA 43 99! 2011-01-01T02:00:00Z Ke$ha en Calgary CA 12 53! ...
38. 2013 COLUMN COMPRESSION · DICTIONARIES ‣ Create ids • Justin Bieber -> 0, Ke$ha -> 1 ‣ Store • page -> [0 0 0 1 1 1] • language -> [0 0 0 0 0 0] timestamp page language city country ... added deleted! 2011-01-01T00:01:35Z Justin Bieber en SF USA 10 65! 2011-01-01T00:03:63Z Justin Bieber en SF USA 15 62! 2011-01-01T00:04:51Z Justin Bieber en SF USA 32 45! 2011-01-01T01:00:00Z Ke$ha en Calgary CA 17 87! 2011-01-01T02:00:00Z Ke$ha en Calgary CA 43 99! 2011-01-01T02:00:00Z Ke$ha en Calgary CA 12 53! ...
39. 2013 BITMAP INDICES ‣ Justin Bieber -> [0, 1, 2] -> [111000] ‣ Ke$ha -> [3, 4, 5] -> [000111] timestamp page language city country ... added deleted! 2011-01-01T00:01:35Z Justin Bieber en SF USA 10 65! 2011-01-01T00:03:63Z Justin Bieber en SF USA 15 62! 2011-01-01T00:04:51Z Justin Bieber en SF USA 32 45! 2011-01-01T01:00:00Z Ke$ha en Calgary CA 17 87! 2011-01-01T02:00:00Z Ke$ha en Calgary CA 43 99! 2011-01-01T02:00:00Z Ke$ha en Calgary CA 12 53! ...
40. 2013 FAST AND FLEXIBLE QUERIES JUSTIN BIEBER [1, 1, 0, 0] KE$HA [0, 0, 1, 1] JUSTIN BIEBER OR KE$HA [1, 1, 1, 1] row page 0 Justin(Bieber 1 Justin(Bieber 2 Ke$ha 3 Ke$ha
41. 2013 BITMAP INDEX COMPRESSION ‣ CONCISE compression • Boolean operations directly on compressed indices • Less memory => faster scan rates ! ‣ More details • http://ricerca.mat.uniroma3.it/users/colanton/concise.html
42. 2013 REPLICATION ROLLING DEPLOYMENTS + RESTARTS GROW = START PROCESSES SHRINK = KILL PROCESSES 2 YEARS · NO DOWNTIME FOR SOFTWARE UPDATE AVAILABILITY
43. DRUID AS A PLATFORM
44. 2013 ASK ABOUT OUR PLUGINS ‣ Extensible architecture • Build your own modules and extend Druid • Add your own complex metrics (cardinality estimation, approximate histograms and quantiles, approximate top-K algorithms, etc.) • Add your own data ingestion modules (Hadoop, Storm, etc.) • Add your own proprietary modules ‣ Build your own node types from existing modules ‣ Available in Druid 0.6
45. 2013 DRUID AS A PLATFORM Batch Ingestion (Hadoop) Streaming Ingestion (Storm) Druid Approximate Algorithms Machine Learning (?) Visualizations
46. 2013 THE COMMUNITY ‣ Open sourced in Oct. 2012 ‣ Growing Community • ~30 contributors (not all publicly listed) from many different companies • In production at multiple companies, we’re hoping for more! • Support through community forums and IRC • We love contributions!
47. 2013 REALTIME INGESTION 10 – 100K RECORDS / SECOND / NODE ! REAL-WORLD CLUSTER
 150K EVENTS/S (7B EVENTS/DAY)
 500MB/S (2 TB/H) DRUID BENCHMARKS
48. 2013 SCAN SPEED ~33M ROWS / SECOND / CORE INDEX COMPRESSION CONCISE SETS 70% SMALLER THAN BINARY ARRAYS “OR” OPERATION WITH 1M DIMENSION VALUES IN ~5S DRUID BENCHMARKS
49. 2013 TAKE-AWAYS ‣ When Druid? • Interactive, fast exploration of large amounts of data • You need analytics (not key value store) • You want to do your analysis on data as it’s happening (realtime) • Your store needs to be always-on • You need extensibility and flexibility
50. 2013 TAKE-AWAYS ‣ Not Druid? • The amount of data you have can easily be handled by MYSQL • You’re querying for individual entries or doing lookups (not analytics) • You need to constantly do updates and deletes • Batch is good enough • Downtime is no big deal
51. DRUID IS OPEN SOURCE ! WWW.DRUID.IO twitter @druidio irc.freenode.net #druid-dev
52. THANK YOU