Fifth Elephant - Scalable Realtime Analytics Using Druid

08094f6299310cd9e8567373ee02cd95?s=47 Nishant
July 29, 2016

Fifth Elephant - Scalable Realtime Analytics Using Druid

08094f6299310cd9e8567373ee02cd95?s=128

Nishant

July 29, 2016
Tweet

Transcript

  1. 2.

    © Hortonworks Inc. 2011 – 2016. All Rights Reserved 2

    What is Scalable Realtime Analytics ?
  2. 3.

    © Hortonworks Inc. 2011 – 2016. All Rights Reserved 3

    What is Scalable Realtime Analytics ? ⬢ Fast Response Time ⬢ Critical for interactive user experience
  3. 4.

    © Hortonworks Inc. 2011 – 2016. All Rights Reserved 4

    What is Scalable Realtime Analytics ? ⬢ Data Freshness ⬢ Immediate insights into current data ⬢ Ability to query an event as soon as it occurs
  4. 5.

    © Hortonworks Inc. 2011 – 2016. All Rights Reserved 5

    What is Scalable Realtime Analytics ?
  5. 6.

    © Hortonworks Inc. 2011 – 2016. All Rights Reserved 6

    Agenda History and Motivation Demo Druid Architecture Storage Internals: What makes Druid Fast ? Druid in Practice
  6. 7.

    © Hortonworks Inc. 2011 – 2016. All Rights Reserved 7

    History ⬢ Development started at Metamarkets in 2011 ⬢ Initial use case – power ad-tech analytics product ⬢ Open sourced in late 2012 – GPL licensed initially – Switched to Apache V2 in early 2015
  7. 8.

    © Hortonworks Inc. 2011 – 2016. All Rights Reserved 8

    Motivation ⬢ Interactive real time visualizations on Complex data streams ⬢ Answer BI questions – How many unique male visitors visited my website last month ? – How many products were sold last quarter broken down by a demographic and product category ? ⬢ Not interested in dumping entire dataset
  8. 9.
  9. 10.

    © Hortonworks Inc. 2011 – 2016. All Rights Reserved 10

    Solutions Evaluated ⬢ RDBMS (Postgres, Mysql) – Star schema – Aggregate tables – Query Caching ⬢ Results – ~5.5M rows/sec/core scan rate – 1 day of summarized aggregates == 60M+ rows – 1 query over 1 week of data, 16 cores took ~ 5 seconds – Query caching helped, arbitrary queries were still slow
  10. 11.

    © Hortonworks Inc. 2011 – 2016. All Rights Reserved 11

    Solutions Evaluated ⬢ RDBMS (Postgres, Mysql) ⬢ Scalable ⬢ Realtime ⬢ Fast ⬢ Data Freshness
  11. 12.

    © Hortonworks Inc. 2011 – 2016. All Rights Reserved 12

    Solutions Evaluated ⬢ NoSql – Pre-aggregate all dimensional combinations – Store results in a NoSql store ⬢ Results – Fast queries – Arbitrary queries not possible – Not continuously updated – Pre processing scales exponentially – Example: 500K records – 11 dimensions: 4.5 hours on 15 node hadoop cluster – 14 dimensions: 9 hours on 25 node hadoop cluster
  12. 13.

    © Hortonworks Inc. 2011 – 2016. All Rights Reserved 13

    Solutions Evaluated ⬢ NoSql ⬢ Scalable (pre-computation) ⬢ Realtime ⬢ Fast ⬢ Data Freshness
  13. 14.
  14. 15.

    © Hortonworks Inc. 2011 – 2016. All Rights Reserved 15

    What is Druid ? ⬢ Column-oriented distributed datastore ⬢ Sub-Second query latency ⬢ Arbitrary slicing and dicing of data ⬢ Realtime streaming ingestion ⬢ Automatic Data Summarization ⬢ Approximate algorithms (hyperLogLog, theta) ⬢ Scalable to petabytes of data ⬢ Highly available
  15. 16.

    © Hortonworks Inc. 2011 – 2016. All Rights Reserved 16

    Solutions Evaluated ⬢ Druid ⬢ Scalable ⬢ Realtime ⬢ Fast ⬢ Data Freshness
  16. 19.

    © Hortonworks Inc. 2011 – 2016. All Rights Reserved 19

    Node Types ⬢ Realtime Nodes ⬢ Historical Nodes ⬢ Broker Nodes ⬢ Coordinator Nodes
  17. 20.

    © Hortonworks Inc. 2011 – 2016. All Rights Reserved Realtime

    Nodes Historical Nodes 20 Druid Architecture Batch Data Event Historical Nodes Broker Nodes Realtime Nodes Streaming Data Historical Nodes Handoff
  18. 21.

    © Hortonworks Inc. 2011 – 2016. All Rights Reserved 21

    Druid Architecture Batch Data Queries Metadata Store Coordinator Nodes Zookeeper Historical Nodes Broker Nodes Realtime Nodes Streaming Data Handoff
  19. 22.

    © Hortonworks Inc. 2011 – 2016. All Rights Reserved 22

    Storage Internals : What makes Druid Fast ?
  20. 23.

    © Hortonworks Inc. 2011 – 2016. All Rights Reserved 23

    Example Wikipedia Edit Dataset timestamp page language city country … added deleted 2011-01-01T00:01:35Z Justin Bieber en SF USA 10 65 2011-01-01T00:03:63Z Justin Bieber en SF USA 15 62 2011-01-01T00:04:51Z Justin Bieber en SF USA 32 45 2011-01-01T00:05:35Z Ke$ha en Calgary CA 17 87 2011-01-01T00:06:41Z Ke$ha en Calgary CA 43 99 2011-01-02T00:08:35Z Selena Gomes en Calgary CA 12 53 Timestamp Dimensions Metrics
  21. 24.

    © Hortonworks Inc. 2011 – 2016. All Rights Reserved 24

    Data Partitioning timestamp page language city country … added deleted 2011-01-01T00:00:00Z Justin Bieber en SF USA 10 65 2011-01-01T01:00:00Z Justin Bieber en SF USA 15 62 2011-01-01T01:00:00Z Ke$ha en Calgary CA 17 87 2011-01-01T02:00:00Z Ke$ha en Calgary CA 43 99 2011-01-01T02:00:00Z Selena Gomes en Calgary CA 12 53 Segment 2011-01-01T00/2011-01-01T01 Segment 2011-01-01T01/2011-01-01T02 Segment 2011-01-01T02/2011-01-01T03 ⬢ multiple shards for same interval ⬢ hash based ⬢ dimension values based
  22. 25.

    © Hortonworks Inc. 2011 – 2016. All Rights Reserved 25

    Data Rollup timestamp page language city country … added deleted 2011-01-01T00:01:35Z Justin Bieber en SF USA 10 65 2011-01-01T00:03:63Z Justin Bieber en SF USA 15 62 2011-01-01T00:04:51Z Justin Bieber en SF USA 32 45 2011-01-01T00:05:35Z Ke$ha en Calgary CA 17 87 2011-01-01T00:06:41Z Ke$ha en Calgary CA 43 99 2011-01-02T00:08:35Z Selena Gomes en Calgary CA 12 53 timestamp page language city country count sum_added sum_deleted min_added max_added …. 2011-01-01T00:00:00Z Justin Bieber en SF USA 3 57 172 10 32 2011-01-01T00:00:00Z Ke$ha en Calgary CA 2 60 186 17 43 2011-01-02T00:00:00Z Selena Gomes en Calgary CA 1 12 53 12 12 Rollup By Hour
  23. 26.

    © Hortonworks Inc. 2011 – 2016. All Rights Reserved 26

    Dictionary Encoding ⬢ Create and store Ids for each value ⬢ e.g. page column ⬢ Values - Justin Bieber, Ke$ha, Selena Gomes ⬢ Encoding - Justin Bieber : 0, Ke$ha: 1, Selena Gomes: 2 ⬢ Column Data - [0 0 0 1 1 2] ⬢ city column - [0 0 0 1 1 1] timestamp page language city country … added deleted 2011-01-01T00:01:35Z Justin Bieber en SF USA 10 65 2011-01-01T00:03:63Z Justin Bieber en SF USA 15 62 2011-01-01T00:04:51Z Justin Bieber en SF USA 32 45 2011-01-01T00:05:35Z Ke$ha en Calgary CA 17 87 2011-01-01T00:06:41Z Ke$ha en Calgary CA 43 99 2011-01-02T00:08:35Z Selena Gomes en Calgary CA 12 53
  24. 27.

    © Hortonworks Inc. 2011 – 2016. All Rights Reserved 27

    Bitmap Indices ⬢ Store Bitmap Indices for each value ⬢ Justin Bieber -> [0, 1, 2] -> [1 1 1 0 0 0] ⬢ Ke$ha -> [3, 4] -> [0 0 0 1 1 0] ⬢ Selena Gomes -> [5] -> [0 0 0 0 0 1] ⬢ Queries filter evaluated by bitmap OR and AND operations ⬢ Justin Bieber or Ke$ha -> [1 1 1 0 0 0] OR [0 0 0 1 1 0] -> [1 1 1 1 1 0] ⬢ language = en and country = CA -> [1 1 1 1 1 1] AND [0 0 0 1 1 1] -> [0 0 0 1 1 1] ⬢ Indexes compressed with Concise or Roaring encoding timestamp page language city country … added deleted 2011-01-01T00:01:35Z Justin Bieber en SF USA 10 65 2011-01-01T00:03:63Z Justin Bieber en SF USA 15 62 2011-01-01T00:04:51Z Justin Bieber en SF USA 32 45 2011-01-01T00:01:35Z Ke$ha en Calgary CA 17 87 2011-01-01T00:01:35Z Ke$ha en Calgary CA 43 99 2011-01-01T00:01:35Z Selena Gomes en Calgary CA 12 53
  25. 29.

    © Hortonworks Inc. 2011 – 2016. All Rights Reserved 29

    Druid in Production ⬢ Largest known Druid cluster – 50 Trillion+ events – 50PB+ of raw data – Over 500TB of compressed query-able data
  26. 30.

    © Hortonworks Inc. 2011 – 2016. All Rights Reserved 30

    Druid in Production ⬢ Realtime Ingestion Performance – 500,000+ events/sec average – 2 million events/sec peak – 10-100K events/sec/core
  27. 31.

    © Hortonworks Inc. 2011 – 2016. All Rights Reserved 31

    Druid in Production ⬢ Query Latency – average - 500ms – 90%ile < 1sec – 95%ile < 5sec – 99%ile < 10 sec ⬢ Query Volume – 1000s queries per minute
  28. 32.

    © Hortonworks Inc. 2011 – 2016. All Rights Reserved 32

    Druid in Production ⬢ No Downtime ⬢ Data redundancy ⬢ Rolling upgrades 1 1 1 1 1 1 1 1 1 2 2 2 2 2 2 2 2 2 3
  29. 33.

    © Hortonworks Inc. 2011 – 2016. All Rights Reserved 33

    Companies Using Druid Companies Using Druid in Production
  30. 34.

    © Hortonworks Inc. 2011 – 2016. All Rights Reserved 34

    When druid is not IDEAL ⬢ Small amounts of data ⬢ OLTP use cases ⬢ Require frequent single row updates ⬢ Dumping entire dataset
  31. 35.

    © Hortonworks Inc. 2011 – 2016. All Rights Reserved 35

    Community ⬢ User google group - druid-user@googlegroups.com ⬢ Dev google group - druid-dev@googlegroups.com ⬢ Github - druid-io/druid ⬢ IRC - #druid-dev on irc.freenode.net
  32. 36.

    © Hortonworks Inc. 2011 – 2016. All Rights Reserved 36

    Thank you ! Questions ? ⬢ Twitter - @NishantBangarwa ⬢ Gmail - nbangarwa@hortonworks.com ⬢ Linkedin - https://www.linkedin.com/in/nishant-bangarwa
  33. 37.

    © Hortonworks Inc. 2011 – 2016. All Rights Reserved 37

    Druid as a Platform Druid Batch Ingestion (Hadoop, Spark, …) Web Services (Fili) Visualizations (Pivot, Graphana, Caravel) Machine Learning (SciPy, R, ScalaNLP) Streaming Ingestion (Storm, Samza, Spark-Streaming, Kafka, ….)
  34. 38.

    © Hortonworks Inc. 2011 – 2016. All Rights Reserved 38

    Current Druid Architecture Hadoop Historical Node Historical Node Historical Node Batch Data Broker Node Queries ETL (Samza, Kafka, Storm, Spark etc) Streaming Data Realtime Node Realtime Node Handoff