Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Druid @ Branch

26290e7e829b985a6bcb44da8213029e?s=47 Imply
March 01, 2017

Druid @ Branch

Branch (http://branch.io/) shares examples of how they use Druid in production.

26290e7e829b985a6bcb44da8213029e?s=128

Imply

March 01, 2017
Tweet

Transcript

  1. Druid @ Branch Enhancing the Data Platform for better Business

    Decisions • Sub Second aggregate queries • Real time analytics dashboard • Live queries for uniques • Instant exploratory analytics Technology powering the Data Platform Performance & Scale Considerations Opportunity for new Apps Monitoring Provisioning & deployment Future Plan Demo Biswajit Das Data Team @biswajit @branch.io Muwon Lum Infra Team @muwon @branch.io
  2. Agenda The Business Problem Technology Gap Data Platform Features Performance

    and Scale Opportunity for new Apps Monitoring Provisioning & deployment Future Plan
  3. The Business Problem • Cannot perform live complex queries •

    Lack of instant access to aggregate data • Gathering unique impressions time consuming • No single pane of glass to view all data • Ad Hoc query requires pre-aggregation Instant access to information at scale was a problem
  4. Technology Gap Key/Value Store (Aerospike) • Pre-compute all permutations of

    possible user queries. • Range scans on event data. • Pre-computing all permutations of all ad-hoc queries can lead to a result sets that grow exponentially with the number of columns of a data sets and can require hours of pre-processing time.
  5. Druid to the rescue…….

  6. High Level Data Pipeline flow SECOR Tranquility Parquet

  7. Batch System

  8. Query path

  9. Performance And Scale • 25 node Production cluster (only Druid)

    • Several hundred terabytes raw data indexed . • Typical complex datasource with 30 dimension and 2 metrics • Real time indexer with ~30k events per second to peak 50k • Hourly bucketed data to support different timezones • Sustained 2B + events day • Thousands of queries per second for online dashboard applications • Serving 11 million query every day
  10. Opportunity for new Apps • Druid helped us to support

    new analytics easily . • Ad hoc reporting . • Visualizing Data. • Exploratory analytics .
  11. None
  12. Provisioning & Deployment SaltStack

  13. Rolling Updates 1 2 3

  14. Future Plan • More robust Query Service . • Migrate

    Hadoop indexer to Spark. • Actively working to migrate streaming pipeline to Flink . • Evaluating to move whole druid stack to Mesos/Docker .
  15. Thank you We are hiring : https://branch.io/careers