drill介绍

Transcript

1. Drill简介 chenpeng0122@hotmail.com 2013-06-27
2. Designer's Description Interactive Query at Scale Apache Drill provides low latency ad-hoc queries to many different data sources, including nested data. Inspired by Google's Dremel, Drill is designed to scale to 10,000 servers and query petabytes of data in seconds. Flexible and Extensible Architecture Well-documented APIs and interface allow for fast additions of new data sources and file formats. New query languages and operators are also easy to add to Drill.
3. Requirements •支持不同数据源 •支持不同查询接口 •低延时/实时 •交互式查询 •扩展性 •可靠性
4. 覆盖场景
5. High-level Architecture
6. • 源于Google’s Dremel • Full SQL – ANSI SQL 2003 • Nested data • Optional schema • Plug-able datasources Key features
7. • No SQL-like • 整合现有工具 – Datameer, Tableau, Excel, SAP, Crystal Reports – Use standard ODBC/JDBC driver ODBC driver driver datameer excel tableau SAP Query parser Query Planner Drillbit client Drillbit Full SQL – ANSI SQL 2003
8. • Nested data – JSON/BSON,XML,PB, Avro – 一些数据源原生支持( 如MongoDB等) • 扩展到ANSI SQL 2003 • Flattening nested data易错 Nested data
9. Nested data local-logs = donuts.json: { "id":"0003"; "type":"donut", "name":"Dole Fashioned", "ppu":0.55, "sales":300, "batters": { "batter": [ {"id":"1001","type":"Regular"}, {"id":"1002","type":"Chocolate"} ] }, "topping": [ {"id":"5001","type":"None"}, {"id":"5002","type":"Glazed"}, {"id":"5003","type":"Chocolate"}, {"id":"5004","type":"Maple"} ] }
10. • 数据源无严格的schema – Schema修改频繁 – 不同record schema各异(如hbase) • 支持对未知schema的查询 • User can define schema or via discovery Optional Schema
11. • Source query (analyst friendly) • Logical Plan (language agnostic, computer friendly) • Physical Plan ? • Execution plan ？ Principled Query Execution
12. Source Query Parser Logical Plan Optimizer Physical Plan Execution SQL 2003 DrQL MongoQL DSL Parser API query:[ { @id:”log”, op:”sequence”, do:[ { op:”scan”, source:”logs” }, { op:”filter”, condition:”x>3” }. Topology CF etc. Scanner API Principled Query Execution
13. SQL RPC Endpoint HiveQL Pig MongoDB Logical Plan Optimizer Physical Plan Operators Foreman Scheduler Storage Engine inteface DFS Engine Hbase Engine Mysql Engine Distributed Cache Parser Drillbit Modules
14. Logic Architecture SQL Query Parser DrQL Parser Logical Plan Metadata Repo Query Planner Storage Engine -Scanner & Writer -Capabilities and pushdown transformations -Live statistice interface Execution Engine Operator Functions Writer Scanner Physical Plan Schema information Statistics Statistics & Schema Information
15. • 节点：Drillbit - 数据本地化 • Distribute: 协调、查询计划、执行等 • foreman: 任何节点都可以作为查询endpoint Drillbit Storage Process node Drillbit Storage Process node Drillbit Storage Process node Drillbit Storage Process node Curator/Zk Wire-level Architecture
16. Drillbit Storage Process node • Distributed cache(Hazelcast): metadata, 本地 信息等 • Curator/zookeeper: 瞬时的集群成员信息 Distribute Cache Drillbit Storage Process node Distribute Cache Drillbit Storage Process node Distribute Cache Drillbit Storage Process node Distribute Cache Curator/Zk Wire-level Architecture
17. Drillbit Storage Process node • 起始Drillbit作为foreman: 管理查询执行、调 度、位置信息等 • Streaming数据通信避免Ser以及De Distribute Cache Drillbit Storage Process node Distribute Cache Drillbit Storage Process node Distribute Cache Drillbit Storage Process node Distribute Cache Curator/Zk Wire-level Architecture
18. Drillbit Storage Process node Distribute Cache Drillbit Storage Process node Distribute Cache Drillbit Storage Process node Distribute Cache Drillbit Storage Process node Distribute Cache Drillbit Storage Process node Distribute Cache Drillbit Storage Process node Distribute Cache Drillbit Storage Process node Distribute Cache Drillbit Storage Process node Distribute Cache Drillbit Storage Process node Distribute Cache foreman位于多级执行树的根节点, 其他叶子节点激活其数据存储接口 Wire-level Architecture
19. Drillbit Storage Process node Distribute Cache Drillbit Storage Process node Distribute Cache Drillbit Storage Process node Distribute Cache Drillbit Storage Process node Distribute Cache Drillbit Storage Process node Distribute Cache Drillbit Storage Process node Distribute Cache Drillbit Storage Process node Distribute Cache Drillbit Storage Process node Distribute Cache Execution Plan Drillbit Storage Process node Distribute Cache
20. • Source query -> parser API • Custom operators, UDF -> logical plan • Serving tree, CF, topology -> physical plan • Data sources & formats -> scanner API Source Query Parser Logical Plan Optimizer Physical Plan Execution Extensibility Points
21. Source Query Parser Logical Plan Optimizer Physical Plan Execution SELET t.cf1.name as name, SUM(t.cf1.sales) as total_sales FROM m7://cluster1/sales t GROUP BY name ORDER BY total_sales desc LIMIT 10;
22. Source Query Parser Logical Plan Optimizer Physical Plan Execution sequence: [ { op: scan, storageengine: m7, selection: {table: sales}} { op: project, projections: [ {ref: name, expr: cf1.name}, {ref: sales, expr: cf1.sales}]} { op: segment, ref: by_name, exprs: [name]} { op: collapsingaggregate, target: by_name, carryovers: [name], aggregations: [{ref: total_sales, expr: sum(name)}]} { op: order, ordering: [{order: desc, expr: total_sales}]} { op: store, storageengine: screen} ]
23. Source Query Parser Logical Plan Optimizer Physical Plan Execution { @id: 1, op: m7scan, cluster: def, table: sales, cols: [cf1.name, cf2.name] } { @id: 2, op: hash-random-exchange, input: 1, expr: 1 } { @id: 3, op: sorting-hash-aggregate, input: 2, grouping: 1, aggr:[sum(2)], carry: [1], sort: ~agrr[0] } { @id: 4, op: screen, input: 4 }
24. Apache Hive Apache Drill Data manipulation Read and write Read-only Query language HiveQL + UDF Flexible, incl.SQL2003,MongoQL,DSL +UDF Query execution MapReduce-based Dremel-based Storage layer HDFS,Hbase,MongoDB Flexible, incl.HDFS, HBASE,MongoDB,CouchDB, Mysql, etc Columnar file formats Focus on ORC Build-in, first-class citizen Nested data Throught SerDe Per default, in development Vectorized execution Planned extension, status see HIVE-4160 Via discovery, support for Hive Metastore is planned Schema-level info Hive MetaStore
25. Basic Demo data source: donuts.json logical plan: simple_plan.json result: out.json
26. • API – DrillClient -Encapsulates endpoint discovery -Supports logical and physical plan submission, query cancellation, query status -Supports streaming return results • JDBC driver, converting JDBC into DrillClient communication • REST proxy for DrillClient User Interfaces
27. Execution Plan •Break physical plan into fragments •Determine quantity of parallelization for each task based on estimated costs •Assign particular nodes based on affinity, load and topology
28. 第三方工程 •Jackson for JSON SerDe for metadata •Typesafe HOCON for configuration and module management •Netty4 as core RPC engine, protobuf for communication •Vanilla java, larray and Netty ByteBuf for off-heap large data structures •Hazelcast for distributed cache •Netflix curator on top of zookeeper for service registry •Optiq for SQL parsing and cost optimization •Parquet(http://parquet.io) as native columnar format •Janino for expression compilation •ASM for ByteCode manipulation •Yammer Metrics for metrics •Guava extensively •Carrot HPC for primitive collcetions