Shark: a better adhoc query engine faster than hive

Transcript

1. Shark: a better adhoc query engine faster than hive ——Zhu Guangbin
2. Agenda What is Shark? What does Shark do? How does Shark perform? further work
3. What is Shark? A data analysis (warehouse) system that builds on Spark (MapReduce deterministic, idempotent tasks), scales out and is fault-tolerant, supports low-latency, interactive queries through in-memory computation, supports both SQL and complex analytics such as machine learning, is compatible with Apache Hive (storage, serdes, UDFs, types, metadata).
4. Shark is Hive on Spark
5. What does Shark do? adhoc and interactive query at least 10x faster than hive combine HQL and complex ML analysis compatible with HQL, full supported hive udf an alternitive of Hive for adhoc query
6. most queries are simple, but using hive is too expensive our query on hiveweb
7. Why Hive is so slow? Disk-based intermediate outputs Inferior data format and layout (no control of data co-partitioning) Execution strategies (lack of optimization based on data statistics) Task scheduling and launch overhead!
8. MapReduce Model shuffle is expensive 1. mapout pesistent (disk io) 2. reduceside fetch mapout (network io )
9. MapReduce Model
10. Why Shark is faster ? In-memory computing cache intermediate data in memory rather than persistent on disk Columnar Memory Store Compact storage JVM garbage collection friendly CPU-efficient compression
11. Spark Model
12. How does DPer use shark? we have integrate Shark to HiveWeb just select shark as query engine write hql like hive and then submit
13. DEMO TIME
14. How does Shark perform? Cluster comparison: Hive/MapReduce Cluster： 37 Node, 436 MapSlot & 436 ReduceSlot total, 1 slot per task Shark/Spark Cluster： 3 node, 48 core & 48G mem total, 12 core max & 4G mem per node for each client
15. Test 1 SQL: our benchmark sql, use hive testcase data Test Result see more on my blog
16. Test2 SQL: we sampled 7 hql from HiveWeb history Test Result: see more on googlesheet
17. Conclusion Shark is full compatible with Hive Shark is faster every has less nodes
18. Our Work on Shark deploy online bugfix for authorization integration with hiveweb performance tuning
19. Further work Scale out Spark Cluster Bugfix for Hive concurrency & kerberos issues migrate adhoc query to Shark
20. Thanks Q&A see more on myblog: zhuguangbin.github.io