How It Works - Spark

A series of talks on data engineering

Transcript

1. HOW IT WORKS: HOW IT WORKS: SPARK SPARK 1
2. PLAN PLAN Hadoop weakpoints Spark core ideas & concepts Applications & Ecosystem Demo 2 . 1
3. RECAP: HADOOP & MAPREDUCE RECAP: HADOOP & MAPREDUCE 3 . 1
4. PROBLEM: HADOOP WEAKPOINTS PROBLEM: HADOOP WEAKPOINTS slow intermediate results are saved to disk complex imperative style, too verbose APIs, not- available to regular humans 4 . 1
5. IDEA IDEA lets keep all data being processed in memory lets treat whole dataset simply as a collection lets build functional API for processing 5 . 1
6. SPARK CORE CONCEPTS SPARK CORE CONCEPTS 6 . 1
7. RDD RDD Resilient Distributed Dataset 6 . 2
8. 6 . 3
9. 6 . 4
10. RDD FEATURES RDD FEATURES immutable lazy partitioned, location-aware & location- transparancy persistence distributed, scalable in-memory fault-tolerant, lineage: child knows its parents functional api: declarative, typed 6 . 5
11. DAG DAG Directed Acyclic Graph 6 . 6
12. 6 . 7
13. 6 . 8
14. 6 . 9
15. EXECUTION MODEL EXECUTION MODEL 6 . 10
16. 6 . 11
17. DEPLOYMENT DEPLOYMENT 6 . 12
18. 6 . 13
19. API API 6 . 14
20. 6 . 15
21. COMPONENTS COMPONENTS 6 . 16
22. 6 . 17
23. SPARK SQL & DATAFRAME SPARK SQL & DATAFRAME 7 . 1
24. 7 . 2
25. 7 . 3
26. SQL api, functional api, typed/untyped interactive, analytical interface, uni ed programming model distributed, scalable code generation, out-of-the-box optimizations = catalyst engine memory & binary & compute optimizations = tungsten engine integration: multiple datasources, single representation, hive metastore 7 . 4
27. 7 . 5
28. 7 . 6
29. ECOSYSTEM & USECASES ECOSYSTEM & USECASES 8 . 1
30. 8 . 2
31. DEMO DEMO spark-shell text le (rdd) load into memory lter, map, group by reduce save show ui show plan, explain caching rdd -> dataframe 9 . 1
32. PLACE OF SPARK IN BIGDATA ECOSYSTEM PLACE OF SPARK IN BIGDATA ECOSYSTEM 10 . 1
33. 10 . 2
35. 10 . 3
36. CALL TO ACTION CALL TO ACTION High Performance Spark - Holden Karau install spark, run spark-shell, load text le, play with it http://learn.mapr.com/dev-360-apache-spark- essentials 11 . 1
37. 12 . 1