Fast interactive distributed data analysis with Spark

Transcript

1. Fast interactive distributed data analysis with Spark
2. HI! !   I`m Grega. !   I work at Celtra. !   I tweet at @gregakespret.
3. I work with data !   5000 new ads per month !   1 billion impressions per month !   100G new data per day !   ... and growing
4. Spark !   Open source cluster data processing system !   Up to 100x faster (for iterative algorithms) than Hadoop MapReduce !   Can read from any Hadoop input source (e.g. HDFS, S3, …) !   APIs in Scala, Java and Python (since 0.7.0)
5. Spark (contd.) !   Basic idea: Work with distributed collections (almost) as you would with local ones !   Can cache datasets in-memory across cluster
6. Why we like Spark? !   High-level API in Scala (data flows written as closures) !   Lean codebase (LOC) !   Active development !   Low latency scheduling jobs (Akka actors) !   Interactive shell (spark-shell) ! Scala ecosystem
7. Counting word occurrences val sc = new SparkContext(master, appName,
 [sparkHome], [jars]) val lines = sc.textFile("hdfs://...")
 
 val counts = lines.flatMap(line => line.split(" "))
 .map(word => (word, 1))
 .reduceByKey(_ + _)
 
 counts.saveAsTextFile("hdfs://...")
8. MapReduce model !   Break computing into managable parts (map, reduce) !   Can be parallelized !   Fault-tolerance
9. Resilient distributed datasets !   RDDs maintain lineage information that can be used to reconstruct lost partitions !   Immutable, distributed collections of objects !   Fault-tolerant !   Created through parallel transformations (map, filter, groupBy, join, …) on data in stable storage !   Can be cached in memory for fast reuse
10. RDD operations " !   Initial file / collection -> transform1 -> transform2 -> … -> action !   Transformations: lazy operations to build RDDs from other RDDs (map, filter, join, …) !   Actions: return a result on driver or write it to some storage (count, reduce, collect, save)
11. Tasks !   Number of slices to cut dataset into !   Optional parameter for each Shuffle operation (reduceByKey, sortByKey, groupByKey) !   Typically you want 2-4 slices / CPU in your cluster
12. Shared variables !   Normally, all variables referenced in a closure of Spark operation are copied to each machine with each task !   Broadcast variables allow you to cache a large read-only variable on each worker (to avoid network and I/O) !   Accumulators are variables that can be efficiently added to in parallel (e.g. counters)
13. THAT’S IT We are hiring J Questions?