Apache Spark—the light at the end of the tunnel?

Apache Spark—the light at the end of the tunnel?

Talk at 6th Data Science Day in Berlin, 2014-05-08

5c3807aaaf0ffefe6c75e3dbbb8588b5?s=128

Michael Hausenblas

May 08, 2014
Tweet

Transcript

  1. Apache Spark—the light at the end of the tunnel? Michael

    Hausenblas, Chief Data Engineer EMEA, MapR Technologies
 6th Data Science Day, Berlin, 2014-05-07
  2. Apache Spark • Originally developed in 2009 in UC Berkeley’s

    AMP Lab • Fully open sourced in 2010 • Top-level Apache Project as of 2014 https://spark.apache.org/
  3. The Spark Community

  4. Most Active Open Source Project in Big Data Giraph! Storm!

    Tez! 0! 20! 40! 60! 80! 100! 120! 140! Project contributors in past year
  5. Easy & Fast Big Data Easy to Develop • Rich

    APIs in Java, Scala, Python • Interactive shell Fast to Run • General execution graphs • In-memory storage
  6. Easy & Fast Big Data Easy to Develop • Rich

    APIs in Java, Scala, Python • Interactive shell Fast to Run • General execution graphs • In-memory storage 2-5 × less code Up to 10 × faster on disk,
 and 100 × in memory https://amplab.cs.berkeley.edu/benchmark/
  7. Easy: Get Started Immediately • Multi-language support • Interactive Shell

    Python lines = sc.textFile(...)
 lines.filter(lambda s: “ERROR” in s).count() Scala val lines = sc.textFile(...)
 lines.filter(x => x.contains(“ERROR”)).count() Java JavaRDD<String> lines = sc.textFile(...);
 lines.filter(new Function<String, Boolean>() {
 Boolean call(String s) {
 return s.contains(“error”);
 }
 }).count();
  8. Easy: Clean API Resilient Distributed Datasets • Collections of objects

    spread across a cluster, stored in RAM or on Disk • Built through parallel transformations • Automatically rebuilt on failure Operations • Transformations
 (e.g. map, filter, groupBy) • Actions
 (e.g. count, collect, save) Write programs in terms of transformations on distributed datasets
  9. Easy: Expressive API map ! ! ! ! ! reduce

  10. Easy: Expressive API map filter groupBy sort union join leftOuterJoin

    rightOuterJoin reduce count fold reduceByKey groupByKey cogroup cross zip sample take first partitionBy mapWith pipe save ...
  11. Easy: Example – Word Count public static class WordCountMapClass extends

    MapReduceBase implements Mapper<LongWritable, Text, Text, IntWritable> { ! private final static IntWritable one = new IntWritable(1); private Text word = new Text(); ! public void map(LongWritable key, Text value, OutputCollector<Text, IntWritable> output, Reporter reporter) throws IOException { String line = value.toString(); StringTokenizer itr = new StringTokenizer(line); while (itr.hasMoreTokens()) { word.set(itr.nextToken()); output.collect(word, one); } } } ! public static class WorkdCountReduce extends MapReduceBase implements Reducer<Text, IntWritable, Text, IntWritable> { ! public void reduce(Text key, Iterator<IntWritable> values, OutputCollector<Text, IntWritable> output, Reporter reporter) throws IOException { int sum = 0; while (values.hasNext()) { sum += values.next().get(); } Hadoop MapReduce val spark = new SparkContext(master, appName, [sparkHome], [jars]) val file = spark.textFile("hdfs://...") val counts = file.flatMap(line => line.split(" ")) .map(word => (word, 1)) .reduceByKey(_ + _) counts.saveAsTextFile("hdfs://...") Spark
  12. Easy: Example – Word Count val spark = new SparkContext(master,

    appName, [sparkHome], [jars]) val file = spark.textFile("hdfs://...") val counts = file.flatMap(line => line.split(" ")) .map(word => (word, 1)) .reduceByKey(_ + _) counts.saveAsTextFile("hdfs://...")
  13. Easy: Works Well With Hadoop Data Compatibility • Access your

    existing Hadoop data (HDFS, HBase, S3, etc.) • Use the same data formats • Adheres to data locality for efficient processing Deployment Models • “Standalone” deployment • YARN-based deployment • Mesos-based deployment • Deploy on existing Hadoop cluster or side-by-side
  14. Example: Logistic Regression data = spark.textFile(...).map(readPoint).cache() ! w = numpy.random.rand(D)

    ! for i in range(iterations): gradient = data .map(lambda p: (1 / (1 + exp(-p.y * w.dot(p.x)))) 
 * p.y * p.x)
 .reduce(lambda x, y: x + y) w -= gradient ! print “Final w: %s” % w
  15. Fast: Logistic Regression Performance Running Time (s) 0 1000 2000

    3000 4000 Number of Iterations 1 5 10 20 30 Hadoop Spark 110  s  /  iteration first  iteration  80  s further  iterations  1  s
  16. Fast: Using RAM, Operator Graphs In-memory Caching • Data Partitions

    read from RAM instead of disk Operator Graphs • Scheduling Optimizations • Fault Tolerance =  cached  partition =  RDD join filter groupBy Stage  3 Stage  1 Stage  2 A: B: C: D: E: F: map
  17. Easy: Fault Recovery RDDs track lineage information that can be

    used to efficiently recompute lost data msgs = textFile.filter(lambda s: s.startsWith(“ERROR”)) .map(lambda s: s.split(“\t”)[2]) HDFS File Filtered RDD Mapped RDD filter
 (func  =  startsWith(…)) map
 (func  =  split(...))
  18. Data platform Execution environment Spark core engine Spark ecosystem 4

    3 2 1 The Spark Stack from 100,000 ft
  19. Easy: Unified Platform Spark SQL (SQL) Spark Streaming (Streaming) MLlib

    (Machine learning) Spark (General execution engine) GraphX (Graph computation) Continued innovation bringing new functionality, e.g.: • BlinkDB (Approximate Queries) • SparkR (R wrapper for Spark) • Tachyon (off-heap RDD caching)
  20. Spark SQL (SQL) Spark Streaming (Streaming) MLlib (Machine learning) Spark

    (General execution engine) GraphX (Graph computation)
  21. Hive Compatibility • Interfaces to access data and code in

    the Hive ecosystem: o Support for writing queries in HQL o Catalog for that interfaces with the 
 Hive MetaStore o Tablescan operator that uses Hive SerDes o Wrappers for Hive UDFs, UDAFs, UDTFs
  22. Parquet Support Native support for reading data stored in Parquet:

    • Columnar storage avoids reading unneeded data. • Currently only supports flat structures (nested data on short-term roadmap). • RDDs can be written to parquet files, preserving the schema. http://parquet.io/
  23. Mixing SQL and Machine Learning val trainingDataTable = sql(""" SELECT

    e.action, u.age, u.latitude, u.logitude FROM Users u JOIN Events e ON u.userId = e.userId""")// Since `sql` returns an RDD, the results of can be easily used in MLlib
 val trainingData = trainingDataTable.map { row =>
 val features = Array[Double](row(1), row(2), row(3))
 LabeledPoint(row(0), features)
 } val model = new LogisticRegressionWithSGD().run(trainingData)
  24. Relationship to Borrows • Hive data loading code / in-

    memory columnar representation • hardened spark execution engine Adds • RDD-aware optimizer/ query planner • execution engine • language interfaces. Catalyst/SparkSQL is a nearly from scratch rewrite that leverages the best parts of Shark
  25. Shark (SQL) Spark Streaming (Streaming) MLlib (Machine learning) Spark (General

    execution engine) GraphX (Graph computation)
  26. Spark Streaming Run a streaming computation as a series of

    very small, deterministic batch jobs Spark Spark Streaming batches  of  X   seconds live  data  stream processed   results • Chop up the live stream into batches of ½ second or more, leverage RDDs for micro-batch processing • Use the same familiar Spark APIs to process streams • Combine your batch and online processing in a single system • Guarantee exactly-once semantics
  27. DStream  of  data Window-based Transformations val  tweets  =  ssc.twitterStream()  

    val  hashTags  =  tweets.flatMap(status  =>  getTags(status))   val  tagCounts  =  hashTags.window(Minutes(1),  Seconds(5)).countByValue() sliding  window   operation window  length sliding  interval window  length sliding  interval
  28. Shark (SQL) Spark Streaming (Streaming) MLlib (Machine learning) Spark (General

    execution engine) GraphX (Graph computation)
  29. MLlib – Machine Learning library Logis]c*Regression,*Linear*SVM*(+L1,*L2),*Decision* Trees,*Naive*Bayes" Linear*Regression*(+Lasso,*Ridge)* Alterna]ng*Least*Squares* KZMeans,*SVD*

    SGD,*Parallel*Gradient* Scala,*Java,*PySpark*(0.9) MLlib Classifica.on:" Regression:" Collabora.ve"Filtering:" Clustering"/"Explora.on:" Op.miza.on"Primi.ves:" Interopera.lity:"
  30. Shark (SQL) Spark Streaming (Streaming) MLlib (Machine learning) Spark (General

    execution engine) GraphX (Graph computation)
  31. Enabling users to easily and efficiently express the entire graph

    analytics pipeline New API Blurs the distinction between Tables and Graphs New System Combines Data-Parallel Graph-Parallel Systems The GraphX Unified Approach
  32. Use Cases

  33. Interactive Exploratory Analytics • Leverage Spark’s in-memory caching and efficient

    execution to explore large distributed datasets • Use Spark’s APIs to explore any kind of data (structured, unstructured, semi-structured, etc.) and combine programming models • Execute arbitrary code using a fully-functional interactive programming environment • Connect external tools via SQL Drivers
  34. Machine Learning • Improve performance of iterative algorithms by caching

    frequently accessed datasets • Develop programs that are easy to reason using a fully-capable functional programming style • Refine algorithms using the interactive REPL • Use carefully-curated algorithms out-of-the-box with MLlib
  35. Power Real-time Dashboards • Use Spark Streaming to perform low-latency

    window-based aggregations • Combine offline models with streaming data for online clustering and classification within the dashboard • Use Spark’s core APIs and/or Spark SQL to give users large-scale, low-latency drill-down capabilities in exploring dashboard data
  36. Faster ETL • Leverage Spark’s optimized scheduling for more efficient

    I/O on large datasets, and in-memory processing for aggregations, shuffles, and more • Use Spark SQL to perform ETL using a familiar SQL interface • Easily port PIG scripts to Spark’s API • Run existing HIVE queries directly on Spark SQL or Shark
  37. http://spark-stack.org

  38. Spark and the Lambda Architecture

  39. Fault tolerance hardware software developer

  40. Human fault tolerance

  41. NEW DATA 
 STREAM QUERY BATCH VIEWS √ View 1

    View 2 View N REAL-TIME VIEWS BATCH LAYER SERVING LAYER SPEED LAYER MERGE IMMUTABLE MASTER DATA PRECOMPUTE VIEWS BATCH RECOMPUTE PROCESS STREAM INCREMENT VIEWS REAL-TIME INCREMENT View 1 View 2 View N Lambda Architecture
  42. Spark hackathon?

  43. Q & A @mhausenblas maprtech mhausenblas@mapr.com MapR maprtech mapr-technologies