Lightning fast Machine Learning with Spark Ludwine Probst @nivdul ScalaIO, Paris 2014-10-23 

me Data engineer at @nivdul Leader of 

Machine Learning 

Lay of the land 

& MapReduce & HDFS 

MapReduce 

HDFS with iterative algorithms 

No content

is a fast and general engine for large-scale data processing 

• big data analytics in memory/disk • complements Hadoop • fast and more flexible • Resilient Distributed Datasets (RDD) • shared variables 

Shared variables broadcast variables accumulators val broadcastVar = sc.broadcast(Array(1, 2, 3)) val accum = sc.accumulator(0, "MyAccumulator") sc.parallelize(Array(1, 2, 3)).foreach(x => accum += x) 

RDD • process in parallel • higher-level operations (transformation & actions) • controllable persistence (memory, disk…) • rebuilt automatically using lineage fault-tolerant immutable distributed collections 

Data flow 

InputFormat Data storage 

Languages interactive shell (scala & python) 

Deployment standalone MESOS YARN 

! val conf = new SparkConf() .setAppName("Spark word count") .setMaster("local") ! val sc = new SparkContext(conf) ! val data = sc.textFile("filepath/wordcount.txt") ! ! val wordCounts = data.flatMap(line => line.split("\\s+")) .map(word => (word, 1)) .reduceByKey(_ + _) ! wordCounts.cache() ! val filteredWordCount = wordCounts.filter{ case (key, value) => value > 2 } ! filteredWordCount.count() ! 

Spark ecosystem 

streaming makes it easy to build scalable fault-tolerant streaming applications 

SQL unifies access to structured data 

is Apache Spark's API for graphs and graph-parallel computation 

MLlib is Apache Spark's scalable machine learning library 

Machine learning with Spark/MLlib 

Machine learning libraries scikits 

Classification classify mail into spam or non-spam with a logistic regression 

val prediction = test.map(p => (model.predict(p.features), p.label)) val accuracy = 1.0 * prediction.filter(x => x._1 == x._2) .count() / test.count() training test model input data validation val parsedData = data.map { line => val parts = line.split(",").map(_.toDouble) ! LabeledPoint(parts(0), Vectors.dense(parts.tail)) } val splits = parsedData.randomSplit(Array(0.8, 0.2)) val training = splits(0).cache() val test = splits(1) val model = LogisticRegressionWithSGD.train(training, 100) 

Collaborative filtering make a recommender system with Alternative Least Square (ALS) 

val ratings = data.map(_.split("\\s+") match { case Array(user, item, rate) => Rating(user.toInt, item.toInt, rate.toDouble) }) val splits = ratings.randomSplit(Array(0.8, 0.2)) val training = splits(0).cache() val test = splits(1) val model = ALS.train(training, rank = 10, iterations = 20, 0.01) val userMovies = test.map { case Rating(user, movie, rate) => (user, movie) } val predictions = model.predict(userMovies).map { case Rating(user, movie, rate) => ((user, movie), rate) } training test model input data validation https://github.com/nivdul/spark-ml-scalaio 

Clustering with K-means 

val parsedData = data.map(s => Vectors.dense(s.split("\\s+").map(_.toDouble))) val clusters = KMeans.train(training, k = 4, maxIterations = 20) // Evaluate clustering by computing Within Set Sum of Squared Errors val WSSSE = clusters.computeCost(parsedData) val splits = ratings.randomSplit(Array(0.8, 0.2)) val training = splits(0).cache() val test = splits(1) training test model input data validation https://github.com/nivdul/spark-ml-scalaio 

Performance Spark core MapReduce http://databricks.com/blog/2014/10/10/spark-breaks-previous-large-scale-sort-record.html 

Performance Spark/MLlib Collaborative filtering with MLlib vs Mahout https://databricks.com/blog/2014/07/23/scalable-collaborative-filtering-with-spark-mllib.html 

Why should I care ? 

https://github.com/nivdul/spark-ml-scalaio http://spark.apache.org/ http://spark.apache.org/mllib/