Lightning-fast Machine Learning with Spark

@nivdul
#DV14 #MLwithSpark
Lightning fast Machine
Learning with Spark
Ludwine Probst

View Slide

@nivdul
#Devoxx #MLwithSpark
me
Data engineer at
Leader of
Duchess
France

View Slide

@nivdul
Machine Learning

View Slide

@nivdul
#DV14 #MLwithSpark
MapReduce
Lay of the land

View Slide

@nivdul
MapReduce

View Slide

@nivdul
HDFS
with iterative algorithms

View Slide

@nivdul

View Slide

@nivdul
is a fast and general engine for large-scale
data processing

View Slide

@nivdul
#DV14 #MLwithSpark
•big data analytics in memory/disk
•complements Hadoop
•fast and more flexible
•Resilient Distributed Datasets (RDD)
•shared variables

View Slide

@nivdul
Shared variables
broadcast variables
accumulators
val broadcastVar = sc.broadcast(Array(1, 2, 3))
val acc = sc.accumulator(0, "MyAccumulator")
sc.parallelize(Array(1, 2, 3)).foreach(x => acc += x)

View Slide

@nivdul
#DV14 #MLwithSpark
RDD
(Resilient Distributed Datasets)
•process in parallel
•controllable persistence (memory, disk…)
•higher-level operations (transformation & actions)
•rebuilt automatically using lineage

View Slide

@nivdul
Data Storage
InputFormat
cassandra
cassandra

View Slide

@nivdul
Spark data flow

View Slide

@nivdul
Languages
interactive shell (scala & python)
Lambda
(Java 8)

View Slide

@nivdul
val conf = new SparkConf()
.setAppName("Spark word count")
.setMaster("local")
!
val sc = new SparkContext(conf)
WordCount example (scala)

View Slide

@nivdul
#DV14 #MLwithSpark
// load the data
val data = sc.textFile("filepath/wordcount.txt")
// map then reduce step
val wordCounts = data.flatMap(line => line.split("\\s+"))
.map(word => (word, 1))
.reduceByKey(_ + _)
// persist the data
wordCounts.cache()

View Slide

@nivdul
#DV14 #MLwithSpark
// keep words which appear more than 3 times
val filteredWordCount = wordCounts.filter {
case (key, value) => value > 2
}
!
filteredWordCount.count()

View Slide

@nivdul
Spark ecosystem

View Slide

@nivdul
streaming
makes it easy to build scalable fault-tolerant streaming applications

View Slide

@nivdul
SQL
unifies access to structured data

View Slide

@nivdul
is Apache Spark's API for graphs
and
graph-parallel computation

View Slide

@nivdul
MLlib
is Apache Spark's scalable machine learning library

View Slide

@nivdul
Machine learning
with
Spark / MLlib

View Slide

@nivdul
Machine learning libraries
scikit

View Slide

@nivdul
Example
make a movies recommender system

View Slide

@nivdul
Collaborative filtering
with Alternating Least Square (ALS)

View Slide

@nivdul
#DV14 #MLwithSpark
1 3 5
1 28 4
2 18 3
2 5 5
userID movieID rating

View Slide

@nivdul
#DV14 #MLwithSpark
// Load and parse the data
val data = sc.textFile("movies.txt")
!
// create a RDD[Rating]
val ratings = data.map(_.split("\\s+") match {
case Array(user, movie, rate) =>
Rating(user.toInt, movie.toInt, rate.toDouble)
})

View Slide

@nivdul
#DV14 #MLwithSpark
// split the data into training set and test set
val splits = ratings.randomSplit(Array(0.8, 0.2))
!
// persist the training set
val training = splits(0).cache()
val test = splits(1)

View Slide

@nivdul
#DV14 #MLwithSpark
// Build the recommendation model using ALS
!
val model = ALS.train(training, rank = 10,
iterations = 20, 1)

View Slide

@nivdul
#DV14 #MLwithSpark
// Evaluate the model
val userMovies = test.map {
case Rating(user, movie, rate) => (user, movie)
}
val predictions = model.predict(userMovies).map {
case Rating(user, movie, rate) => ((user, movie), rate)
}
!
val ratesAndPreds = test.map {
case Rating(user, movie, rate) => ((user, movie), rate)
}.join(predictions)
//measuring the Mean Squared Error of rating prediction
val MSE = ratesAndPreds.map { case ((user, movie), (r1, r2)) =>
val err = (r1 - r2)
err * err
}.mean()

View Slide

@nivdul
#DV14 #MLwithSpark
// recommending movies
!
val recommendations = model.recommendProducts(2, 10)
.sortBy(- _.rating)
!
var i = 1
recommendations.foreach { r =>
println(r.product + " with rating " + r.rating)
i += 1
}

View Slide

@nivdul
Performance Spark core
Hadoop MapReduce
http://databricks.com/blog/2014/10/10/spark-breaks-previous-large-scale-sort-record.html
How fast a system can sort 100 TB of data on disk ?

View Slide

@nivdul
Performance Spark / MLlib
Collaborative filtering with MLlib vs Mahout
https://databricks.com/blog/2014/07/23/scalable-collaborative-filtering-with-spark-mllib.html

View Slide

@nivdul
Why should I care ?
fast and easy Machine Learning
with MLlib
fast & flexible
in-memory /on-disk SQL
Streaming
MLlib

View Slide

View Slide

Lightning-fast Machine Learning with Spark

Lightning-fast Machine Learning with Spark

Probst Ludwine

More Decks by Probst Ludwine

Other Decks in Programming

Featured

Transcript