Spark Machine Learning 101 @HadoopCon

Transcript

1. Spark Machine Learning 101 Chu-Yu Hsu @ HadoopCon 2015
2. About Me Chu-Yu Hsu, 許儲⽻羽 • Software Engineer • Machine Learning Practicer • Used Spark ML and Python in daily work and Kaggle competition • http://blog.chuyuhsu.ml
3. Outline • Introduction to Spark ML • Alternative Least Squares (ALS) • Hands-on example
5. Apache Spark MLlib • To Make practical machine learning easy and scalable • spark.mllib - the primary API • spark.ml - a higher-level API for constructing ML workflows Apache Spark spark.mllib spark.ml
6. What’s in MLlib Utilities Data types Basic statistics Classification and regression SVM Logistic regression Linear regression Naive Bayes Decision trees Ensembles of trees Isotonic regression Collaborative filtering Alternating least squares (ALS) Clustering K-means Gaussian mixture Power iteration clustering Latent Dirichlet allocation Streaming k-means Dimensionality reduction SVD PCA Frequent pattern mining FP-growth Optimization Stochastic gradient descent Limited-memory BFGS https://spark.apache.org/docs/latest/mllib-guide.html
7. ML Workflow can be VERY complex
8. Types of Recommenders • Editorial and hand curated • Simple aggregates • Tailored to individual users
9. Who Uses Recommenders
10. Approaches • Content based method • Item based method • Model based method
11. Collaborative Filtering • One of mostly known “Recommendation Algorithm” • Widely used in E-commerce application • The data size can be enormous • Need to be delivered as soon as possible
12. Collaborative Filtering Main idea: Find set N of other users whose ratings are “similar” to X’s ratings
13. Users Preferences • This is a baby example • Users: > 2M • Items: > 30M • Sparsity: > 2%
14. Low Rank Assumption • Matrix can be reduced to the product of low rank matrixes • That is also understood as “latent factors” • We assume that the low factor can represent the hidden factors we do not know Action Romance Thriller
15. Low Rank Assumption Action Romance Thriller Action Romance Thriller
16. Matrix Factorization
17. • Our goal is to find P and Q such that (Sum of Square Error):
 
 
 • Root Mean Square Error (RMSE)
 
 
 

18. Alternative Least Squares • Because p and q are both unknown, the object function is not convex • If fix one of the unknowns > can be solved as a least squares problem
19. Amazon Reviews Dataset 35 million ratings, 6.6 million users, 2.4 million products
 on 16-node (m3.2xlarge) https://github.com/apache/spark/pull/3720
20. Resources
21. Resources
22. And More Resources • Source code examples
 https://github.com/apache/spark/tree/master/ examples • Apache Spark JIRA
 https://issues.apache.org/jira/browse/spark
23. Dataset • MovieLens Dataset
 http://grouplens.org/datasets/movielens/ • “ratings.dat”
 UserID::MovieID::Rating::Timestamp • “movies.dat”
 MovieID::Title::Genres
24. Conclusion • Spark MLlib grows fast, but still need some time • Spark MLlib is a strong tool, if you use it right • Sharpening ML skills is first priority
25. Q&A Visit me on:
 http://blog.chuyuhsu.ml Github:
 http://github.com/ChuyuHsu Thanks
26. References • https://spark.apache.org/docs/latest/mllib-guide.html • http://www.slideshare.net/jeykottalam/mllib • http://www.slideshare.net/PetrZapletal1/mllib-and-machine-learning-on-spark • https://databricks.com/blog/2014/07/23/scalable-collaborative-filtering-with- spark-mllib.html • https://github.com/apache/spark/pull/3720 • https://www.hakkalabs.co/articles/spark-mllib-making-practical-machine- learning-easy-and-scalable • http://www.slideshare.net/databricks/practical-machine-learning-pipelines- with-mllib