Apache Spark & MLlib

Apache Spark is an emerging cluster computing platform that allows data processing programs to run up to 100x faster than Hadoop MapReduce in memory, or 10x faster on disk. Spark also has a built in machine learning library, MLlib, that implements many common supervised and unsupervised machine learning algorithms. In this talk we will discuss how Spark has improved cluster computing and data processing along with an overview of the MLlib algorithms available. After getting familiar with the basics, we will explore how you can create a product recommendation engine for eCommerce utilizing Collaborative Filtering and the Alternating Least Squares algorithm.

Transcript

1. & APACHE SPARK MLlib APACHE SPARK MLlib
2. 2 PARTS: APACHE SPARK MLlib
3. 2 PARTS: APACHE SPARK what is
4. 2 PARTS: MLlib how do I use on spark?
5. PART I APACHE SPARK what is
6. DATA
7. DATA IS EVERYWHERE
8. DATA IS UNLIMITED
9. DATA IS BIG
10. BIG DATA IS HARD and
11. CHALLENGES WITH BIG DATA 1. Capturing
12. CHALLENGES WITH BIG DATA 1. Capturing 2. Storing
13. CHALLENGES WITH BIG DATA 1. Capturing 2. Storing 3. Analyzing
14. REAL WORLD EXAMPLE
15. 1 TWEET @addamh: Tacos are the best . Tom Brady is the best. So, Tom Brady is . @addamh: Tacos are the best . Tom Brady is the best. So, Tom Brady is .
16. 6000 TWEETS / SECOND X 12 MB / SECOND = ~ 200 BYTES / TWEET X 1 TWEET @addamh: Tacos are the best . Tom Brady is the best. So, Tom Brady is .
17. 6000 TWEETS / SECOND X 12 MB / SECOND = ~ 200 BYTES / TWEET X 24 HOURS X 1TB / DAY = 1 TWEET @addamh: Tacos are the best . Tom Brady is the best. So, Tom Brady is . @addamh: Tacos are the best . Tom Brady is the best. So, Tom Brady is .
18. 6000 TWEETS / SECOND X 12 MB / SECOND = ~ 200 BYTES / TWEET X 24 HOURS X 1TB / DAY = 30 DAYS / MONTH X 30TB / MONTH = 1 TWEET @addamh: Tacos are the best . Tom Brady is the best. So, Tom Brady is . @addamh: Tacos are the best . Tom Brady is the best. So, Tom Brady is .
19. …and that’s just Twitter…on an average day.
20. BIG DATA IS NOT BULLSHIT
21. CHALLENGES WITH BIG DATA 1. Capturing 2. Storing 3. Analyzing
22. CHALLENGES WITH BIG DATA 1. Capturing 2. Storing 3. Analyzing
24. DISTRIBUTED COMPUTING PLATFORM
25. IS NOT A BETTER HADOOP.
26. WORKS IN MEMORY. NOT ON DISK.
27. USES RESILIENT DISTRIBUTED DATASETS.
28. WHAT IS AN RDD?
29. DATA ARRAY RDD RDD Partition Host 02 RAM partition RAM partition RAM partition RAM partition Host 01 RAM partition RAM partition RAM partition RAM partition Host n RAM partition RAM partition RAM partition RAM partition RDD Partition RDD Partition RDD Partition RDD Partition
30. MACHINE LEARNING ON SPARK?
31. MACHINE LEARNING ON SPARK? MLlib
32. WHY MLlib?
33. BUILT ON SPARK. FOR SPARK.
34. BUILT ON SPARK. FOR SPARK. ML ALGORITHMS WITH 100X SPEED INCREASES OVER MAP REDUCE
35. PART II MLlib how do I use on spark?
36. MLlib is a machine learning toolbox that is tightly integrated into Spark and has an RDD API. This allows MLlib’s algorithms to run on a distributed Spark cluster.
37. MLlib Classification: Logistic Regression Naive Bayes Decision Trees: Random Forests Gradient-Boost Trees Clustering: K-means Topic Modeling: latent Dirichlet allocation Recommendation: Alternating Least Squares Regression: Linear Regression Isotonic Regression is a machine learning toolbox that is tightly integrated into Spark and has an RDD API. This allows MLlib’s algorithms to run on a distributed Spark cluster.
38. MLlib Classification: Logistic Regression Naive Bayes Decision Trees: Random Forests Gradient-Boost Trees Clustering: K-means Topic Modeling: latent Dirichlet allocation Recommendation: Alternating Least Squares Regression: Linear Regression Isotonic Regression is a machine learning toolbox that is tightly integrated into Spark and has an RDD API. This allows MLlib’s algorithms to run on a distributed Spark cluster.
39. COLLABORATIVE FILTERING
44. ?
46. ? ? ?
47. ? ? ? 2 3 5 2 4 3 2 4 5 5 1 4 5
48. 3 5 ? 5 2 5 1 4 ? 4 2 3 5 2 ? 4 ω = Our data as a dense matrix:
50. Minimization Problem Non-Convex Problem
51. WHERE IS THE GLOBAL MINIMA? Non-Convex
52. WHERE IS THE GLOBAL MINIMA? Not easy to find Non-Convex
53. LOW-RANK MATRIX FACTORIZATION IS A NON-CONVEX PROBLEM. THIS MAKES FINDING THE GLOBAL MINIMA VERY COSTLY. THE ALTERNATING LEAST SQUARES ALGORITHM ALLOWS US CONVERT TO A CONVEX PROBLEM.
54. WHERE IS THE GLOBAL MINIMA? Much Easier Convex
56. IT’S NOT THIS HARD I PROMISE
57. MLlib HAS ALREADY DONE THE HARD WORK.
58. LET’S LOOK AT SOME CODE
59. COLLECT YOUR DATA
60. Movielens Data movieId,title,genres 1,Toy Story (1995),Adventure|Animation|Children|Comedy|Fantasy 2,Jumanji (1995),Adventure|Children|Fantasy 3,Grumpier Old Men (1995),Comedy|Romance 4,Waiting to Exhale (1995),Comedy|Drama|Romance 5,Father of the Bride Part II (1995),Comedy 6,Heat (1995),Action|Crime|Thriller 7,Sabrina (1995),Comedy|Romance 8,Tom and Huck (1995),Adventure|Children 9,Sudden Death (1995),Action 10,GoldenEye (1995),Action|Adventure|Thriller 11,"American President, The (1995)",Comedy|Drama|Romance 12,Dracula: Dead and Loving It (1995),Comedy|Horror 13,Balto (1995),Adventure|Animation|Children
61. Ratings userId,movieId,rating,timestamp 1,31,2.5,1260759144 1,1029,3.0,1260759179 1,1061,3.0,1260759182 1,1129,2.0,1260759185 1,1172,4.0,1260759205 1,1263,2.0,1260759151 1,1287,2.0,1260759187 1,1293,2.0,1260759148 1,1339,3.5,1260759125
62. PULL IN MLlib
64. SPLIT OUR DATA
66. TRAIN THE MODEL AND TUNE THE PARAMETERS
68. Train multiple models to find best rank parameter
69. Train multiple models to find best rank parameter Compute Root-Mean-Square Error
70. Train multiple models to find best rank parameter Compute Root-Mean-Square Error Lowest wins
71. INPUT YOUR NEW RATINGS
73. (userId, movieId, rating)
74. (userId, movieId, rating) Distribute RDD to the Spark cluster
75. TRAIN A NEW MODEL BASED ON YOUR RATINGS
76. Merge new ratings with full rating set
77. Merge new ratings with full rating set Train an updated model with the new ratings
78. GET PREDICTIONS WITH THE UPDATED MODEL
79. Get IDs of movies just rated
80. Remove just rated movies from ratings RDD Get IDs of movies just rated
81. Remove just rated movies from ratings RDD Make predictions for unrated movies from full dataset Get IDs of movies just rated
82. FIND THE MOVIES YOU DIDN’T KNOW YOU ❤
83. Get top 10 recommendations
84. Get top 10 recommendations Print them out
86. YOU DON’T HAVE TO BE A DATA SCIENTIST TO START GETTING VALUE FROM THE TOOLS.
87. ADDAM HARDY @addamh Come talk to us about your data