Apache Spark

Talk I gave at Hyderabad Software Architects meetup on 20-Jan-2018

Transcript

1. Apache Spark Presenter: MH http://bit.ly/mahboob
2. Spark Overview • General purpose cluster computing system • High-level APIs in Java, Scala, Python and R • Supports general execution graphs • Spark SQL • MLlib • GraphX • Spark Streaming
3. Deployment Modes • Amazon EC2: ◦ scripts that let you launch a cluster on EC2 in about 5 minutes • Standalone Deploy Mode: ◦ launch a standalone cluster quickly without a third-party cluster manager • Mesos: ◦ deploy a private cluster using Apache Mesos • YARN: ◦ deploy Spark on top of Hadoop NextGen (YARN)
4. FAQ Hadoop vs Spark File system based | Memory based Map-Reduce Paradigm | Any distributed computing workload Details: https://docs.google.com/document/d/1hcv3JOc009AVer6bVFEeGnHrt_Jb0v3v3g2qGPgRy1Y
5. The Spark Driver Source: http://mapr.com
6. Very good resource https://mapr.com/ebooks/spark/ Downloadable as pdf
7. K-Means Clustering First, decide the number of clusters k. Then: 1. Initialize the center of the clusters 2. Attribute the closest cluster to each data point 3. Set the position of each cluster to the mean of all data points belonging to that cluster 4. Repeat steps 2-3 until convergence Create k points for starting centroids (often randomly) While any point has changed cluster assignment for every point in the dataset: for every centroid calculate distance between centroid and point assign point to the cluster with the lowest distance for every cluster calculate mean of points in that cluster assign the centroid to the mean
8. Example 1 Applying k-Means Algorithm using Python Program: kMeans.py* Data: x-y coordinates in file testSet.txt * Source: Chapter 10, Machine Learning in Action
9. Example 2 Applying k-Means Algorithm using Java API to Spark Program: Spark_KMeans.java Data: Random numbers in marks.txt
10. Example 3 https://mapr.com/ebooks/spark/08-machine-learning-mllib-spark-use-case.html Techniques applied for NLP • Hashing • Stemming • TF-IDF • Naive Bayes • Random Forest Regression • Gradient Boosted Trees Regression • Code repository: https://github.com/joebluems/Mockingbird/issues Is Harper Lee’s Go Set a Watchman a discarded rough draft that was to become the universally beloved classic To Kill A Mockingbird, or was it a truly separate work?
11. Who is using Spark? Source: https://medium.com/ai rbnb- engineering/data- infrastructure-at- airbnb-8adfb34f169c
12. Spark @ AirBnB Spark is also an important tool for Airbnb. The team actually built something called Airstream, which is a computational framework that sits on top of Spark Streaming and Spark SQL, allowing engineers and the data team to get quick insights. Ultimately, for an organization that depends on predictions and machine learning, something like Spark - alongside other open source machine learning libraries - is crucial in the Airbnb stack. Source: https://www.packtpub.com/books/content/what-software-stack-does-airbnb-use
13. Thank You