! ! ! ! ! ! “Simple things simple, complex things possible.” ! ! - Alan Kay 

Apache Spark: Unified Platform for Big Data Reynold Xin, Databricks 

Apache Spark: Unified Platform for Big Data Reynold Xin, Databricks /\ / \ | | | | | | | | / \ -————- The tower at Berkeley campus 

Founded last year by creators of Apache Spark ! ! Building a cloud platform to drastically simplify Big Data 

! ! ! ! ! ! “Simple things simple, complex things possible.” ! ! - Alan Kay 

Big Data Space Today 

Big Data Space Today ! ! ! Zoo of tools to learn, deploy, connect, and maintain. ! ! ! ! ! “Simple things complex, complex things impossible!” ! ! 

– Alan Kay “Simple things simple, complex things possible.” 

Looking for the “lingua franca” platform for ! Big Data 

Why a Platform Matters Good for developers: one system to learn Good for users: take apps anywhere Good for distributors: more applications 

Apache Spark “The Lingua franca for Big Data” — Eric Baldeschwieler 

Apache Spark “The Lingua franca for Big Data” — Eric Baldeschwieler ! 1. Unified processing platform 2. Rich standard libraries 3. Availability 4. Scalability 

1. Unified Processing Platform for Big Data Batch Interactive Streaming Hadoop Cassandra Mesos … … Cloud Providers … Uniform API for diverse workloads over diverse storage systems and runtimes 

2. Standard Library for Big Data Big Data apps lack libraries
 of common algorithms ! Spark’s generality + support
 for multiple languages make it
 suitable to offer this Core SQL ML graph … Python Scala Java 

Spark MLlib One of the most actively developed library. • classification: logistic regression, linear support vector machine (SVM), naive Bayes, classification tree • regression: generalized linear models (GLMs), regression tree • collaborative filtering: alternating least squares (ALS) • clustering: k-means • decomposition: singular value decomposition (SVD), principal component analysis (PCA) • statistics: summary statistics • evaluation: binary classification • optimization: gradient descent, L-BFGS 

Spark MLlib 1.1 (Aug/Sep 2014) improvements • standardized interfaces: dataset, algorithm, params, and model • multi-model training • multiclass support for classification tree • Java/Python APIs for decision trees • SVD via Lanczos • standardized text format for training data new • statistics: stratified sampling, linear/ rank correlation, hypothesis testing • non-negative matrix factorization (NMF) • preprocessing: tf-idf • evaluation: multiclass metrics • online model updates with streaming • and your contribution! 

3. Available Everywhere All major Hadoop distributions include Spark ! ! 
 Beyond Hadoop 

4. Fault-tolerance & Scalability Cluster size: in production clusters with 1000+ nodes Data size: processes data many times size of memory
 - petabyte sort on lots of machines
 - or handle tons of data on a few machines (sorting 5TB compressed/node; file system breaking before Spark did) 

A Simple Demo 

Spark is the most active open-source Big Data project by - active contributors (300+) - commits - line of code changes 

– Alan Kay “Simple things simple, complex things possible.” Let’s work towards 

And or course, we are hiring ! [email protected]