UpSkill Workshop 9: Introduction to Spark/Databricks McLean, VA | March 30th 2017 val words = sc.textFile("poem.txt") .flatMap(line ⇒ line.split("\\s+")) .map(word ⇒ (word.length, word)) // Number of words by length val wordLen: Map[Int, Long] = words.countByKey()
UpSkill Workshop 9: Introduction to Spark/Databricks McLean, VA | March 30th 2017 • Library of Machine Learning • Algorithms of different types: • Classification • Regression • Clustering • Collaborative filtering • … • Implemented using Spark specific features: • Partitioning of the data • Iterative processing performed in memory • Optimized for large volumes of data
UpSkill Workshop 9: Introduction to Spark/Databricks McLean, VA | March 30th 2017 • Machine Learning library built on top of RDDs • Specific data structures • Vector: a list of features, can be dense or sparse • LabeledPoint: a vector + a label • In supervised learning: • Training data: RDD[LabeledPoint] • Test data: RDD[Vector] • No longer recommended → Spark ML
UpSkill Workshop 9: Introduction to Spark/Databricks McLean, VA | March 30th 2017 ● Machine Learning library built on top of DataFrames ➔ New in Spark 1.2 ● API with high-level components: ➔ Estimators: they generate a Model from training data ➔ Transformers: they transform the data ➔ A Model is itself a Transformer ➔ A Machine Learning algorithm is an Estimator
UpSkill Workshop 9: Introduction to Spark/Databricks McLean, VA | March 30th 2017 ● Steps: ➔ On training data, for each stage: if the stage is an Estimator, call its fit() method and the transform() method of the resulting Model; if the stage is a Transformer, call its transform() method ➔ To make predictions on test or new data, call the transform() method of each Transformer (including Models from the training step) ● Tedious → use Pipelines
UpSkill Workshop 9: Introduction to Spark/Databricks McLean, VA | March 30th 2017 ● Spark ML provides an API to build a Pipeline composed of Estimators and Transformers ● Steps: ➔ Assemble the Pipeline ➔ Call Pipeline.fit() on training data to get a PipelineModel: the fit() method is called on each Estimator and the transform() method is called on each Transformer (including Models) ➔ Call PipelineModel.transform() on data to make predictions: the transform() method is called on each component of the Pipeline
UpSkill Workshop 9: Introduction to Spark/Databricks McLean, VA | March 30th 2017 ● Conventions for column names: ➔ The vector of features: “features” ➔ The label in training data: “label” ➔ The predicted label: “prediction” ● These column names are used unless they are explicitly specified
UpSkill Workshop 9: Introduction to Spark/Databricks McLean, VA | March 30th 2017 Spark ML provides support for Grid Search optimization through k-fold Cross Validation • With k=10, the model is trained and evaluated on a first split of 9/10 of the data for training and 1/10 for validation. • The model is then trained on another split, and so on 10 times. The final accuracy is the average of the accuracies measured for each fold
UpSkill Workshop 9: Introduction to Spark/Databricks McLean, VA | March 30th 2017 ● Parameters are defined through a ParamGridBuilder ● Performance is measured using an Evaluator: ➔ RegressionEvaluator: for numerical predictions ➔ BinaryClassificationEvaluator: for binary predictions ➔ MultiClassClassificationEvaluator: for multi-category predictions ● Instantiate a CrossValidator: ➔ Define the number of folds, the Pipeline, the Estimator, the param grid and the Evaluator ➔ Then call fit() to search for the most accurate model
aseigneurin
doradora09
ryohatanmonolog
nihonbuson
_kensh
keithyokoma
takagimeow
eiki
hosimesi11
takaking22
spinscale
hirosatogamo
ryomaru0825
3n
bkeepers
kevinliebholz
reverentgeek
philhawksworth
gr2m
mclloyd
cromwellryan
samanthasiow
jennybc
thekraken
samlambert