Scalable Machine Learning with Apache Spark

The talk demonstrates an end to end design (architecture, implementation and deployment) of Downpour-like stochastic gradient descent using Apache Spark. Spark is the next generation cluster computing framework from the UC Berkeley and Databricks teams.

Transcript

1. Scalable  ML   with  Apache   Spark   Saket Bhushan @5aket
2. Contents • Basic LR with Theano • Spark 101 • ML with Spark • ML Model in Production
5. Basic  Maths   y1 x1 x2 x3 w12 w21 w31 Y  =  Activation(WiXi  +  b)  
8. Theano   X Y -­‐‑ Z Compilation substr(_,  _) 20 5 15 Symbolic  Variables
9. Theano   X -­‐‑ x1 Shared  Variables + 1 2 x2
10. GPU  vs  CPU hSp://deeplearning.net/software/theano/tutorial/using_gpu.html
11. Show  me  the  Code!
12. Apache  Spark
13. Spark  Ecosystem
14. Spark  Performance Source  :  hSps://databricks.com/blog/2014/10/10/spark-­‐‑petabyte-­‐‑sort.html
15. Spark  Architecture
16. Spark  Pipelines •  Dataframe – Data Frame from Spark SQL •  Transformer – Creates prediction Data Frame from Feature Data Frames •  Estimator – A learning algo is an example of estimator •  Pipeline – Collection of Transformers & Estimators.
17. Show  me  some  Code!
18.  ML  Pipeline Data Train   Classifier Predict ?
19. Actual  Pipeline •  Data o  Ingestion o  Cleaning o  Storing o  Shuffling o  Feature Extraction •  Model in Development o  Chose a model o  Train on the Data o  Tune Parameters, Validate o  Deploy •  Model in Production o  Evaluation o  Monitoring o  Logging o  Versioning o  Reports, Charts, Dashboards
20. Questions?