Building Scalable Data Science Pipeline

"In theory, there is no difference between theory and practice. But in practice, there is." - Yogi Berra

Once the task of prototyping a data science solution has been accomplished on a local machine, the real challenge begins in how to make it work in production. To ensure that the plumbing of the data pipeline will work in production at scale is both an art and a science. The science involves understanding the different tools and technologies needed to make the data pipeline connect, while the art involves making the trade-offs needed to tune the data pipeline so that it flows.

In this workshop, you will learn how to build a scalable data science platform with set up and conduct data engineering using Pandas and Luigi, build a machine learning model with Apache Spark and deploy it as predictive api with Flask

Transcript

1. Building Scalable Data Science Pipeline luigi | spark | flask www.unnati.xyz Raghotham S Nischal HP
2. Agenda • Introduction • Data engineering • Machine Learning • Data Pipelines • API • Hands on
3. Introduction Applying software engineering principles to Data Science
4. Data Engineering Process of acquiring, cleaning, transforming & persisting data
5. Machine Learning Art & science of choosing a model & scaling it
6. Data Pipelines Plumbing data engineering & machine learning tasks
7. API Expose data science as a service
8. Project Structure
9. Hands on Dataset: Bay Area Bike Share Hypothesis based solution
10. Apache Spark • Distributed in-memory computing • Distributed machine learning framework • 100x faster than Hadoop • RDDs
11. Luigi • Complex pipelines • Dependency resolution • Workflow management • Visualization • Exception handling
12. What did we learn today?
13. What did we learn today? Building scalable data science platform is easy
14. What did we learn today? Building scalable data science platform is easy
15. Thank You @unnati_xyz