Experimentation with Jupyter, Papermill, and MLFlow

Presented at "One Week Workshop on the Internet of Things (IoT)" under the ATAL Program Sponsored by AICTE and organized by "The Department of Computer Science & Technology (Central University of Jharkhand, Ranchi)"

Transcript

1. Experimentation with jupyter, papermill, and mlflow Shadab Hussain Data Scientist https://shadabhussain.com/
2. A tool for data science at scale • Rich web client (HTML, images, videos, LaTeX) • Code (40+ programming languages) • Results • Share (email, Dropbox, GitHub) • Reproduce https://jupyter.org/ https://towardsdatascience.com/optimizing-jupyter-notebook-tips-tricks-and-nbextensions-26d75d502663
3. Tool for parameterizing and executing Notebooks https://papermill.readthedocs.io/en/latest/ index.ipynb SSE.ipynb GOLD.ipynb SNP.ipynb NIKKEI.ipynb Input Notebook Output Notebooks parametrize and run
4. Running Papermill Notebook Notebook Sources Database File Services Parameters SNP GOLD SSE HANGSENG NIKKEI Index Runtime Manager Runtime Process source notebook parameter values stream i/o messages execute cells kernel messages input { } Notebook Sinks Database File Services output notebook store
5. It adds notebook isolation. • Immutable inputs • Immutable outputs • Parameterization of notebook runs • Configurable sourcing/sinking Solves our problems for automated execution! How does this change the notebook experience?
6. https://medium.com/faun/mlflow-on-google-cloud-platform-cd8c9b04a2d8 https://mlflow.org/docs/latest/index.html An open source platform to manage the ML lifecycle Tracking Projects Models Registry Record and query experiments: code, data, config, results Packaging Data Science Code for reproducible runs on any platform General format for sending models to diverse deploy tools Store, annotate and manage models in a repository Components of MLflow
7. Tracking Server UI API Notebooks Cloud Jobs Local Apps Tracking APIS
8. Project Spec Code Dependencies Config Data Local Execution Remote Execution
9. Model Format Run Sources Inference Code Batch & Stream Processing Serving Tools Flavor1 Flavor2 Model Flavors
10. Model Registry Downstream Users Automated Jobs Rest Serving Model Developer Reviewers, CI/CD Tools
11. http://bit.ly/Experiment-with-Jupyter-Papermill-and-MLFlow DEMO
12. Useful Links: https://ljvmiranda921.github.io/notebook/2020/03/06/jupyter-notebooks-in-2020/ https://ljvmiranda921.github.io/notebook/2020/03/16/jupyter-notebooks-in-2020-part-2/ https://ljvmiranda921.github.io/notebook/2020/03/30/jupyter-notebooks-in-2020-part-3/ https://towardsdatascience.com/introduction-to-papermill-2c61f66bea30 https://pbpython.com/papermil-rclone-report-1.html https://pbpython.com/papermil-rclone-report-2.html https://medium.com/@jacky308082/intro-to-mlflow-in-python-d22367cbaa97 https://towardsdatascience.com/manage-your-machine-learning-lifecycle-with-mlflow-part-1-a7252c859f72
13. Contact Me LinkedIn linkedin.com/in/shadabhussain96/ Website shadabhussain.com Twitter twitter.com/shadabhusain786
14. THANK YOU!