Automating Machine Learning and Deep Learning workflows with Polyaxon

Automating Machine Learning and Deep Learning workflows with Polyaxon

Transcript

1. Automa'ng Machine Learning and Deep Learning Workflows !1
2. Information •Name: Mourad Mourafiq •Author of an open source plaAorm: Polyaxon •twiEer: @mmourafiq •GitHub: mouradmourafiq !2
3. What is Polyaxon •Solves the machine learning life cycle •Can be deployed on premise or on any cloud plaAorm •Is open source •Works with any library or framework •Can be used by single users or large organiza'ons •Provides compliance, audi'ng, and security !3
4. Why do you need a tool to manage your ML operations? •SoPware development is mature •Why not use the same tools? •What is the difference between soPware development and ML development? •What is the difference between soPware deployment and ML deployment? !4
5. Difference between software development and ML development •Development objec'ves •VeUng and quality assurance •Development stack !5
6. Difference between software deployment and ML deployment •ML deployment needs a Feedback Loop •Itera'on and refinement •People involved in the deployment cycle !6
7. What should an ML platform answer •Should be flexible to support open source ini'a'ves •Provides different deployment op'ons •Ideally open source •Works with any library or framework •Scales with users •Provides compliance, audi'ng, and security !7
8. ML development lifecycle • Data access • Data exploration and Feature engineering • Experimentation: iteration, packaging, reusability, reproducibility. • Scaling: Scheduling, orchestration and optimization • Tracking: code, data, params, artifacts, metrics • Insights, reporting, and knowledge distribution • Model management: packaging, deployment, and distribution • Compliance, auditing, and access management. • Automation, events, and workflows • User experience !8
9. • Data access !9
10. • Data exploration & Feature engineering !10
11. • Experimentation • Different environments: local, remote, cluster • Portability and reusability • Reproducibility !11
12. • Experimentation: Different environments !12
13. • Experimentation: Packaging •polyaxon run -f polyxonfile.yaml •polyaxon run -f polyxonfile.yaml —local !13
14. • Scheduling & Orchestration !14
15. • Hyperparams tuning & distributed training !15
16. • Experiments tracking !16
17. • Experiments tracking !17
18. • Insights, reporting, and knowledge distribution !18
19. • Model Management !19
20. • Compliance & Governance • Manage model development and deployment • Rigorous and auditable workflows !20
21. • Automation & Events • Simple yet effective specification to create workflows and automation • Integration with other pipelining tools, e.g. airflow • Events and triggers based on data, code, metrics, … !21
22. mourad@polyaxon.com twitter: @mmourafiq GitHub: mouradmourafiq https://polyaxon.com twitter: @polyaxonai GitHub: polyaxon !22