Standardizing Cloud Native ML Computing Platforms

Transcript

1. Standardizing Cloud Native ML Computing Platforms Yuki Iwai / Member of Cycloud ML Platform Team
2. 1. What’s ML Computing Platforms 2. Kubernetes Community Effort for AI/ML 3. Cloud Native ML Computing Platform 4. Collaboration in OSS communities
3. ML Lifecycle Data Preparation Model Development Hyperparameter Optimization Model Optimization Model Training Model Fine-Tuning Model Serving Application Server Model Registry Metadata Model Artifacts Offline Feature Store Feature Store Online Feature Store Data Producers Data Products Feature Generation Feature Extraction loop Pipelines
4. ML Lifecycle Data Preparation Model Development Hyperparameter Optimization Model Optimization Model Training Model Fine-Tuning Model Serving Application Server Model Registry Metadata Model Artifacts Offline Feature Store Feature Store Online Feature Store Data Producers Data Products Feature Generation Feature Extraction loop Pipelines Computing Platform Data Platform
5. ML Lifecycle Data Preparation Model Development Hyperparameter Optimization Model Optimization Model Training Model Fine-Tuning Model Serving loop Pipelines CUT Computing Platform
6. Kubernetes Community for AI/ML • Committees (Steering Committee etc.. • Special Interest Groups (SIG Autoscaling etc.. • Working Groups (WG Batch etc..
7. Kubernetes Community for AI/ML • Committees (Steering Committee etc.. • Special Interest Groups (SIG Autoscaling etc.. • Working Groups (WG Batch etc.. WG Batch (with Device Management) 2021 2022 2023 2024
8. Kubernetes Community for AI/ML • Committees (Steering Committee etc.. • Special Interest Groups (SIG Autoscaling etc.. • Working Groups (WG Batch etc.. WG Batch (with Device Management) WG Device Management WG Serving 2021 2022 2023 2024
9. Cloud Native ML Computing Platform • Managing Devices by Kubernetes Native and Declarative ways ◦ Device Sharing between multiple Pods ◦ Dynamically Virtual Device creation / deletion ◦ Fine-grained Device-Aware Pod scheduling ◦ etc ... • Device Management (DRA)
10. Cloud Native ML Computing Platform Pod Pod Pod Pod Group • Pod Scheduling • Container Lifecycle Management • Pod Group as a new Primitive • Enhanced Container Restarting Strategies • Device Management (DRA)
11. Cloud Native ML Computing Platform Pod Pod Pod Pod Group • Pod Scheduling • Container Lifecycle Management Workload α Workload β Workload γ • Workload Orchestration • Workload Scheduling • Quota Management Quota A Quota B Quota C • Device Management (DRA) • Enhanced Pod Restarting Strategies • Resource Fairness across cluster • Pre-Scheduling for Workload
12. Cloud Native ML Computing Platform Load Balancer • Traffic Scheduling Pod Pod Pod Pod Group model • Pod Scheduling • Container Lifecycle Management Workload α Workload β Workload γ • Workload Orchestration • Workload Scheduling • Quota Management Quota A Quota B Quota C • Device Management (DRA) • ML Model / cache Placement-Aware Routing • LLM token-size based Load Balancing
13. Cloud Native ML Computing Platform Load Balancer WG Device Management WG Serving • Traffic Scheduling Pod Pod Pod Pod Group model • Pod Scheduling • Container Lifecycle Management WG Batch Workload α Workload β Workload γ • Workload Orchestration • Workload Scheduling • Quota Management WG Serving WG Batch Quota A Quota B Quota C • Device Management (DRA)
14. Cloud Native ML Computing Platform Load Balancer • Device Management (DRA) • Traffic Scheduling Pod Pod Pod Pod Group model • Pod Scheduling • Container Lifecycle Management Workload α Workload β Workload γ • Workload Orchestration • Workload Scheduling • Quota Management Quota A Quota B Quota C
15. Work together with ML Ecosystem Training Inference
16. Collaboration in OSS communities • OSS as a business • Collaborate with professional contributors and maintainers • Handling user stories conflits
17. Collaboration in OSS communities • Going far together with the OSS community • Standardization of technology
18. Thank you