Standardizing Cloud Native ML Computing Platforms Yuki Iwai / Member of Cycloud ML Platform Team 

1. What’s ML Computing Platforms 2. Kubernetes Community Effort for AI/ML 3. Cloud Native ML Computing Platform 4. Collaboration in OSS communities 

ML Lifecycle Data Preparation Model Development Hyperparameter Optimization Model Optimization Model Training Model Fine-Tuning Model Serving Application Server Model Registry Metadata Model Artifacts Offline Feature Store Feature Store Online Feature Store Data Producers Data Products Feature Generation Feature Extraction loop Pipelines 

ML Lifecycle Data Preparation Model Development Hyperparameter Optimization Model Optimization Model Training Model Fine-Tuning Model Serving Application Server Model Registry Metadata Model Artifacts Offline Feature Store Feature Store Online Feature Store Data Producers Data Products Feature Generation Feature Extraction loop Pipelines Computing Platform Data Platform 

ML Lifecycle Data Preparation Model Development Hyperparameter Optimization Model Optimization Model Training Model Fine-Tuning Model Serving loop Pipelines CUT Computing Platform 

Kubernetes Community for AI/ML • Committees (Steering Committee etc.. • Special Interest Groups (SIG Autoscaling etc.. • Working Groups (WG Batch etc.. 

Kubernetes Community for AI/ML • Committees (Steering Committee etc.. • Special Interest Groups (SIG Autoscaling etc.. • Working Groups (WG Batch etc.. WG Batch (with Device Management) 2021 2022 2023 2024 

Kubernetes Community for AI/ML • Committees (Steering Committee etc.. • Special Interest Groups (SIG Autoscaling etc.. • Working Groups (WG Batch etc.. WG Batch (with Device Management) WG Device Management WG Serving 2021 2022 2023 2024 

Cloud Native ML Computing Platform ● Managing Devices by Kubernetes Native and Declarative ways ○ Device Sharing between multiple Pods ○ Dynamically Virtual Device creation / deletion ○ Fine-grained Device-Aware Pod scheduling ○ etc ... ● Device Management (DRA) 

Cloud Native ML Computing Platform Pod Pod Pod Pod Group ● Pod Scheduling ● Container Lifecycle Management ● Pod Group as a new Primitive ● Enhanced Container Restarting Strategies ● Device Management (DRA) 

Cloud Native ML Computing Platform Pod Pod Pod Pod Group ● Pod Scheduling ● Container Lifecycle Management Workload α Workload β Workload γ ● Workload Orchestration ● Workload Scheduling ● Quota Management Quota A Quota B Quota C ● Device Management (DRA) ● Enhanced Pod Restarting Strategies ● Resource Fairness across cluster ● Pre-Scheduling for Workload 

Cloud Native ML Computing Platform Load Balancer ● Traffic Scheduling Pod Pod Pod Pod Group model ● Pod Scheduling ● Container Lifecycle Management Workload α Workload β Workload γ ● Workload Orchestration ● Workload Scheduling ● Quota Management Quota A Quota B Quota C ● Device Management (DRA) ● ML Model / cache Placement-Aware Routing ● LLM token-size based Load Balancing 

Cloud Native ML Computing Platform Load Balancer WG Device Management WG Serving ● Traffic Scheduling Pod Pod Pod Pod Group model ● Pod Scheduling ● Container Lifecycle Management WG Batch Workload α Workload β Workload γ ● Workload Orchestration ● Workload Scheduling ● Quota Management WG Serving WG Batch Quota A Quota B Quota C ● Device Management (DRA) 

Cloud Native ML Computing Platform Load Balancer ● Device Management (DRA) ● Traffic Scheduling Pod Pod Pod Pod Group model ● Pod Scheduling ● Container Lifecycle Management Workload α Workload β Workload γ ● Workload Orchestration ● Workload Scheduling ● Quota Management Quota A Quota B Quota C 

Work together with ML Ecosystem Training Inference 

Collaboration in OSS communities • OSS as a business • Collaborate with professional contributors and maintainers • Handling user stories conflits 

Collaboration in OSS communities • Going far together with the OSS community • Standardization of technology 

Thank you