CERN-Kubeflow-MLOps-Summit-2022

Transcript

1. Deploying and Managing a Machine Learning Platform with Kubeflow at CERN Dejan Golubovic
2. Dejan Golubovic Software Engineer at CERN dejan.golubovic.27@gmail.com linkedin.com/in/dejangolubovic Contact 2
3. Content About CERN Motivation Kubeflow at CERN Infrastructure Resource Management Use Cases Conclusions 3
4. CERN 4 Particle physics research organization based in Geneva, Switzerland Seeks to answer the fundamental questions What are the origins of the universe? What is the universe made of? International collaboration, established in 1954 17 000 employees, 110 nationalities Collaboration with institutes in 70 countries
5. Large Hadron Collider - LHC 5 Largest particle accelerator in the world 27 km ring of superconducting magnets 100 m underground Accelerate particles from opposite directions Near the speed of light Analyze collision events 4 collision points - detectors CMS, ATLAS, LHCb, ALICE
10. High Energy Physics Data at CERN 10 Around 40 million collisions per second in LHC 90 petabytes of data per year produced by all experiments Potential for machine learning in different stages of data acquisition The amount of data accessible can benefit machine learning algorithms CMS Data Acquisition System
11. Machine Learning at CERN 11 CMS L1 trigger - selecting interesting collisions for further processing Historically boosted decision trees, nowadays researching deep learning approaches Running inference in detector FPGAs, low latency Search for new physics - detecting rare events Classical, convolution, and variational autoencoders Simulations of detector outputs - faster alternative to Monte Carlo 3DGANs and Progressive GANs Anomaly detection in CERN cloud infrastructure
12. Facilitate machine learning developments at CERN Offer research groups access to hardware accelerators GPUs, FPGAs, TPUs Provide a user friendly platform for running scalable workloads ML pipelines, hyperparameter optimization, distributed training… Motivation for Centralized ML Platform 12
13. Improved Resource Usage 13 Accelerators are typically spread between different groups (Very) hard to get access to this type of resource Inefficient overall resource usage
14. Accelerators are typically spread between different groups (Very) hard to get access to this type of resource Poor overall resource usage Much improved efficiency with centralized resources Improved Resource Usage 14
15. Improved Resource Usage 15 Accelerators are typically spread between different groups (Very) hard to get access to this type of resource Poor overall resource usage Much improved efficiency with centralized resources Integrate public cloud resources
16. Kubeflow 16 Open source machine learning toolkit for Kubernetes First announced by Google in 2017 Applied to become a CNCF incubating project in October 2022 Offering variety of components to support full ML lifecycle Notebooks, pipelines, distributed training, AutoML, inference services, tensorboards… Utilizing Kubernetes to scale the workloads Declarative API, operators, auto healing, application and cluster auto scaling
17. Kubeflow at CERN 17 Initial deployments and proof of concept in 2019 In production since April 2021 Positive feedback from CERN scientific users Various use cases running in the cluster Number of users scaling according to available GPUs Different Kubeflow features utilized
18. Infrastructure 18 Openstack based on premise private cloud Exposing GPUs directly or as virtual GPUs Creating clusters on demand Infra managed by cloud and Kubernetes teams GPU resources Nvidia V100, V100s and T4 (In progress) Nvidia A100 Openstack Virtual Nodes Kubernetes Physical Nodes Kubeflow
19. GitOps with ArgoCD 19 Single source of truth for Kubeflow instances Manage multiple instances Production, staging, gcp, azure, aws… Keep history of modification for each instance Easily deploy and manage additional applications Velero for backups Resource quotas for specific profiles Custom prometheus exporters
20. EOS network file system, most common storage for datasets Enabled as a daemon set in Kubernetes cluster User authentication via Kerberos CernVM file system (CVMFS) enabled with CSI Openstack Manila for PVCs S3 object storage supported via S3 clients authentication s3.cern.ch or public cloud providers (Amazon S3, Google Object Storage…) Internal CERN Harbor registry for images 20 Storage Integration
21. Resources assigned per profile Memory, CPU, GPU, Kubernetes resources… ResourceQuota and LimitRange defined for each profile Personal profiles have a quota of 1 GPU by default Quotas for group profiles can be increased Email contact by users Group profiles quotas versioned, updated with ArgoCD 21 apiVersion: v1 kind: ResourceQuota metadata: name: kf-resource-quota namespace: dgolubov status: hard: limits.cpu: "5" limits.memory: 10Gi limits.nvidia.com/gpu: "1” (soon) limits.nvidia.com/gpu.shared: "1" requests.cpu: "5" requests.memory: 10Gi requests.nvidia.com/gpu: "1" (soon) requests.nvidia.com/gpu.shared: "1" persistentvolumeclaims: 15 pods: 100 Resource Management
22. Multiple pods can access a single GPU, allowing for time sharing Useful for smaller workloads, ex. notebooks with infrequent GPU utilization Issue - some pods (workloads) can take entire GPU memory Potential solution Define a GPU memory threshold (for example ¼ of total memory) Implement a daemonset for terminating all GPU processes above the memory threshold Inform users to limit their workloads within the defined threshold Pros - improve resource utilization and guarantee GPU availability Cons - reduce available GPU memory for each user 22 NVIDIA GPU Time Slicing
23. Support for physically sliced NVIDIA A100 GPUs A GPU memory is physically split, without concurrent access Useful for medium to large workloads that require constant GPU access 23 NVIDIA Multi Instance GPU (MIG)
24. 24 Use Case - Openlab 3DGAN Accelerating GAN training using highly parallel hardware on public cloud, CHEP 2021 https://doi.org/10.1051/epjconf/202125102073
25. 1 to 128 GPUs 3550 to 35 seconds per epoch x100 speedup Almost the same total cost Use Case - Openlab 3DGAN 25
26. 26 Use Case - CMS Jet Tagging Use detector coordinates to represent jets as particle clouds Analogous to point clouds in computer vision problems x z
27. 27 Use Case - CMS Jet Tagging
28. 28 Use Case - DESY 3GAN
29. Machine learning growing as a solution for different problems in high energy physics With Kubeflow at CERN, we are demonstrating benefits of using a centralized ML platform End-to-end open-source infrastructure Improved resource utilization Improved user experience, access to more resources and features Kubeflow very well received across CERN community Upstream Kubeflow manifests ready for deployment Customizations might be needed per organization’s requirements and infrastructure After the CNCF incubation, Kubeflow expected to gain even more traction 29 Conclusions
30. KubeCon Europe 2022, May 17 2022 Jet Energy Corrections with GNN Regression using Kubeflow @ CERN KubeCon North America 2021, October 12 2021 A Better and More Efficient ML Experience for CERN Users 25th International Conference on Computing in High-Energy and Nuclear Physics, May 20 2021 Training and Serving ML workloads with Kubeflow at CERN KubeCon Europe 2021, May 6 2021 Building and Managing a Centralized ML Platform with Kubeflow at CERN 30 Previous Talks
31. Thank you! Q&A