Operationalizing Ray Serve

In this session, we will introduce you to a new declarative REST API for Ray Serve, which allows you to configure and update your Ray Serve applications without modifying application files. Incorporate this API into your existing CI/CD process to manage applications on Ray Serve as part of your MLOps lifecycle.

Transcript

1. Operationalizing Ray Serve Shreyas Krishnaswamy @ Anyscale
2. • Existing workflow for deploying Ray Serve • Ray Serve’s new ops-friendly workflow • Walk-through examples from the new Serve CLI • Integration with deployment graphs Outline
3. As you might have seen…
4. Configuring Deployments
5. Goal: Python-Centric API + Centralized Config CLI REST Version Control MLOps
6. Current Ray Serve Workflow MLOps Developers Config Change Deploy to Prod Develop App
7. Challenges with Config in Python • No source of truth • Configuration mixed with code • Tough to build custom ops tooling on top of Serve
8. Ray Serve’s New Ops-Friendly Workflow MLOps Developers Deploy to Prod Develop App
9. Operational Loop Current Workflow New Workflow
10. Operational Advantages • Structured config is single source of truth • Automation: easier access to configurations options • Enables custom ops tooling for Serve using the new YAML config interface
11. Structured Config File
12. Ray Serve Offers the Best of Both Worlds Developer Operator • Quick updates • Few Replicas • Python • Consistent updates • Many Replicas • YAML
13. Updates: While Developing
14. Updates: In Production New!
15. Deployment Info: While Developing
16. Deployment Info: In Production New!
17. Deployment Health Monitoring New!
18. Deployment Graphs: serve run New!
19. Deployment Graphs: serve build New!
20. Future Plans: Improved Kubernetes Support • Structured config is basis for a better Kubernetes integration • Easily deploy, update, and monitor Ray Serve on K8s • Enable automated workflows like CI/CD, continual learning
21. Future Plans: MLOps Integrations • Ray Serve is a scalable, compute layer • Integrations with best-in-breed MLOps tooling • Model monitoring • Drift detection • Experiment tracking • Model management
22. • Join the community ◦ discuss.ray.io ◦ github.com/ray-project/ray ◦ @raydistributed and @anyscalecompute • Fill out our survey (QR code) for: ◦ Feedback to help shape the future of Ray Serve ◦ One-on-one sessions with developers ◦ Updates about upcoming features Please get in touch 22