Ray Serve: Overview and future roadmap

In this introductory session, we’ll discuss the motivation behind Ray Serve, who’s using Ray Serve and why, and recent features and updates, including a look at the future feature roadmap as we approach Ray 2.0.

Transcript

1. Overview and Roadmap Edward Oakes eoakes@anyscale.com
2. Outline • Ray Serve Introduction • Feedback from the Community • Plans for Ray 2.0 and beyond ◦ Preview for other talks today! 2
3. Native Libraries 3rd Party Libraries universal framework for distributed computing Ray Ecosystem 3
4. Ray Serve TL;DR Flexible, scalable compute for model serving 1. Scalable 2. Low latency 3. Efficient First-class support for multi-model serving Python-native: mix business logic & ML 4
5. Multi-model Serving Pattern: multiple models making up a single application 5 Standing Cat
6. Write a unified Python program Use your favorite tools & libraries Scale across CPUs and GPUs 6 Multi-model Serving
7. Feedback from the Community Multi-model serving is a big need and key strength 💸💸 💸 ML inference is expensive! Efficiency is key. We need better support & documentation for CI/CD • Emerging pattern: continual learning 7
8. In-progress for Ray 2.0 Double down on multi-model: Deployment Graph API 1. REST API & improved Kubernetes support 2. Integrations with best-in-breed MLOps tooling Seamless interoperability with Ray AIR Hear from Jiao later today! 8 Hear from Shreyas later today! 🤩🤩 🤩
9. Extended Roadmap • Scale-to-zero • gRPC support • Model multiplexing (100s-1000s of small models) • Shared memory for model weights • … 9 We want to hear from you!
10. • Join the community ◦ discuss.ray.io ◦ github.com/ray-project/ray ◦ @raydistributed and @anyscalecompute • Fill out our survey (QR code) for: ◦ Feedback to help shape the future of Ray Serve ◦ One-on-one sessions with developers ◦ Updates about upcoming features Please get in touch 10