Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Operationalizing Ray Serve

Operationalizing Ray Serve

In this session, we will introduce you to a new declarative REST API for Ray Serve, which allows you to configure and update your Ray Serve applications without modifying application files. Incorporate this API into your existing CI/CD process to manage applications on Ray Serve as part of your MLOps lifecycle.



April 14, 2022

More Decks by Anyscale

Other Decks in Technology


  1. Operationalizing Ray Serve Shreyas Krishnaswamy @ Anyscale

  2. • Existing workflow for deploying Ray Serve • Ray Serve’s

    new ops-friendly workflow • Walk-through examples from the new Serve CLI • Integration with deployment graphs Outline
  3. As you might have seen…

  4. Configuring Deployments

  5. Goal: Python-Centric API + Centralized Config CLI REST Version Control

  6. Current Ray Serve Workflow MLOps Developers Config Change Deploy to

    Prod Develop App
  7. Challenges with Config in Python • No source of truth

    • Configuration mixed with code • Tough to build custom ops tooling on top of Serve
  8. Ray Serve’s New Ops-Friendly Workflow MLOps Developers Deploy to Prod

    Develop App
  9. Operational Loop Current Workflow New Workflow

  10. Operational Advantages • Structured config is single source of truth

    • Automation: easier access to configurations options • Enables custom ops tooling for Serve using the new YAML config interface
  11. Structured Config File

  12. Ray Serve Offers the Best of Both Worlds Developer Operator

    • Quick updates • Few Replicas • Python • Consistent updates • Many Replicas • YAML
  13. Updates: While Developing

  14. Updates: In Production New!

  15. Deployment Info: While Developing

  16. Deployment Info: In Production New!

  17. Deployment Health Monitoring New!

  18. Deployment Graphs: serve run New!

  19. Deployment Graphs: serve build New!

  20. Future Plans: Improved Kubernetes Support • Structured config is basis

    for a better Kubernetes integration • Easily deploy, update, and monitor Ray Serve on K8s • Enable automated workflows like CI/CD, continual learning
  21. Future Plans: MLOps Integrations • Ray Serve is a scalable,

    compute layer • Integrations with best-in-breed MLOps tooling • Model monitoring • Drift detection • Experiment tracking • Model management
  22. • Join the community ◦ discuss.ray.io ◦ github.com/ray-project/ray ◦ @raydistributed

    and @anyscalecompute • Fill out our survey (QR code) for: ◦ Feedback to help shape the future of Ray Serve ◦ One-on-one sessions with developers ◦ Updates about upcoming features Please get in touch 22