[Ray Summit 2022] Achieving Scalability and Interactivity with Ray Serve

Creating a scalable and highly interactive data pipelining platform is a challenging task. Ikigai Labs is tackling this problem by enabling users to inject custom Python code into a traditional data pipelining platform. The challenges of such a flexible platform create non-traditional problems. Ray and Ray Serve enable solutions at scale without hurting data interactivity. We’ll explore the challenges we faced and how Ray and Ray Serve provided excellent flexibility in resolving those challenges. We will conclude with the potential of Ray Client to push the bar even further in the future.

Transcript

1. Achieving Scalability and Interactivity with Ray Serve Jaehyun Sim, Amar Shah Ikigai Labs
2. Who We Are Vinayak Ramesh CEO/Co-founder Devavrat Shah CTO/Co-founder Jaehyun Sim Engineering Manager Robbie Jung Software Engineer James Oh Product Manager Aldreen Venzon Customer Success Amar Shah Cloud Architect Robert Xin Data Engineer Saša Mršić UI/UX Engineer
3. The Ikigai Platform: AI-Charged Spreadsheets Ikigai Labs offers a next-generational data processing and analytics tool that empowers data operators with AI and BI Connectors Datasets Flows Dashboards Automation Human in the Loop
4. Outline • What Is The Ikigai Platform Trying To Achieve? • Why Ray? • How Does Ray Serve Resolve The Challenges We Faced? • How Do We Scale Ray Serve? • How Can Ray Client Take Us Further?
5. What Is The Ikigai Platform Trying To Achieve?
6. Mission-Critical Data Pipelining “We want to collect all attached PDF files from emails we received today and use them as source data”
7. Highly Interactive Data Pipelining
8. Instantly Browsable Data Pipelining
9. Why Ray?
10. Earning Scalability with Minimal Code Change to
11. How Does Ray Serve Resolve The Challenges We Faced?
12. Introducing Ray Serve Ray Serve is “easy-to-use” and “scalable” model serve library built on Ray. • Framework-agnostic Can serve not only DL models but also “arbitrary Python business logic” • Python-first Configure model serving with “pure Python code”
13. Challenge: Conflicting Python Libraries numpy pandas nltk boto3==1.1.0 numpy pandas beautifulsoup4 boto3==1.16.0 numpy pandas scipy boto3==1.7.0
14. Defining Custom Python Scripts with Ray Serve
15. Dependency Management with Ray Serve numpy pandas nltk boto3==1.1.0 numpy pandas beautifulsoup4 boto3==1.16.0 numpy pandas scipy boto3==1.7.0
16. Challenge: Task Overhead for ‘Peek’ Ikigai Platform Supports two different types of Data Pipelining 1. Scalable Pipeline Run - To run large volumes of data at scale 2. ‘Peek’ Run - To run samples of data quickly for sub-second data interfacing Problem: Task submission with Ray usually takes about 10 seconds If custom Python scripts are on Ray serve, how could we maintain sub-second ‘peek’ time?
17. Dynamic Function Invocation with Ray Serve Scalable Pipeline Run (Invoke inside of Ray Cluster) ‘Peek’ Run (Invoke from outside of Ray Cluster) No task overhead from Task submission
18. Generating HTTP Endpoints for ‘Peek’ to
19. Ikigai with Ray Serve, on Kubernetes Ray Serve Manager ➔ Create/Update Ray Serve Deployment ➔ Manages Conda environment for all Ray worker pods ➔ Optimize the number of Ray Serve instance, based on dependency requirements
20. How Do We Scale Ray Serve?
21. Scaling Ray Serve on Kubernetes Cluster Easy Autoscaler Deployment ➔ `ray up cluster.yaml` ➔ Easy bootstrapping with Ray Cluster Launcher ➔ The Ray Cluster Launcher will automatically enable a load-based autoscaler
22. Challenge: Conda Env with Autoscaler Missing conda env when autoscaler adds a new node
23. Defining Conda Env in Deployments
24. Challenge: Skewed Traffic on Serve Instances
25. Scaling a Specific Serve Instance Individually
26. Challenge: Deployment Race Condition
27. Explicit Deployment Versioning This will fail if the existing version in the system is not equal to `old_v` This will fail if the existing version in the system is not equal to `old_v`
28. Ikigai’s Ray Serve on Kubernetes Cluster
29. How Can Ray Client Take Us Further?
30. Even More Flexibility with Ray Client Ray now offers a remote client to connect to Ray cluster • More Natural to Dynamic Task Definitions No need to generate Python file for different task definitions • Full Ray API Support Makes the migration extremely easy
31. Data Pipelining without Task Submission Extremely minimal changes to utilize full Ray API
32. Ikigai with Ray Serve and Client All microservices and Python workers have access to the Ray Core and Serve
33. Thank You! Special Thanks to Edward Oakes @ Anyscale