Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Achieving Scalability and Interactivity with Ray Serve (Jaehyun Sim & Amar Shah, Ikigai Labs)

Achieving Scalability and Interactivity with Ray Serve (Jaehyun Sim & Amar Shah, Ikigai Labs)

Creating a scalable and highly interactive data pipelining platform is a challenging task. Ikigai Labs is tackling this problem by enabling users to inject custom Python code into a traditional data pipelining platform. The challenges of such a flexible platform create non-traditional problems. Ray and Ray Serve enable solutions at scale without hurting data interactivity. We’ll explore the challenges we faced and how Ray and Ray Serve provided excellent flexibility in resolving those challenges. We will conclude with the potential of Ray Client to push the bar even further in the future.

Anyscale
PRO

July 19, 2021
Tweet

More Decks by Anyscale

Other Decks in Technology

Transcript

  1. Achieving Scalability and Interactivity with Ray Serve Jaehyun Sim, Amar

    Shah Ikigai Labs
  2. Who We Are Vinayak Ramesh CEO/Co-founder Devavrat Shah CTO/Co-founder Jaehyun

    Sim Engineering Manager Robbie Jung Software Engineer James Oh Product Manager Aldreen Venzon Customer Success Amar Shah Cloud Architect Robert Xin Data Engineer Saša Mršić UI/UX Engineer
  3. The Ikigai Platform: AI-Charged Spreadsheets Ikigai Labs offers a next-generational

    data processing and analytics tool that empowers data operators with AI and BI Connectors Datasets Flows Dashboards Automation Human in the Loop
  4. Outline • What Is The Ikigai Platform Trying To Achieve?

    • Why Ray? • How Does Ray Serve Resolve The Challenges We Faced? • How Do We Scale Ray Serve? • How Can Ray Client Take Us Further?
  5. What Is The Ikigai Platform Trying To Achieve?

  6. Mission-Critical Data Pipelining “We want to collect all attached PDF

    files from emails we received today and use them as source data”
  7. Highly Interactive Data Pipelining

  8. Instantly Browsable Data Pipelining

  9. Why Ray?

  10. Earning Scalability with Minimal Code Change to

  11. How Does Ray Serve Resolve The Challenges We Faced?

  12. Introducing Ray Serve Ray Serve is “easy-to-use” and “scalable” model

    serve library built on Ray. • Framework-agnostic Can serve not only DL models but also “arbitrary Python business logic” • Python-first Configure model serving with “pure Python code”
  13. Challenge: Conflicting Python Libraries numpy pandas nltk boto3==1.1.0 numpy pandas

    beautifulsoup4 boto3==1.16.0 numpy pandas scipy boto3==1.7.0
  14. Defining Custom Python Scripts with Ray Serve

  15. Dependency Management with Ray Serve numpy pandas nltk boto3==1.1.0 numpy

    pandas beautifulsoup4 boto3==1.16.0 numpy pandas scipy boto3==1.7.0
  16. Challenge: Task Overhead for ‘Peek’ Ikigai Platform Supports two different

    types of Data Pipelining 1. Scalable Pipeline Run - To run large volumes of data at scale 2. ‘Peek’ Run - To run samples of data quickly for sub-second data interfacing Problem: Task submission with Ray usually takes about 10 seconds If custom Python scripts are on Ray serve, how could we maintain sub-second ‘peek’ time?
  17. Dynamic Function Invocation with Ray Serve Scalable Pipeline Run (Invoke

    inside of Ray Cluster) ‘Peek’ Run (Invoke from outside of Ray Cluster) No task overhead from Task submission
  18. Generating HTTP Endpoints for ‘Peek’ to

  19. Ikigai with Ray Serve, on Kubernetes Ray Serve Manager ➔

    Create/Update Ray Serve Deployment ➔ Manages Conda environment for all Ray worker pods ➔ Optimize the number of Ray Serve instance, based on dependency requirements
  20. How Do We Scale Ray Serve?

  21. Scaling Ray Serve on Kubernetes Cluster Easy Autoscaler Deployment ➔

    `ray up cluster.yaml` ➔ Easy bootstrapping with Ray Cluster Launcher ➔ The Ray Cluster Launcher will automatically enable a load-based autoscaler
  22. Challenge: Conda Env with Autoscaler Missing conda env when autoscaler

    adds a new node
  23. Defining Conda Env in Deployments

  24. Challenge: Skewed Traffic on Serve Instances

  25. Scaling a Specific Serve Instance Individually

  26. Challenge: Deployment Race Condition

  27. Explicit Deployment Versioning This will fail if the existing version

    in the system is not equal to `old_v` This will fail if the existing version in the system is not equal to `old_v`
  28. Ikigai’s Ray Serve on Kubernetes Cluster

  29. How Can Ray Client Take Us Further?

  30. Even More Flexibility with Ray Client Ray now offers a

    remote client to connect to Ray cluster • More Natural to Dynamic Task Definitions No need to generate Python file for different task definitions • Full Ray API Support Makes the migration extremely easy
  31. Data Pipelining without Task Submission Extremely minimal changes to utilize

    full Ray API
  32. Ikigai with Ray Serve and Client All microservices and Python

    workers have access to the Ray Core and Serve
  33. Thank You! Special Thanks to Edward Oakes @ Anyscale