Ray Observability: Present and Future

Machine Learning offline and online serving users leverage Ray in radically different ways, which makes it challenging to provide a common architecture to inspect and debug performance problems.

In this talk, we discuss Ray 2. x's current observability architecture, including how to view metrics and logs and inspect the state of tasks, actors, and other resources in Ray. We discuss features like the newly revamped ray dashboard and the added ray metrics and present a roadmap for where Ray observability is going in the future with a unified observability data model.

This talk is for you if you're interested in the following:

* What’s new with ray metrics and dashboard revamp
* How to debug Ray programs using the CLI, Dashboard, and logs
* Learn how to implement observability in a general-purpose distributed system such as Ray.

Transcript

1. SangBin Cho, Anyscale Ricky Xu, Anyscale Ray Observability: Present and Future
2. We are… Software engineers at Anyscale Ray core developers Currently focusing on improving observability
3. Anyscale Who we are::Original creators of Ray, a unified framework for scalable computing What we do: Scalable compute for AI And Python Why we do it: Scaling is a necessity, scaling is hard; make distributed computing easy and simple for everyone
4. How to debug Ray Applications? If you are interested in
5. What tools are available for debugging? If you are interested in How do I debug Ray Applications?
6. How do I debug Ray Applications? What tools are available for debugging? What’s new observability features in recent Ray releases If you are interested in
7. Outline 1. Requirements in Ray Observability 2. Ray Observability before 2.0 3. What’s new in Ray 2.0? 4. Demo (Ray 2.1 + 2.2 features) 5. Future roadmap
8. Outline 1. Requirements in Ray Observability 2. Ray Observability before 2.0 3. What’s new in Ray 2.0? 4. Demo (Ray 2.1 + 2.2 features) 5. Future roadmap
9. Ray: APIs to build Distributed Systems
12. Ray Head Node
13. Ray Head Node Ray Worker Node Ray Worker Node Ray Worker Node Ray Worker Node
14. Ray Head Node
15. Ray Head Node
16. What if anything goes wrong?
17. Ray Head Node
18. Ray Head Node Actor failed
19. Ray Head Node Actor failed
20. Ray Head Node Actor failed
21. How can we still make debugging easy?
22. How can we still make debugging easy? Understand the Workflow
23. How can we still make debugging easy? Understand the Workflow Tools
24. Visibility into applications Monitoring Key Workflow for Debugging Surface problems Error Debugging with Tools and Data Fix it! Mitigation
25. Tools for the Workflow Monitor State Dashboard Time-series App Logs Events Progress Report Integrations (high level ML) Tracing Surface Error Exceptions Logs Time-series Good error message Tools and Data Debugger Profiler Tracing Integrations (metrics, logs, tracing) Logs Time-series Events State State Dashboard Progress Report App Logs Exceptions Good error message Debugger Profiler
26. Outline 1. Requirements in Ray Observability 2. Ray Observability before 2.0 3. What’s new in Ray 2.0? 4. Demo 5. Future roadmap (post Ray 2.0)
27. Application Failures (failures from user code) Debugging System failures (Hanging, OOM, etc.) Case Studies
28. Application Failures (failures from user code) Debugging System failures (Hanging, OOM, etc.) Case Studies
29. Develop Ray Applications Locally @ray.remote class Actor: def print_task(self): return “hello” actors = [ Actor.remote() for in range(4) ] refs = [ actor.print_task.remote() for actor in actors ] ray.get(refs) Actor 1 Actor 2 ref = actor.print_task.remote() Actor 3 Actor 4
30. Monitoring: Logs @ray.remote class Actor: def print_task(self): return “hello” actors = [ Actor.remote() for in range(4) ] refs = [ actor.print_task.remote() for actor in actors ] ray.get(refs) Actor 1 Actor 2 ref = actor.print_task.remote() Actor 3 Actor 4
31. Monitoring: Logs @ray.remote class Actor: def print_task(self): print(“hello”) return “hello” actors = [ Actor.remote() for in range(4) ] refs = [ actor.print_task.remote() for actor in actors ] ray.get(refs) Actor 1 Actor 2 ref = actor.print_task.remote() Actor 3 Actor 4
32. Monitoring: Logs @ray.remote def print_task(): print(“haha”) print_task.remote() • All logs from tasks/actors are printed to the driver program • Support `ray logs` API to access logs of actors, tasks, workers, and system logs.
33. Surface Error: Exceptions @ray.remote class Actor: def print_task(self): return 4 / 0 actors = [ Actor.remote() for in range(4) ] refs = [ actor.print_task.remote() for actor in actors ] ray.get(refs) Actor 1 Actor 2 ref = actor.print_task.remote() Actor 3 Actor 4
34. Ray Exceptions • All ray primitives (task, object, actor) generates a “object reference”. • If anything goes wrong with the generator of object reference, `ray.get` raises an exception @ray.remote class Actor: def print_task(self): return 4 / 0 actors = [ Actor.remote() for in range(4) ] refs = [ actor.print_task.remote() for actor in actors ] ray.get(refs)
35. Ray Exceptions • Any failures (application error & system error) should be raised as exception • All ray primitives (task, object, actor) generates a “object reference”. • If anything goes wrong with the generator of object reference, `ray.get` raises an exception class Actor: def print_task(self): return 4 / 0 actors = [ Actor.remote() for in range(4) ] refs = [ actor.print_task.remote() for actor in actors ] ray.get(refs) RayActorError: The actor died unexpectedly before finishing this task. class_name: G actor_id: e818d2f0521a334daf03540701000000 pid: 61251 namespace: 674a49b2-5b9b-4fcc-b6e1-5a1d4b9400d2 ip: 127.0.0.1 The actor is dead because its worker process has died. Worker exit type: UNEXPECTED_SYSTEM_EXIT Worker exit detail: Worker unexpectedly exits with a connection error code 2. End of file. There are some potential root causes. (1) The process is killed by SIGKILL by OOM killer due to high memory usage. (2) ray stop --force is called. (3) The worker is crashed unexpectedly due to SIGSEGV or other unexpected errors.
36. Debugging: Debugger @ray.remote class Actor: def print_task(self): return 4 / 0 actors = [ Actor.remote() for in range(4) ] refs = [ actor.print_task.remote() for actor in actors ] ray.get(refs) Actor 1 Actor 2 ref = actor.print_task.remote() Actor 3 Actor 4
37. Debugging: Debugger @ray.remote class Actor: def print_task(self): breakpoint() return 4 / 0 actors = [ Actor.remote() for in range(4) ] refs = [ actor.print_task.remote() for actor in actors ] ray.get(refs) Actor 1 Actor 2 ref = actor.print_task.remote() Actor 3 Actor 4
38. Interactive Debugging: Ray Debugger • When developing Ray, users want to use familiar debugger such as pdb • Ray has its own pdb integration to support “distributed debugging”
39. Debugging: Tracing @ray.remote class Actor: def print_task(self): big_obj = do_busy_work() return big_obj actors = [ Actor.remote() for in range(4) ] refs = [ actor.print_task.remote() for actor in actors ] ray.get(refs) Actor 1 Actor 2 ref = actor.print_task.remote() Actor 3 Actor 4
40. Debugging: Tracing @ray.remote class Actor: def print_task(self): big_obj = do_busy_work() return big_obj actors = [ Actor.remote() for in range(4) ] refs = [ actor.print_task.remote() for actor in actors ] ray.get(refs) Actor 1 Actor 2 ref = actor.print_task.remote() Actor 3 Actor 4 Why is actor 1’s print_task slow?
41. Performance: Ray Timeline • Sometimes, users want to trace low-level per-worker operation to find bottlenecks in performance • Ray supports `ray timeline` API for this!
42. Visualization • Metrics • Dashboard Huge improvement in Ray 2.1 & 2.2!
43. Outline 1. Requirements in Ray Observability 2. Ray Observability before 2.0 3. What’s new in Ray 2.0? 4. Demo (Ray 2.1 + 2.2 features) 5. Future roadmap
44. What’s new in Ray 2.0 Visibility into applications State APIs Dashboard Time-series Logs Events Progress Bar Integrations (high level ML) Tracing Surface problems Exceptions Logs Time-series Good error message Debugging with tools and data Debugger Profiler Tracing Integrations (metrics, logs, tracing) Logs Time-series Events State APIs State APIs State API + Better Ray Exceptions State APIs Good error message State APIs State APIs
45. CLIs Python SDKs > ray summary actors > ray list tasks > ray logs import ray.experimental.state.api as api api.summarize_tasks() api.list_nodes() api.get_log() Key Ray 2.0 Feature: State APIs
46. State API example ray summary objects tasks actors
47. State API example > ray summary tasks [--address <RAY_ADDRESS>]
48. State API example > ray summary actors
49. State API example > ray summary objects
50. State API example ray list / get objects tasks actors workers runtime-envs placement groups nodes jobs
51. State API example > ray list tasks
52. State API example > ray list tasks -f scheduling_state!=FINISHED -f name=PPO.train
53. State API example 53 > ray list actors -f node_id=<node_id>
54. State API example > ray get actors <actor id>
55. State API example > ray get actors <actor id>
56. State API example > ray get actors <actor id>
57. State API example ray logs workers cluster actors
58. State API example > ray logs cluster
59. State API example > ray logs worker*
60. State API example > ray logs worker* > ray logs <log file name> –f
61. Key Ray 2.0 Feature: State APIs https://docs.ray.io/en/latest/ray-observability/state/state-api.html Check out the documentation for more details: 👇
62. Outline 1. Requirements in Ray Observability 2. Ray Observability before 2.0 3. What’s new in Ray 2.0? 4. Demo (Ray 2.1 + 2.2 features) 5. Future roadmap
63. Demo Read Images Preprocessing Training 16 TrainWorker actors Job Supervisor S3 Storage (Images) Read -> split -> preprocessing (100 of workers)
64. Outline 1. Requirements in Ray Observability 2. Ray Observability before 2.0 3. What’s new in Ray 2.0? 4. Demo 5. Future roadmap
65. - State API Beta Future Roadmap
66. State API Beta More info to come: - Historical Task State Information - Resource usage/requirements of actors and tasks. - Duration, exit code - Relationship between different resources - What tasks have an actor runs? - What logical resources is used on a node? More stable API Better usability in options and formatting
67. - State API Beta - Dashboard Usability Improvement Future Roadmap
68. Dashboard Usability Improvement
69. - State API Beta - Dashboard Usability Improvement - Advanced Task Drilldown Future Roadmap
70. Advanced Task Drilldown
71. Advanced Task Drilldown
72. Advanced Task Drilldown
73. - State API Beta - Dashboard Usability Improvement - Advanced Task Drilldown - Advanced Visualization Future Roadmap
74. Advanced Visualization
75. 💬 Help us shape the future of Ray Observability Join #ray_observability in Ray slack or scan the QR code 👇
76. Thank you.