Ray Community Meetup Jan 25, 2023: Ray OOM Monitor

We are delighted to kick off New Year with our first January Ray Meetup with talks from Ray community users and committers. Join us to hear from the Ray team at Anyscale and Shopify about Ray and its usage.

Agenda:
5:00 p.m. Welcome remarks, Year 2022: Ray in Review & upcoming announcements - Jules Damji, Anyscale
Talk 1 (35-40 mins): Monitor & prevent out-of-memory problems with Ray OOM monitor - Clarence Ng, Anyscale
Q & A (10 mins)
Talk 2 (35-40 mins): How Shopify used Ray<>Tensorflow to build a Product Hierarchical Categorization model to auto classify billions of products using NLP and Computer Vision, Kshetrajna Raghavan, Shopify

Transcript

1. Ray Memory Monitor Ray team @ Anyscale
2. Agenda Introduction Problem with the existing solution How does Ray memory monitor work Deep dive into preemption policy Demo
3. Memory capacity Introduction
4. Memory capacity Introduction
5. OS (Linux) OOM killer → Triggers when the system runs of free memory pages → Kills the most memory-hungry task → Frequency is capped : stalls other processes → Ray: sets process priority for the tasks
6. Ray stalling
7. Ray stalling
8. Before: did not finish Tasks Memory used Cluster unreachable due to low memory
9. After Retried tasks from OOM Tasks Memory used
10. Thrashing 90% done 30% done OOM!
11. Thrashing 90% done 30% done
12. Thrashing 30% done
13. Thrashing 90% done 30% done
14. Observability Task A OOM! Infinite retry
15. Ray memory monitor Embedded memory monitor Worker processes Task Operating System Task Actor Actor Raylet Get resources usage Process stats
16. Ray memory monitor Embedded memory monitor Operating System Raylet Using too much Memory? Worker processes Task Task Actor Actor
17. Ray memory monitor Embedded memory monitor Operating System Raylet preempt Worker processes Task Task Actor Actor
18. Ray memory monitor Embedded memory monitor Operating System Raylet Worker processes Task Actor Actor
19. Preemption policy Requirements: If the application cannot complete we should surface that information to simplify debugging and path to resolution The application will finish even when it tries to overload the cluster → It should finish in a reasonable amount of time → workload shouldn’t hang
20. Preemption policy (Ray 2.2) → Prefer killing retriable task → Prefer killing newest task → Limited retry : could deadlock otherwise
21. Retriable task Task Crash! max_retries
22. Newest executed task Task Start time of execution = 14:38 PM Task Start time of execution = 14:22 PM OOM kill
23. Hyperparameter tuning Driver Trial Trainer Trial Trainer Data Data Data Data Data Data
24. Preempt retriable tasks first Driver Trial Trainer Trial Trainer Data Data Data Data Data Data
25. Preempt newest task Driver Trial Trainer Trial Trainer Data Data Data Data Data Data
26. Problem: starvation Driver Trial Trainer Trial Trainer Data Data Data
27. Ideal Driver Trial Trainer Trial Trainer Data Data Data
28. Problem: deadlock Task A OOM!
29. Preemption policy (Ray 2.3) → Group tasks that have the same parent if it is retriable → Preempt retriable groups → Preempt largest group → Preempt newest task within the group → Always retry task unless the task is the last member of the group
30. Group tasks (non-retriable) Driver Trial Trainer Trial Trainer Data Data Data Data Data Data
31. Group tasks (same parent) Driver Trial Trainer Trial Trainer Data Data Data Data Data Data
32. Kill tasks from the largest group Driver Trial Trainer Trial Trainer Data Data Data Data Data
33. Kill tasks from the largest group Driver Trial Trainer Trial Trainer Data Data Data Data
34. Kill tasks from the largest group Driver Trial Trainer Trial Trainer Data Data Data
35. Kill tasks from the largest group Driver Trial Trainer Trial Trainer Data Data
36. Workload fails (group has no task) Driver Trial Trainer Trial Trainer Data Data
37. Workload fails (group has no task) Task A OOM!
38. Demo
39. Summary • Ray memory monitor improves cluster stability • Latest release of Ray (2.2) ◦ preemptively kills task to prevent the node from failing ◦ Improved observability for debugging memory issues • Next release of Ray (2.3) ◦ Detects when a workload gets stuck and reports the error ◦ Fairness across tasks to avoid starvation
40. Thank you clarence@anyscale.com
41. Demo: single task Task A OOM!
42. Demo: Two trials (HPO) Driver Trial Trainer Trial Trainer Data Data Data Data Data Data
43. “Deadlock” Task A OOM!
44. “Deadlock” Task A Task B Task C OOM!
45. “Deadlock” Driver Task B Task C Task B Task C
46. “Deadlock” Driver Task B Task C OK
47. “Deadlock” Driver Task B Task B
48. “Deadlock” Driver Task B Task C OOM! Task B
49. Thrashing Driver Task B
50. Thrashing Driver Task B OOM! Task B
51. Thrashing Driver Task B
52. Thrashing Driver Task B OOM! Task B
53. Thrashing Driver Task B
54. How to use this template? Please DO NOT edit this master template. If you want to use these styles in your presentations, please create a copy of this template before you edit. OR copy slides from here into your deck. When creating a copy, please change the location of the copy to your My Drive or another location to avoid cluttering this central folder.
55. Colors 55
56. Presentation Title Slide (Light) Name, Name
57. Presentation Title - Light Name, Name
58. Presenter Information Name Name Name
59. Here is a basic information page - Light Lorem ipsum Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod tempor incididunt ut labore et dolore magna aliqua. Lorem ipsum Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod tempor incididunt ut labore et dolore magna aliqua. Lorem ipsum Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod tempor incididunt ut labore et dolore magna aliqua.
60. Here’s a basic information page Click to add text here
61. Here is a basic information page Lorem ipsum Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod tempor incididunt ut labore et dolore magna aliqua. Lorem ipsum Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod tempor incididunt ut labore et dolore magna aliqua. Lorem ipsum Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod tempor incididunt ut labore et dolore magna aliqua.
62. Slide with 2 points Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod tempor incididunt ut labore et dolore magna aliqua.Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod tempor incididunt ut labore et dolore magna aliqua.Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod tempor incididunt ut labore et dolore magna aliqua. Slide with 2 points Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod tempor incididunt ut labore et dolore magna aliqua.Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod tempor incididunt ut labore et dolore magna aliqua.Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod tempor incididunt ut labore et dolore magna aliqua.
63. Section Title
64. Section Title Section Description
65. Section Title
66. Section Title Section Description
67. Some important points Point 1 Point 2 Point 3
68. Some more important points Point 1 Point 2
69. Impact slide Description
70. Image title IMAGE SUBTITLE Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod tempor incididunt ut labore et dolore magna aliqua.Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod tempor incididunt ut labore et dolore magna aliqua.Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod tempor incididunt ut labore et dolore magna aliqua.
71. Image title slide
72. IMAGE TITLE IMAGE SUBTITLE Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod tempor incididunt ut labore et dolore magna aliqua.Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod tempor incididunt ut labore et dolore magna aliqua.Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod tempor incididunt ut labore et dolore magna aliqua.
73. Image title
74. Image Title
75. Slide with 2 columns Column title Column title
76. Slide with 3 columns Column title Column title Column title
77. Slide title Column title Column title Column title Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod tempor incididunt ut labore et dolore magna aliqua.Lorem ipsum dolor sit amet, Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod tempor incididunt ut labore et dolore magna aliqua.Lorem ipsum dolor sit amet, Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod tempor incididunt ut labore et dolore magna aliqua.Lorem ipsum dolor sit amet,
78. Thank you Any questions?