Ray Internals: Object Management with the Ownership Model (Stephanie Wang & Yi Cheng, Anyscale)

In this talk, we'll do a deep dive into Ray's distributed object management layer. We'll explain the Ray execution model and the basics behind the Ray distributed object store. Next, we'll describe the challenges with achieving both performance and reliability for object management. We'll present our solution to this problem, which is based on a novel concept called ownership that ensures object metadata consistency with low overhead. Finally, we'll present some exciting upcoming work on how to extend ownership to better support recent use cases.

Transcript

1. Ray Internals: Object Management with the Ownership Model Stephanie Wang Yi Cheng
2. Applications Why ownership in Ray 1.0?
3. Application: Model serving Request 1. Tasks are dynamically generated 2. Tasks in the 10s of ms 3. Requires efficient data movement Request Request Image data Model Model Router
4. Application: Model serving Router becomes a bottleneck because it must copy all preprocessed images to the workers.
5. Application: Model serving Passing ObjectRefs instead of data avoids the Router bottleneck, but latency still suffers from a centralized design.
6. Application: Model serving With ownership, application gets automatic and decentralized memory management for image data.
7. Evaluation: Online video processing frame3 ... ... ... ... ... transform’ transform frame0 Decode Flow Cumulative Sum Smooth Sink Invocation Legend Task (RPC) 1. Tasks in the milliseconds 2. Complex data dependencies 3. Pipelined parallelism State dependency frame1 frame2
8. Evaluation: Online video processing (60 videos) Centralized = Ray modified with writes to a centralized metadata store
9. Evaluation: Online video processing (60 videos) TODO: Add leases plot
10. Evaluation: Online video processing (60 videos) Latency with ownership is lower because each video has a different owner.
11. Evaluation: Online video processing with failures Recovery when the owner is intact, with lineage reconstruction.
12. Evaluation: Online video processing with failures Recovery from owner failure using application-level checkpoints to bound re-execution.
13. Live input video Stabilized video
14. An overview of ObjectRefs in Ray Combining distributed memory + futures
15. RPC model Driver Worker 1 Worker 2 o1=f() o1 o2=f() o2 o3=add( o1,o2) o3 o2 o1 o1 = f() o2 = f() o3 = add(o1, o2) Problems: • Data movement • Parallelism
16. Data movement: RPC model +distributed memory Driver Worker 1 Worker 2 o1=f() o3=add( o1,o2) o1 o1 o3 o3 o2=f() o2 Distributed memory: Ability to reference data stored in the memory of a remote process. • Application can pass by reference • System manages data movement
17. Parallelism: RPC model +futures Worker 1 Worker 2 o2=f() o1=f() o1 o2 o3=add( o1,o2) o2 o1 o3 Driver Futures: Ability to reference data that has not yet been computed. • Application can specify parallelism and data dependencies • System manages task scheduling
18. ObjectRefs in Ray: distributed memory + futures • Performance: System handles data movement and parallelism • Generality: RPC-like interface (data is immutable). Application does not specify when or where computation should execute. Driver Worker 1 Worker 2 o1=f() o2=f() o1 o2 o3=add( o1,o2) o3 o3 o1
19. System challenges and requirements What is the right architecture for managing ObjectRefs?
20. System requirements For generality, the system must impose low overhead. Analogy: gRPC can execute millions of tasks/s. Can we do the same for ObjectRefs in Ray? Goal: Build a system that guarantees fault tolerance with low task overhead for ObjectRefs.
21. Challenge: ObjectRefs introduce distributed shared state f() f() driver add(o1,o2) o2 o1 Invocation Legend Task (RPC) Data dependency o1
22. Multiple processes refer to the same value. add(o1,o2) f() f() driver o2 o1 o1 o1 Dereferencing an ObjectRef requires coordination. 1. The process that specifies how the value is created and used. 2. The process that creates the value. 3. The process that uses the value. 4. The physical location of the value. Challenge: ObjectRefs introduce distributed shared state
23. Requirements for dereferencing a value: • Retrieval: The location of the value • Garbage collection: Whether the value is referenced Requirements in the presence of failures: • Detection: The location of the task that returns the value. • Recovery: A description of the task and its dependencies. • Persistence: Metadata should survive failures. System requirements
24. Requirements for dereferencing a value: • Retrieval: The location of the value • Garbage collection: Whether the value is referenced Requirements in the presence of failures: • Detection: The location of the task that returns the value. • Recovery: A description of the task and its dependencies. • Persistence: Metadata should survive failures. Challenge: Recording this metadata, while ensuring latency and throughput for dynamic and fine-grained tasks. System requirements
25. Existing solutions Architecture Coordination Performance Leases (decentralized, Ray <v0.8) Workers coordinate. For example, use leases to detect a task failure. Asynchronous metadata updates. Scale by adding more worker nodes. Centralized master Master records all metadata updates and handles all failures. Can scale through sharding, but high overhead due to synchronous updates.
26. Master Option 1: A centralized architecture (Dask, Spark, etc.) Driver Worker Worker Performance: Can scale through sharding, but high overhead for short tasks due to synchronous updates Coordination: Master records all metadata updates and handles all failures
27. Option 2: Decentralized leases (Ray <v0.8) Driver Worker Worker Lease manager Performance: Asynchronous metadata updates. Scale by adding more worker nodes. Coordination: Workers coordinate by acquiring (e.g., by acquiring leases on task execution).
28. The ownership model Achieving fault tolerance without giving up performance.
29. Our approach: Ownership Existing solutions do not take advantage of the inherent structure of a Ray application. f() f() driver add(o1,o2) o2 o1 1. Task graphs are hierarchical. 2. An ObjectRef is often passed within the scope of the caller (“passing downwards”).
30. Our approach: Ownership Existing solutions do not take advantage of the inherent structure of a Ray application. 1. Task graphs are hierarchical. Insight: By leveraging the structure of Ray applications, we can decentralize without requiring expensive coordination. 2. An ObjectRef is often passed within the scope of the caller (“passing downwards”). f() f() add(o1,o2) o2 o1 driver
31. Our approach: Ownership Insight: By leveraging the structure of Ray applications, we can decentralize without requiring expensive coordination. Architecture Failure handling Performance Ownership: The worker that calls a task owns the returned ObjectRef. Each worker is a “centralized master” for the objects that it owns. No additional writes on the critical path of task execution. Scaling through nested function calls.
32. Ownership: Challenges • Failure recovery ◦ Recovering a lost worker ◦ Recovering a lost owner • Garbage collection and memory safety
33. Ownership: Challenges • Failure recovery • Recovering a lost worker • Recovering a lost owner • Garbage collection and memory safety
34. Node 3 Object Store Worker Task scheduling Node 2 Object Store Worker Node 1 Worker Obj Task Val Loc C Y A X B A Obj Task Val Loc Y C(X) Obj Task Val Loc X B()
35. Node 3 Object Store Worker Task scheduling Node 2 Object Store Worker Node 1 Worker Obj Task Val Loc X B() Y C(X) A C X Y A B B 2 A task’s pending location is written locally at the owner. N2 1
36. Node 3 Object Store Worker Distributed memory management Node 2 Object Store Worker Node 1 Worker Obj Task Val Loc X B() Y C(X) A B X: N2 4 X 3 C Y A X B *X N2 5 Owner tracks locations of objects stored in distributed memory.
37. Node 2 Object Store X Worker Node 3 Object Store Worker Task scheduling with dependencies Node 1 Worker Obj Task Val Loc X B() *X N2 Y C(X) A X Y B C A Obj O. X W1 C N3
38. Node 2 Object Store X Worker Node 3 Object Store Worker Worker failure Node 1 Worker Obj Task Val Loc X B() *X N2 Y C(X) N3 A C Obj O. X W1 B Y A X C Reference holders only need to check whether the owner is alive.
39. Node 3 Object Store Worker C Obj O. X W1 Worker recovery Node 4 Object Store Worker Owner coordinates lineage reconstruction. Node 1 Worker Obj Task Val Loc X B() *X N2 Y C(X) N3 A B *X N4 B Y A X C
40. Node 2 Object Store X Worker Node 3 Object Store Worker Owner failure Node 1 Worker Obj Task Val Loc X B() *X N2 Y C(X) N3 A C Obj O. X W1 B Y A X C
41. Node 2 Object Store Worker Node 3 Object Store Worker Owner recovery C Obj O. X W1 B Y A X C References fate-share with the object’s owner. X
42. Node 2 Object Store Worker Node 3 Object Store Worker Owner recovery C Obj O. X W1 References fate-share with the object’s owner. X C X Y B A A’s owner
43. Node 2 Object Store Worker Node 3 Object Store Worker Owner recovery Node 4 Worker Obj Task Val Loc A Leveraging the application’s hierarchical structure: the owner of A recovers A. C X Y B A A’s owner Obj Task Val Loc X B() Y C(X)
44. Ownership transfer Addressing the limitations of ownership
45. Limitations of ownership Ownership model: Whoever creates an ObjectRef owns the metadata. + Using a single owner guarantees consistency and low latency + Decentralize the system according to the application structure + Use lineage reconstruction to recover from failures BUT: - Ray’s flexibility makes it difficult to apply lineage reconstruction in all scenarios - Objects (and their lineage) fate-share with their owner - This is fine if ObjectRefs are only passed downwards, but what if they’re not?
46. Limitations of ownership Ownership model: Whoever creates an ObjectRef owns the metadata. + Using a single owner guarantees consistency and low latency + Decentralize the system according to the application structure + Use lineage reconstruction to recover from failures BUT: - Ray’s flexibility makes it difficult to apply lineage reconstruction in all scenarios - Objects (and their lineage) fate-share with their owner - This is fine if ObjectRefs are only passed downwards, but what if they’re not?
47. Yi’s content starts here - Problem statement - What happens if the key assumption that ObjectRefs get passed downward is broken? - Key use cases where this happens - Present solution + (briefly, optionally) 1 alternative - Finish with “we’re looking for feedback”, summary of other open questions (fault tolerance?)
48. Ownership transfer • Where does the issue come from? • Real world use cases • Designs: when & how
49. Ownership transfer • Where does the issue come from? • Real world use cases • Designs: when & how
50. Where does the issue come from? • The key assumption • object reference is passed downwards. g() driver f() X Invocation Legend Worker Owner Object x = f.remote() • What if the assumption is broken? g.remote(x) • What if the worker run f failed? • What if the driver failed? OK OK
51. Upward pattern Invocation Legend Worker Owner Object • What if the worker run f exits? @ray.remote def f(): x = ray.put("Hello World") # msg is owned by the worker return [x] x_list = f.remote() # `x_list` is owned by the driver x = ray.get(ray.get(x_list)[0]) • Driver will fail to get the inner object • The inner object won’t be able to be reconstructed. driver f() x x_list = [x] x_list • What if the driver exits? OK NOT OK
52. Lateral pattern • What if the driver exits? Invocation Legend Worker Owner Object x = f.remote() # `x` is owned by the driver actor = Actor.options(lifetime="detached").remote() actor.g.remote([x]) # `actor` doesn't own `x` g() Detached actor driver X f() • The actor won’t be able to access x any more • X can’t be reconstructed because the owner died • What if the worker run f failed? OK NOT OK
53. Ownership transfer • Where does the issue come from? • Real world use cases • Designs: when & how
54. Why these cases are important? Or what if we don’t take care of these cases? - Scale down a cluster will be hard - Won’t be able to remove zombie process - Can’t reconstructed with failures
55. Real world use case - RayDP Invocation Legend Worker Owner Object driver Data partition List of data partition Data processing
56. Real world use case Invocation Legend Worker Owner Object Cache the created tensor object Detached actor driver Create tensor object tensor object
57. Ownership transfer • Where does the issue come from? • Real world use cases • Designs: when and how
58. Two questions to answer • When to do the transfer && how to do the transfer • It needs to be easy to use • It’ll introduce as little overhead as possible • It should cover the most cases
59. When to transfer • Manually transfer the ownership • Automatically transfer by detecting scope change ◦ Detect upward transfer ◦ Detect lateral transfer ▪ To cover most cases: transfer if either owner or receiver is a detached actor ▪ Idea solution: transfer if it’s in different ownership scope
60. How to transfer Invocation Legend Worker Owner Object g() Detached actor driver X f() x = f.remote() # `x` is owned by the driver actor = Actor.options(lifetime="detached").remote() actor.g.remote([x]) # `actor` doesn't own `x` Driver Actor (x, driver) (x, driver) Plasma Store Object: x owns
61. How to transfer Invocation Legend Worker Owner Object g() Detached actor driver X f() Driver Actor (x, driver) (x, actor) Plasma Store Object: x owns owns • Ownership sharing in reference counting layer ◦ Most changes in reference counting layer ◦ Avoid physical copy of the object
62. How to transfer - alternative Invocation Legend Worker Owner Object g() Detached actor driver X f() Driver Actor (x, GCS) (x, GCS) Plasma Store Object: x GCS (x, GCS) owns • Transfer the ownership to GCS ◦ Able to have HA support if GCS support HA ◦ Potentially becomes a bottleneck
63. Ownership transfer - summary • There are three patterns of object passing • Upward/downward and lateral • Automatically transferring ownership of an object via pattern detecting • Sharing ownership among workers to “transfer” ownership
64. Ownership is a way to decentralize ObjectRef management for performance and stability. Ray whitepaper: tinyurl.com/ray-white-paper Thank you!