Actor Systems in Rust (Boston Rust Meetup Version)

Transcript

1. Actor Systems in Rust: Techniques and Tradeoffs Andrew J. Stone Boston Rust Meetup March 14, 2018
2. Haret Distributed Coordinator and KV Store
3. Haret Multiple Independent Consensus Groups
4. Haret Protocol Viewstamped Replication Revisited for Consensus
5. Haret Backend • Versioned persistent trie per consensus group • CAS on entire subtrees • Typed Leaves • Blob • Queue • Set
6. node3@10.10.0.3 node1@10.10.0.1 node2@10.10.0.2 p1 p2 p3 Request (1) Prepare (2) PrepareOk (3) Prepare (2) PrepareOk (3) Reply (4)
7. Actors Processes That Communicate Only via Asynchronous Msg Passing
8. Actors Straightforward Representation of Peers in a Consensus Group
9. Rabble Actor Abstractions for Implementing Distributed Algorithms
10. Rabble Dynamic Cluster Membership, Transport and Msg Routing
11. Rabble Designed for Testability
12. Rabble A Work in Progress
13. Rabble Provides a clean way to implement VRR in Haret
14. Naming pub struct Pid { pub group: Option<String>, pub name: String, pub node: NodeId, }
15. Messages pub enum Msg<T> { User(T), Timeout(TimerId) }
16. Envelopes pub struct Envelope<T> { pub to: Pid, pub from: Pid, pub msg: Msg<T> }
17. pub trait Process<M, T=ExecutorTerminal<M>> : Send where T: Terminal<M> { /// Handle messages from other actors fn handle(&mut self, msg: Msg<M>, from: Pid, terminal: &mut T); } Processes
18. Terminal pub trait Terminal<T> { fn send(&mut self, to: Pid, msg: T); fn start_timer(&mut self, Duration) -> TimerId; fn cancel_timer(&mut self, TimerId); }
19. Echo Server P1 (Client) Envelope { to: P2, from: P1, msg: Msg::User<Hello> } Envelope { to: P1, from: P2, msg: Msg::User<Hello> } P2 (Server)
20. impl<T> Process<Hello, T> for EchoServer where T: Terminal<Hello> { fn handle(&mut self, msg: Msg<Hello>, from: Pid, terminal: &mut T) { match msg { Msg::User(Hello) => { terminal.send(from, Hello); }, _ => () } } }
21. Services Interact with Processes from Outside Rabble
22. Senders pub trait Sender<T: Send> : Send { fn send(&self, msg: T) -> Result<()> }
23. Executor Locates Processes and Services and Routes Envelopes
25. Property Based Testing 1. Construct a group of processes in an initial state 2. Generate a schedule of test messages 3. Call the handle method of a process with a test message 4. Collect output messages of handle method from terminal 5. Schedule output messages
26. Test Scheduling Trigger Protocol State Transitions with Explicit Timeouts
27. Test Assertions •Pre/Post conditions •Global state invariants
28. Failing Tests Failing Schedules Run as a Regression Suite
29. Debugging Each Message Receipt is One Step in a Test Schedule
30. Cluster Server Non-blocking Single Threaded Server with TCP Connections to every node
31. Membership Dynamic Membership Maintained in an Observed-Remove Set
32. Nodes A Way for Users to Interact with Actors
33. Services Actors that Run Blocking and CPU Intensive Operations
34. Takeaways
35. Takeaway Dynamic Messaging is Complex Expensive, and Unergonomic for Distributed Actors in Rust
36. Takeaway Actors are not a Universal Mechanism for Concurrency in Rust
37. Takeaway Design your systems to make deterministic testing easier
38. Takeaway Compromise and Pragmatism allow a good enough solution that works NOW
39. Improvements • Use String Interning for Pids • Add more histograms and better debugging support • Use associated types instead of generics for processes? • Add support for supervision via catch_unwind • Rewrite the docs to cover the latest design
40. Links • https://github.com/andrewjstone/rabble • https://github.com/andrewjstone/orset • https://github.com/vmware/haret
41. Thanks! Tom Santero Justin Sheehy VMware