Managing state in realtime distributed systems

Transcript

1. Managing state in realtime distributed systems Keeping it Realtime Martyn Loughran – @mloughran 7th November, 2011
2. Pusher: • Is a web service which helps developers add real-time functionality to their web applications • Scales the last mile delivery to the browser, adding higher level concepts • Is a real time distributed system
3. Distributed system
4. “A distributed system is a collection of independent computers that appears to its users as a single coherent system” Distributed Systems: Principles and Paradigms, Tanenbaum and Steen 2006
5. Why? • Scaling • High Availability
6. How to build a distributed system: • Decouple the application so that each function is handled by a separate component • Scale components horizontally, and independently • Make components tolerant to failure
7. Easier said than done • Components need to share state, which constantly changes • Components need to communicate • Handling failure is hard
8. The problems of state & solving problems in real time
9. State
10. SQL
11. What is state?
12. Long term state
13. State specific to one process
14. The kind of questions we’d like to answer in a real time distributed system
15. How many times was a URL tweeted?
16. Is a user currently online?
17. Consistent global state
18. How many users subscribed to a channel in the last 20s?
19. It is impossible for a distributed computer system to simultaneously provide all three of the following guarantees: - Consistency (all nodes see the same data at the same time) - Availability (a guarantee that every request receives a response about whether it was successful or failed) - Partition tolerance (the system continues to operate despite arbitrary message loss) http://en.wikipedia.org/wiki/CAP_theorem CAP theorem
20. Consistency isn’t free
21. Eventual consistency
22. Answer questions with map/reduce
23. “Do not communicate by sharing memory; instead, share memory by communicating.” Effective Go, Google State Messaging
24. Actors
25. Messaging
26. AMQP The SQL of messaging?
27. AMQP • Centralised message broker • Complex • Hard to scale • Hard to get high availability
28. ZeroMQ Build your own messaging
29. ZeroMQ: What is it? • Socket abstraction designed for messaging (not bytes) • Sockets include queuing • Abstracts the underlying sockets • Connect the sockets to form topologies • Messaging patterns • Devices
30. Share state by communicating • Components publish events • Package up state in bundles to solve problems
31. Problems with this approach • Scaling - how to shard your state • Event publishers and consumers easily become coupled • Handling failure is hard • Testing the stack is hard • ZeroMQ is still too low level most of the time
32. The wish list • Make sharding state easy • Define the messaging problem, rather than configuring ZMQ • Handle failure automatically • Make testing easy
33. Storm The solution?
34. Realtime map/reduce http://howfuckedismydatabase.com/nosql/
35. Storm • Stream processing • Topologies - describe computation as a graph • Graph built from sensible primitives: • Spouts & Bolts • Stream groupings: shuffle, fields, all, global • Automatic management of workers • Handles failure
36. In Conclusion • Storing state is a mess of compromises • Share state by communicating • Package up state in bundles to solve problems • Real-time map reduce is a world of new possibilities
37. Thanks! Martyn Loughran martyn@pusher.com @mloughran Come