From the event loop to the distributed system

Talk for RubyConf Brazil introducing Pusher, some patterns for managing complexity in evented code, and some thoughts on distributed systems - specifically how we manage state and messaging at Pusher.

Transcript

1. From the event loop to the distributed system RubyConf Brazil Martyn Loughran – @mloughran 3rd November, 2011
2. From the event loop to the distributed system • An introduction to Pusher • The event loop • Why you’d use it • Managing complexity • The distributed system • Some general considerations • Some specific problems and how we solved them
3. Who am I? • Martyn Loughran • CTO of Pusher • We’re based in London, England • Rubyist and EventMachine enthusiast • Started building Pusher in January 2010 • Eu não falo Português
4. Part I – An introduction to Pusher
5. So what is Pusher anyway? • A web service which helps developers add real-time functionality to their web applications • It makes scaling easy • Complex distributed system I
6. Notifications
7. Chat
8. Collaboration
9. Data Sync CloudApp
10. WebSocket, the basics: • A pretty silly logo • Sockets for the web • Bidirectional • Low latency • Bandwidth efficient • Already supported in Safari, Chrome, and Firefox • Coming to IE in version 10
11. We use EventMachine • Evented IO for Ruby • em-websocket • em-hiredis
12. “Ruby doesn’t scale!”
13. 13,969,264 Number of API requests made yesterday (There are 86,400 seconds in a day)
14. 35,552,810,379 Total number of messages sent to clients since launch
15. < 10ms Mean end to end latency (excluding the internet)
16. Part II – The Event Loop
17. Why use an event loop? • To handle massive numbers of connections • To share data without Mutexes • Efficient scheduling of work
18. It’s really easy to use in Ruby require 'eventmachine' EM.run do # Start a server # Make some network connections # Create a timer # etc. end
19. WebSocket WebSocket WebSocket WebSocket Timer Timer Timer Redis The Pusher event loop Redis Pubsub ZeroMQ
20. Never block the reactor
21. Don’t get caught up in callback spaghetti http://rubylearning.com/blog/2010/10/01/an-introduction-to-eventmachine-and-how-to-avoid-callback-spaghetti/
22. Using callbacks and deferrable objects EM.run { stream = TwitterStream.new('yourtwitterusername', 'pass', 'term') stream.ontweet { |tweet| LanguageDetector.new(tweet).callback { |lang| puts "New tweet in #{lang}: #{tweet}" } } }
23. Return a deferrable from a function def do_something_complex df = EM::DefaultDeferrable.new use_lots_of_callbacks { ... { df.succeed(result) } ... df.fail(error) } return df end
24. Pass a deferrable to a strategy Juggler.juggle(:send_webhook, 100) do |df, job_params| http = EM::HttpRequest.new(job_params['url'].post({ :body => job_params["data"] }) http.callback do |response| df.success end http.errback do df.fail end end
25. Or try Fibers http://www.igvita.com/2010/03/22/untangling-evented-code-with-ruby-fibers/
26. Part III – The distributed system
27. “A distributed system is a collection of independent computers that appears to its users as a single coherent system” Distributed Systems: Principles and Paradigms, Tanenbaum and Steen 2006
28. The distributed system • Why would I build one? • Work doesn’t fit on a single machine any more • You need better availability • How can I make one? • Decouple the application so that each function is handled by a separate component • Scale components horizontally, and independantly • Make components tolerant to failure
29. State
30. Messaging
31. “Do not communicate by sharing memory; instead, share memory by communicating.” Effective Go, Google State Messaging
32. It is impossible for a distributed computer system to simultaneously provide all three of the following guarantees: - Consistency (all nodes see the same data at the same time) - Availability (a guarantee that every request receives a response about whether it was successful or failed) - Partition tolerance (the system continues to operate despite arbitrary message loss) http://en.wikipedia.org/wiki/CAP_theorem State: CAP theorem
33. State: More questions • What performance do you need? • How durable does it need to be? • How much data do you need to store? • Does it need to be highly available? • Does it need to be consistent / eventually consistent?
34. State – so what do we use?
35. MySQL ~ 20GB • Consistent • Durable • Not highly available - but this doesn’t matter • Rails models • Aggregated usage statistics
36. Redis ~ 500MB • Consistent • Very fast • Shared memory for all processes • Some current statistics, waiting to be aggregated
37. ZooKeeper ~ 1MB • Slow • Consistent • Highly available • Not partition tolerant • Processes state, and assignment of roles
38. Messaging • Central broker • AMQP - the SQL of messaging? • A single all powerful box • Simple, but hard to scale • Custom messaging topologies • ZeroMQ - point to point, fanout, pubsub, load balanced • Lots of choices, therefore complex • This is the future, but we’re not quite there yet
39. Messaging – what do we use? • Redis pub/sub • ZeroMQ • Beanstalkd
40. Some examples
41. Usage statistics and latency metrics • Loads of events • Collect incrementers and distributions in memory • Flush to redis every minute • Eventually consistent state In memory Redis MySQL
42. Storing presence information • Need to know when a user joins or leaves a channel • Needs to be consistent across processes • Use redis incrementers • Needs to survive process failure • Use a global hash, and a hash per process, with redis transactions • Consistent state
43. Optimising internal messaging • Debug console shows all events for all connections • Unnecessary messaging, most of the time • Only publish data when it’s needed • Eventually consistent, distributed state, cached in memory
44. Redis caches, and live caches # (pseudo simplified version) set = RedisLiveSet.new("debug_open") set.add('42') # redis.sadd("debug_open", 42) # redis.publish("debug_open", ["sadd", "42"]) # On another process set.member?('42') # Checks the in memory set
45. Recovering from process failure • Store process UUIDs in ZooKeeper as ephemeral files • Leader process notices process failure, and takes required action • Low volume, highly available, and consistent
46. Some other thoughts...
47. Avoid configuration
48. Distributed locking
49. Delay anything you can
50. Think about concurrency
51. In Conclusion • Consider an event loop for concurrency • EventMachine is great, you don’t need to use node.js • Think about state & messaging • It’s all about compromises; there are no right answers • Find creative solutions to your problems
52. Thanks for listening Martyn Loughran martyn@pusher.com @mloughran Come