ZeroMQ: scriptable sockets

ZeroMQ is a socket abstraction and concurrency framework that's changing the way we think and reason about distributed systems. Mailboxes, atomic message delivery and swappable transports allow for fast, flexible and resilient network topologies. It's I/O model also sits very well with all Ruby implementations. In this talk we'll discuss :

* What's wrong with socket I/O ?
* Supported messaging patterns
* Transport agnostic messaging
* Resiliency (operations and upgrades)
* Building out topologies just in time (interjection principle)
* Performance and throughput
* Mongrel2 Ruby adapter
* How to use it from Ruby
* Small case study

Transcript

1. Wildfire Interactive, Inc. | 1600 Seaport Boulevard, Suite 500, Redwood City, CA 94063 | (888) 274-0929 ZeroMQ: scriptable sockets Lourens Naudé Helsinki Thursday, September 20, 12
2. Wildfire Interactive, Inc. | 1600 Seaport Boulevard, Suite 500, Redwood City, CA 94063 | (888) 274-0929 In 2012 it’s still too “expensive” to create and maintain integration points. Why this talk ? Thursday, September 20, 12
3. Wildfire Interactive, Inc. | 1600 Seaport Boulevard, Suite 500, Redwood City, CA 94063 | (888) 274-0929 Agenda What it is and isn’t Messages and patterns Transports and devices Resiliency Rewby Thursday, September 20, 12
4. Wildfire Interactive, Inc. | 1600 Seaport Boulevard, Suite 500, Redwood City, CA 94063 | (888) 274-0929 What’s wrong with socket I/O ? Thursday, September 20, 12
5. Wildfire Interactive, Inc. | 1600 Seaport Boulevard, Suite 500, Redwood City, CA 94063 | (888) 274-0929 Socket I/O problems Strong dev profile required Portability Flexibility Thursday, September 20, 12
6. Wildfire Interactive, Inc. | 1600 Seaport Boulevard, Suite 500, Redwood City, CA 94063 | (888) 274-0929 Zero broker, admin, cost, waste, price tag and (almost) zero latency What is ZeroMQ ? Thursday, September 20, 12
7. Wildfire Interactive, Inc. | 1600 Seaport Boulevard, Suite 500, Redwood City, CA 94063 | (888) 274-0929 IM for apps Not a server or message queue Communication / concurrency library Messaging semantics for sockets BSD sockets API Async by design Thursday, September 20, 12
8. Wildfire Interactive, Inc. | 1600 Seaport Boulevard, Suite 500, Redwood City, CA 94063 | (888) 274-0929 How did we get here ? Thursday, September 20, 12
9. Wildfire Interactive, Inc. | 1600 Seaport Boulevard, Suite 500, Redwood City, CA 94063 | (888) 274-0929 No really ... Martin Sustrik + Pieter Hintjens Everything AMQP isn’t Messages, not byte streams Financial services - optimization iMatix && community aka Crossroads I/O (fork) Thursday, September 20, 12
10. Wildfire Interactive, Inc. | 1600 Seaport Boulevard, Suite 500, Redwood City, CA 94063 | (888) 274-0929 Rich bindings ecosystem 30+ language bindings ... Thursday, September 20, 12
11. Wildfire Interactive, Inc. | 1600 Seaport Boulevard, Suite 500, Redwood City, CA 94063 | (888) 274-0929 In the wild ... and financial industry + startups Thursday, September 20, 12
12. Wildfire Interactive, Inc. | 1600 Seaport Boulevard, Suite 500, Redwood City, CA 94063 | (888) 274-0929 Super (scriptable) sockets What is a ZeroMQ socket ? Thursday, September 20, 12
13. Wildfire Interactive, Inc. | 1600 Seaport Boulevard, Suite 500, Redwood City, CA 94063 | (888) 274-0929 How ? An API model Embeddable servers Multiple transports Socket metaphor Familiar BSD socket API Multiplexing, framing etc. Thursday, September 20, 12
14. Wildfire Interactive, Inc. | 1600 Seaport Boulevard, Suite 500, Redwood City, CA 94063 | (888) 274-0929 bind() or connect() to multiple endpoints, simultaneously Multiple endpoints Thursday, September 20, 12
15. Wildfire Interactive, Inc. | 1600 Seaport Boulevard, Suite 500, Redwood City, CA 94063 | (888) 274-0929 bind() or connect() in any order Async connect Thursday, September 20, 12
16. Wildfire Interactive, Inc. | 1600 Seaport Boulevard, Suite 500, Redwood City, CA 94063 | (888) 274-0929 Socket types PUB SUB REQ REP PUSH PULL Thursday, September 20, 12
17. Wildfire Interactive, Inc. | 1600 Seaport Boulevard, Suite 500, Redwood City, CA 94063 | (888) 274-0929 Touching the water PUSH PULL PUSH PULL PULL PUSH PULL PUSH VENTILATOR SINK Thursday, September 20, 12
18. Wildfire Interactive, Inc. | 1600 Seaport Boulevard, Suite 500, Redwood City, CA 94063 | (888) 274-0929 Opaque binary data (blobs) Messages Thursday, September 20, 12
19. Wildfire Interactive, Inc. | 1600 Seaport Boulevard, Suite 500, Redwood City, CA 94063 | (888) 274-0929 Message format Framed wire protocol No contracts Serialize as you wish No auth / security Thursday, September 20, 12
20. Wildfire Interactive, Inc. | 1600 Seaport Boulevard, Suite 500, Redwood City, CA 94063 | (888) 274-0929 Message delivery Multipart messages supported Send 5GB if you like Atomic delivery NO PERSISTENCE Thursday, September 20, 12
21. Wildfire Interactive, Inc. | 1600 Seaport Boulevard, Suite 500, Redwood City, CA 94063 | (888) 274-0929 ENOTAMQ Thursday, September 20, 12
22. Wildfire Interactive, Inc. | 1600 Seaport Boulevard, Suite 500, Redwood City, CA 94063 | (888) 274-0929 Request / Reply Supported patterns Thursday, September 20, 12
23. Wildfire Interactive, Inc. | 1600 Seaport Boulevard, Suite 500, Redwood City, CA 94063 | (888) 274-0929 Request / reply REQ REP REQ REP REP REP REQ REQ Thursday, September 20, 12
24. Wildfire Interactive, Inc. | 1600 Seaport Boulevard, Suite 500, Redwood City, CA 94063 | (888) 274-0929 Request / Reply usage REQ socket load balances Lock step Bi-directional Custom routing Downstream topology flow control Thursday, September 20, 12
25. Wildfire Interactive, Inc. | 1600 Seaport Boulevard, Suite 500, Redwood City, CA 94063 | (888) 274-0929 Publish / subscribe Supported patterns Thursday, September 20, 12
26. Wildfire Interactive, Inc. | 1600 Seaport Boulevard, Suite 500, Redwood City, CA 94063 | (888) 274-0929 Publish / subscribe PUB SUB SUB SUB Thursday, September 20, 12
27. Wildfire Interactive, Inc. | 1600 Seaport Boulevard, Suite 500, Redwood City, CA 94063 | (888) 274-0929 Logging SUB PUB PUB PUB Thursday, September 20, 12
28. Wildfire Interactive, Inc. | 1600 Seaport Boulevard, Suite 500, Redwood City, CA 94063 | (888) 274-0929 Publish / subscribe usage Market data, game state etc. PUB socket distributes SUB socket filters Subscription forwarding (3.x) Thursday, September 20, 12
29. Wildfire Interactive, Inc. | 1600 Seaport Boulevard, Suite 500, Redwood City, CA 94063 | (888) 274-0929 Pipeline Supported patterns Thursday, September 20, 12
30. Wildfire Interactive, Inc. | 1600 Seaport Boulevard, Suite 500, Redwood City, CA 94063 | (888) 274-0929 Pipeline PUSH PULL PULL PULL Thursday, September 20, 12
31. Wildfire Interactive, Inc. | 1600 Seaport Boulevard, Suite 500, Redwood City, CA 94063 | (888) 274-0929 Pipeline usage Like REQ/REP, downstream reply Extendable - UNIX pipes Round-robin distribution Tasks with variable processing Thursday, September 20, 12
32. Wildfire Interactive, Inc. | 1600 Seaport Boulevard, Suite 500, Redwood City, CA 94063 | (888) 274-0929 Transport agnostic messaging Transports Thursday, September 20, 12
33. Wildfire Interactive, Inc. | 1600 Seaport Boulevard, Suite 500, Redwood City, CA 94063 | (888) 274-0929 Supported transports inproc:// - threads in a process ipc:// - processes on a box tcp:// - nodes in a network Uniform API - it just works Thursday, September 20, 12
34. Wildfire Interactive, Inc. | 1600 Seaport Boulevard, Suite 500, Redwood City, CA 94063 | (888) 274-0929 Lightweight switches Devices Thursday, September 20, 12
35. Wildfire Interactive, Inc. | 1600 Seaport Boulevard, Suite 500, Redwood City, CA 94063 | (888) 274-0929 eg. pub/sub persistence SUB PUB PUB SUB SUB Thursday, September 20, 12
36. Wildfire Interactive, Inc. | 1600 Seaport Boulevard, Suite 500, Redwood City, CA 94063 | (888) 274-0929 Device main loop f = ctx.connect(:SUB, "tcp:// 0.0.0.0:5110") b = ctx.bind(:PUB, “tcp:// 0.0.0.0:5111”) loop do m = f.recv process(m) b.send(m) end Thursday, September 20, 12
37. Wildfire Interactive, Inc. | 1600 Seaport Boulevard, Suite 500, Redwood City, CA 94063 | (888) 274-0929 Uses Proxy Routing (LRU) Filtering / de-dup Translation Persistence Thursday, September 20, 12
38. Wildfire Interactive, Inc. | 1600 Seaport Boulevard, Suite 500, Redwood City, CA 94063 | (888) 274-0929 Interjection Principle Scalable topologies Thursday, September 20, 12
39. Wildfire Interactive, Inc. | 1600 Seaport Boulevard, Suite 500, Redwood City, CA 94063 | (888) 274-0929 eg. saving bandwidth SUB PUB PUB SUB SUB SUB SUB Thursday, September 20, 12
40. Wildfire Interactive, Inc. | 1600 Seaport Boulevard, Suite 500, Redwood City, CA 94063 | (888) 274-0929 Asynchronous external dependencies. Resiliency Thursday, September 20, 12
41. Wildfire Interactive, Inc. | 1600 Seaport Boulevard, Suite 500, Redwood City, CA 94063 | (888) 274-0929 ZeroMQ resiliency Atomic message delivery Delayed retry on failure Multi bind() and connect() Unordered bind() or connect() SOA - easy upgrades Thursday, September 20, 12
42. Wildfire Interactive, Inc. | 1600 Seaport Boulevard, Suite 500, Redwood City, CA 94063 | (888) 274-0929 Water marks Throttling / flow control High Water Mark (HWM) Socket option DROPS - REP, PUB, SUB BLOCKS - REQ, PUSH Thursday, September 20, 12
43. Wildfire Interactive, Inc. | 1600 Seaport Boulevard, Suite 500, Redwood City, CA 94063 | (888) 274-0929 99% of deployments don’t need enterprise reliability Of reliability Thursday, September 20, 12
44. Wildfire Interactive, Inc. | 1600 Seaport Boulevard, Suite 500, Redwood City, CA 94063 | (888) 274-0929 Then ... Costly transactions Small topology Large ops team Little churn Thursday, September 20, 12
45. Wildfire Interactive, Inc. | 1600 Seaport Boulevard, Suite 500, Redwood City, CA 94063 | (888) 274-0929 ... now Negligible transaction cost Large topology Small ops team Rapid churn rate Thursday, September 20, 12
46. Wildfire Interactive, Inc. | 1600 Seaport Boulevard, Suite 500, Redwood City, CA 94063 | (888) 274-0929 Reliability as patterns Lazy Pirate (REQ / REP) Majordomo (queue) Titanic (persistence) Binary Star (High Availability) See http://zguide.zeromq.org Thursday, September 20, 12
47. Wildfire Interactive, Inc. | 1600 Seaport Boulevard, Suite 500, Redwood City, CA 94063 | (888) 274-0929 Low latency, high throughput. Performance Thursday, September 20, 12
48. Wildfire Interactive, Inc. | 1600 Seaport Boulevard, Suite 500, Redwood City, CA 94063 | (888) 274-0929 Lab: 6 000 000 msgs/s Ruby: 200 000 msgs/s (16 byte payload) Numbers Thursday, September 20, 12
49. Wildfire Interactive, Inc. | 1600 Seaport Boulevard, Suite 500, Redwood City, CA 94063 | (888) 274-0929 Measured per message. Latency Thursday, September 20, 12
50. Wildfire Interactive, Inc. | 1600 Seaport Boulevard, Suite 500, Redwood City, CA 94063 | (888) 274-0929 Measured at specific points Throughput Thursday, September 20, 12
51. Wildfire Interactive, Inc. | 1600 Seaport Boulevard, Suite 500, Redwood City, CA 94063 | (888) 274-0929 ZeroMQ optimizations Dedicated I/O thread I/O model suits scripting languages Syscalls and allocations etc. Thursday, September 20, 12
52. Wildfire Interactive, Inc. | 1600 Seaport Boulevard, Suite 500, Redwood City, CA 94063 | (888) 274-0929 Opportunistic batching Latency > TCP Throughput > TCP Avoid the network stack Scales to any pipe size Thursday, September 20, 12
53. Wildfire Interactive, Inc. | 1600 Seaport Boulevard, Suite 500, Redwood City, CA 94063 | (888) 274-0929 gem install rbczmq (rzmq || ffi-rzmq) Rewby. Thursday, September 20, 12
54. Wildfire Interactive, Inc. | 1600 Seaport Boulevard, Suite 500, Redwood City, CA 94063 | (888) 274-0929 ctx = ZMQ::Context.new(1) rep = ctx.bind(:REP, “inproc://p” req = ctx.connect(:REQ, “inproc://p” req.send(“message”) rep.receive(“message”) ctx.destroy Context - sockets state Thursday, September 20, 12
55. Wildfire Interactive, Inc. | 1600 Seaport Boulevard, Suite 500, Redwood City, CA 94063 | (888) 274-0929 ctx = ZMQ::Context.new(1) sub = ctx.socket(:SUB) sub.subscribe(“APPLE”) sub.connect(“tcp://0.0.0.0:5000”) sub.connect(“ipc://other.pub”) loop do process(sub.recv) end Initialize a socket Thursday, September 20, 12
56. Wildfire Interactive, Inc. | 1600 Seaport Boulevard, Suite 500, Redwood City, CA 94063 | (888) 274-0929 ZeroMQ concurrency PUSH PULL PULL PULL MAIN THREAD THREAD 1 THREAD 2 THREAD 3 I/O THREAD Thursday, September 20, 12
57. Wildfire Interactive, Inc. | 1600 Seaport Boulevard, Suite 500, Redwood City, CA 94063 | (888) 274-0929 ctx = ZMQ::Context.new s = ctx.bind(:PUSH, "inproc://work") 3.times{ ws << spawn_worker(ctx) } loop do msg = work.shift break if msg == “exit” s.send(msg) end Master (work distributor) Thursday, September 20, 12
58. Wildfire Interactive, Inc. | 1600 Seaport Boulevard, Suite 500, Redwood City, CA 94063 | (888) 274-0929 Thread.new do s = ctx.connect(:PULL, “inproc:// work”) loop do msg = s.recv break if msg == "quit" process(msg) end end Workers (consumer threads) Thursday, September 20, 12
59. Wildfire Interactive, Inc. | 1600 Seaport Boulevard, Suite 500, Redwood City, CA 94063 | (888) 274-0929 Mongrel2 Case study Thursday, September 20, 12
60. Wildfire Interactive, Inc. | 1600 Seaport Boulevard, Suite 500, Redwood City, CA 94063 | (888) 274-0929 How it works Multi-protocol ZeroMQ for backends Async backend requests Language agnostic (bindings) Thursday, September 20, 12
61. Wildfire Interactive, Inc. | 1600 Seaport Boulevard, Suite 500, Redwood City, CA 94063 | (888) 274-0929 Mongrel2 topology PULL PUB PULL PUB PUSH SUB BACKEND A BACKEND B MONGREL2 CLIENT Thursday, September 20, 12
62. Wildfire Interactive, Inc. | 1600 Seaport Boulevard, Suite 500, Redwood City, CA 94063 | (888) 274-0929 Best practices Patterns - learn them Stable bind() Unstable connect() Interjection principle Set water marks Read http://zguide.zeromq.org Thursday, September 20, 12
63. Wildfire Interactive, Inc. | 1600 Seaport Boulevard, Suite 500, Redwood City, CA 94063 | (888) 274-0929 Questions ? (... or sync up after) O fim fork @methodmissing follow @methodmissing Thursday, September 20, 12