ZeroMQ: Scriptable Sockets

ZeroMQ is a socket abstraction and concurrency framework that's changing the way we think and reason about distributed systems. Mailboxes, atomic message delivery and swappable transports allow for fast, flexible and resilient network topologies. Its I/O model also sits very well with all Ruby implementations. In this talk we'll discuss:

* What's wrong with socket I/O ?
* Supported messaging patterns
* Transport agnostic messaging
* Resiliency (operations and upgrades)
* Building out topologies just in time (interjection principle)
* Performance and throughput
* The Mongrel2 Ruby adapter
* How to use it from Ruby
* Small case study

Transcript

1. Wildfire Interactive, Inc. | 1600 Seaport Boulevard, Suite 500, Redwood City, CA 94063 | (888) 274-0929 ZeroMQ: scriptable sockets Lourens Naudé European Ruby Camp 2012 Berlin Sábado, 18 de Agosto de 12
2. Wildfire Interactive, Inc. | 1600 Seaport Boulevard, Suite 500, Redwood City, CA 94063 | (888) 274-0929 In 2012 it’s still too “expensive” to create and maintain integration points. Why this talk ? Sábado, 18 de Agosto de 12
3. Wildfire Interactive, Inc. | 1600 Seaport Boulevard, Suite 500, Redwood City, CA 94063 | (888) 274-0929 Agenda 35k feet view Messages and patterns Transports and devices Resiliency Rewby Sábado, 18 de Agosto de 12
4. Wildfire Interactive, Inc. | 1600 Seaport Boulevard, Suite 500, Redwood City, CA 94063 | (888) 274-0929 What’s wrong with socket I/O ? Sábado, 18 de Agosto de 12
5. Wildfire Interactive, Inc. | 1600 Seaport Boulevard, Suite 500, Redwood City, CA 94063 | (888) 274-0929 Skills Strong dev profile required Multiplexed I/O knowledge Understand protocols Buffer management Sábado, 18 de Agosto de 12
6. Wildfire Interactive, Inc. | 1600 Seaport Boulevard, Suite 500, Redwood City, CA 94063 | (888) 274-0929 Not portable Standards compliance, meh Socket options Error codes Ruby abstracts most of these Sábado, 18 de Agosto de 12
7. Wildfire Interactive, Inc. | 1600 Seaport Boulevard, Suite 500, Redwood City, CA 94063 | (888) 274-0929 Not flexible 2 patterns Bi-directional or multicast Lock-step connection Fixed network topology Sábado, 18 de Agosto de 12
8. Wildfire Interactive, Inc. | 1600 Seaport Boulevard, Suite 500, Redwood City, CA 94063 | (888) 274-0929 Zero broker, admin, cost, waste, price tag and (almost) zero latency What is ZeroMQ ? Sábado, 18 de Agosto de 12
9. Wildfire Interactive, Inc. | 1600 Seaport Boulevard, Suite 500, Redwood City, CA 94063 | (888) 274-0929 IM for apps Not a server or message queue Communication / concurrency library Messaging semantics for sockets BSD sockets API Async by design Sábado, 18 de Agosto de 12
10. Wildfire Interactive, Inc. | 1600 Seaport Boulevard, Suite 500, Redwood City, CA 94063 | (888) 274-0929 How did we get here ? Sábado, 18 de Agosto de 12
11. Wildfire Interactive, Inc. | 1600 Seaport Boulevard, Suite 500, Redwood City, CA 94063 | (888) 274-0929 No really ... Martin Sustrik + Pieter Hintjens Everything AMQP isn’t Messages, not byte streams Financial services - optimization iMatix && community aka Crossroads I/O (fork) Sábado, 18 de Agosto de 12
12. Wildfire Interactive, Inc. | 1600 Seaport Boulevard, Suite 500, Redwood City, CA 94063 | (888) 274-0929 Rich bindings ecosystem 30+ language bindings ... Sábado, 18 de Agosto de 12
13. Wildfire Interactive, Inc. | 1600 Seaport Boulevard, Suite 500, Redwood City, CA 94063 | (888) 274-0929 In the wild ... and financial industry + startups Sábado, 18 de Agosto de 12
14. Wildfire Interactive, Inc. | 1600 Seaport Boulevard, Suite 500, Redwood City, CA 94063 | (888) 274-0929 Super (scriptable) sockets What is a ZeroMQ socket ? Sábado, 18 de Agosto de 12
15. Wildfire Interactive, Inc. | 1600 Seaport Boulevard, Suite 500, Redwood City, CA 94063 | (888) 274-0929 How ? An API model Embeddable servers Multiple transports Socket metaphor Familiar BSD socket API Multiplexing, framing etc. Sábado, 18 de Agosto de 12
16. Wildfire Interactive, Inc. | 1600 Seaport Boulevard, Suite 500, Redwood City, CA 94063 | (888) 274-0929 bind() or connect() to multiple endpoints, simultaneously Multiple endpoints Sábado, 18 de Agosto de 12
17. Wildfire Interactive, Inc. | 1600 Seaport Boulevard, Suite 500, Redwood City, CA 94063 | (888) 274-0929 bind() or connect() in any order Async connect Sábado, 18 de Agosto de 12
18. Wildfire Interactive, Inc. | 1600 Seaport Boulevard, Suite 500, Redwood City, CA 94063 | (888) 274-0929 Socket types PUB SUB REQ REP PUSH PULL Sábado, 18 de Agosto de 12
19. Wildfire Interactive, Inc. | 1600 Seaport Boulevard, Suite 500, Redwood City, CA 94063 | (888) 274-0929 Touching the water PUSH PULL PUSH PULL PULL PUSH PULL PUSH VENTILATOR SINK Sábado, 18 de Agosto de 12
20. Wildfire Interactive, Inc. | 1600 Seaport Boulevard, Suite 500, Redwood City, CA 94063 | (888) 274-0929 Opaque binary data (blobs) Messages Sábado, 18 de Agosto de 12
21. Wildfire Interactive, Inc. | 1600 Seaport Boulevard, Suite 500, Redwood City, CA 94063 | (888) 274-0929 Message format Framed wire protocol No contracts Serialize as you wish No auth / security Sábado, 18 de Agosto de 12
22. Wildfire Interactive, Inc. | 1600 Seaport Boulevard, Suite 500, Redwood City, CA 94063 | (888) 274-0929 Message delivery Multipart messages supported Send 5GB if you like Atomic delivery NO PERSISTENCE Sábado, 18 de Agosto de 12
23. Wildfire Interactive, Inc. | 1600 Seaport Boulevard, Suite 500, Redwood City, CA 94063 | (888) 274-0929 ENOTAMQ Sábado, 18 de Agosto de 12
24. Wildfire Interactive, Inc. | 1600 Seaport Boulevard, Suite 500, Redwood City, CA 94063 | (888) 274-0929 Request / Reply Supported patterns Sábado, 18 de Agosto de 12
25. Wildfire Interactive, Inc. | 1600 Seaport Boulevard, Suite 500, Redwood City, CA 94063 | (888) 274-0929 Request / reply REQ REP REQ REP REP REP REQ REQ Sábado, 18 de Agosto de 12
26. Wildfire Interactive, Inc. | 1600 Seaport Boulevard, Suite 500, Redwood City, CA 94063 | (888) 274-0929 Request / Reply usage REQ socket load balances Lock step Bi-directional Custom routing Downstream topology flow control Sábado, 18 de Agosto de 12
27. Wildfire Interactive, Inc. | 1600 Seaport Boulevard, Suite 500, Redwood City, CA 94063 | (888) 274-0929 Publish / subscribe Supported patterns Sábado, 18 de Agosto de 12
28. Wildfire Interactive, Inc. | 1600 Seaport Boulevard, Suite 500, Redwood City, CA 94063 | (888) 274-0929 Publish / subscribe PUB SUB SUB SUB Sábado, 18 de Agosto de 12
29. Wildfire Interactive, Inc. | 1600 Seaport Boulevard, Suite 500, Redwood City, CA 94063 | (888) 274-0929 Logging SUB PUB PUB PUB Sábado, 18 de Agosto de 12
30. Wildfire Interactive, Inc. | 1600 Seaport Boulevard, Suite 500, Redwood City, CA 94063 | (888) 274-0929 Publish / subscribe usage Market data, game state etc. PUB socket distributes SUB socket filters Subscription forwarding (3.x) Sábado, 18 de Agosto de 12
31. Wildfire Interactive, Inc. | 1600 Seaport Boulevard, Suite 500, Redwood City, CA 94063 | (888) 274-0929 Pipeline Supported patterns Sábado, 18 de Agosto de 12
32. Wildfire Interactive, Inc. | 1600 Seaport Boulevard, Suite 500, Redwood City, CA 94063 | (888) 274-0929 Pipeline PUSH PULL PULL PULL Sábado, 18 de Agosto de 12
33. Wildfire Interactive, Inc. | 1600 Seaport Boulevard, Suite 500, Redwood City, CA 94063 | (888) 274-0929 Pipeline usage Like REQ/REP, downstream reply Extendable - UNIX pipes Round-robin distribution Tasks with variable processing Sábado, 18 de Agosto de 12
34. Wildfire Interactive, Inc. | 1600 Seaport Boulevard, Suite 500, Redwood City, CA 94063 | (888) 274-0929 Parallel pipeline PUSH PULL PUSH PULL PULL PUSH PULL PUSH VENTILATOR SINK Sábado, 18 de Agosto de 12
35. Wildfire Interactive, Inc. | 1600 Seaport Boulevard, Suite 500, Redwood City, CA 94063 | (888) 274-0929 Transport agnostic messaging Transports Sábado, 18 de Agosto de 12
36. Wildfire Interactive, Inc. | 1600 Seaport Boulevard, Suite 500, Redwood City, CA 94063 | (888) 274-0929 Supported transports inproc:// - threads in a process ipc:// - processes on a box tcp:// - nodes in a network Uniform API - it just works Sábado, 18 de Agosto de 12
37. Wildfire Interactive, Inc. | 1600 Seaport Boulevard, Suite 500, Redwood City, CA 94063 | (888) 274-0929 Lightweight switches Devices Sábado, 18 de Agosto de 12
38. Wildfire Interactive, Inc. | 1600 Seaport Boulevard, Suite 500, Redwood City, CA 94063 | (888) 274-0929 eg. pub/sub persistence SUB PUB PUB SUB SUB Sábado, 18 de Agosto de 12
39. Wildfire Interactive, Inc. | 1600 Seaport Boulevard, Suite 500, Redwood City, CA 94063 | (888) 274-0929 Device main loop f = ctx.connect(:SUB, "tcp:// 0.0.0.0:5110") b = ctx.bind(:PUB, “tcp:// 0.0.0.0:5111”) loop do m = f.recv process(m) b.send(m) end Sábado, 18 de Agosto de 12
40. Wildfire Interactive, Inc. | 1600 Seaport Boulevard, Suite 500, Redwood City, CA 94063 | (888) 274-0929 Uses Proxy Routing (LRU) Filtering / de-dup Translation Persistence Sábado, 18 de Agosto de 12
41. Wildfire Interactive, Inc. | 1600 Seaport Boulevard, Suite 500, Redwood City, CA 94063 | (888) 274-0929 Interjection Principle Scalable topologies Sábado, 18 de Agosto de 12
42. Wildfire Interactive, Inc. | 1600 Seaport Boulevard, Suite 500, Redwood City, CA 94063 | (888) 274-0929 eg. saving bandwidth SUB PUB PUB SUB SUB SUB SUB Sábado, 18 de Agosto de 12
43. Wildfire Interactive, Inc. | 1600 Seaport Boulevard, Suite 500, Redwood City, CA 94063 | (888) 274-0929 Asynchronous external dependencies. Resiliency Sábado, 18 de Agosto de 12
44. Wildfire Interactive, Inc. | 1600 Seaport Boulevard, Suite 500, Redwood City, CA 94063 | (888) 274-0929 ZeroMQ resiliency Atomic message delivery Delayed retry on failure Multi bind() and connect() Unordered bind() or connect() SOA - easy upgrades Sábado, 18 de Agosto de 12
45. Wildfire Interactive, Inc. | 1600 Seaport Boulevard, Suite 500, Redwood City, CA 94063 | (888) 274-0929 Water marks Throttling / flow control High Water Mark (HWM) Socket option DROPS - REP, PUB, SUB BLOCKS - REQ, PUSH Sábado, 18 de Agosto de 12
46. Wildfire Interactive, Inc. | 1600 Seaport Boulevard, Suite 500, Redwood City, CA 94063 | (888) 274-0929 99% of deployments don’t need enterprise reliability Of reliability Sábado, 18 de Agosto de 12
47. Wildfire Interactive, Inc. | 1600 Seaport Boulevard, Suite 500, Redwood City, CA 94063 | (888) 274-0929 Then ... Costly transactions Small topology Large ops team Little churn Sábado, 18 de Agosto de 12
48. Wildfire Interactive, Inc. | 1600 Seaport Boulevard, Suite 500, Redwood City, CA 94063 | (888) 274-0929 ... now Negligible transaction cost Large topology Small ops team Rapid churn rate Sábado, 18 de Agosto de 12
49. Wildfire Interactive, Inc. | 1600 Seaport Boulevard, Suite 500, Redwood City, CA 94063 | (888) 274-0929 Reliability as patterns Lazy Pirate (REQ / REP) Majordomo (queue) Titanic (persistence) Binary Star (High Availability) See http://zguide.zeromq.org Sábado, 18 de Agosto de 12
50. Wildfire Interactive, Inc. | 1600 Seaport Boulevard, Suite 500, Redwood City, CA 94063 | (888) 274-0929 http://zguide.zeromq.org Best documentation. Ever. Sábado, 18 de Agosto de 12
51. Wildfire Interactive, Inc. | 1600 Seaport Boulevard, Suite 500, Redwood City, CA 94063 | (888) 274-0929 Low latency, high throughput. Performance Sábado, 18 de Agosto de 12
52. Wildfire Interactive, Inc. | 1600 Seaport Boulevard, Suite 500, Redwood City, CA 94063 | (888) 274-0929 Lab: 6 000 000 msgs/s Ruby: 200 000 msgs/s (16 byte payload) Numbers Sábado, 18 de Agosto de 12
53. Wildfire Interactive, Inc. | 1600 Seaport Boulevard, Suite 500, Redwood City, CA 94063 | (888) 274-0929 Measured per message. Latency Sábado, 18 de Agosto de 12
54. Wildfire Interactive, Inc. | 1600 Seaport Boulevard, Suite 500, Redwood City, CA 94063 | (888) 274-0929 Measured at specific points Throughput Sábado, 18 de Agosto de 12
55. Wildfire Interactive, Inc. | 1600 Seaport Boulevard, Suite 500, Redwood City, CA 94063 | (888) 274-0929 ZeroMQ optimizations Dedicated I/O thread I/O model suits scripting languages Syscalls and allocations etc. Sábado, 18 de Agosto de 12
56. Wildfire Interactive, Inc. | 1600 Seaport Boulevard, Suite 500, Redwood City, CA 94063 | (888) 274-0929 Opportunistic batching Latency > TCP Throughput > TCP Avoid the network stack Scales to any pipe size Sábado, 18 de Agosto de 12
57. Wildfire Interactive, Inc. | 1600 Seaport Boulevard, Suite 500, Redwood City, CA 94063 | (888) 274-0929 gem install rbczmq Rewby. Sábado, 18 de Agosto de 12
58. Wildfire Interactive, Inc. | 1600 Seaport Boulevard, Suite 500, Redwood City, CA 94063 | (888) 274-0929 ctx = ZMQ::Context.new(1) rep = ctx.bind(:REP, “inproc://p” req = ctx.connect(:REQ, “inproc://p” req.send(“message”) rep.receive(“message”) ctx.destroy Context - sockets state Sábado, 18 de Agosto de 12
59. Wildfire Interactive, Inc. | 1600 Seaport Boulevard, Suite 500, Redwood City, CA 94063 | (888) 274-0929 ctx = ZMQ::Context.new(1) sub = ctx.socket(:SUB) sub.subscribe(“APPLE”) sub.connect(“tcp://0.0.0.0:5000”) sub.connect(“ipc://other.pub”) loop do process(sub.recv) end Initialize a socket Sábado, 18 de Agosto de 12
60. Wildfire Interactive, Inc. | 1600 Seaport Boulevard, Suite 500, Redwood City, CA 94063 | (888) 274-0929 ZeroMQ concurrency PUSH PULL PULL PULL MAIN THREAD THREAD 1 THREAD 2 THREAD 3 I/O THREAD Sábado, 18 de Agosto de 12
61. Wildfire Interactive, Inc. | 1600 Seaport Boulevard, Suite 500, Redwood City, CA 94063 | (888) 274-0929 ctx = ZMQ::Context.new s = ctx.bind(:PUSH, "inproc://work") 3.times{ ws << spawn_worker(ctx) } loop do msg = work.shift break if msg == “exit” s.send(msg) end Master (work distributor) Sábado, 18 de Agosto de 12
62. Wildfire Interactive, Inc. | 1600 Seaport Boulevard, Suite 500, Redwood City, CA 94063 | (888) 274-0929 Thread.new do s = ctx.connect(:PULL, “inproc:// work”) loop do msg = s.recv break if msg == "quit" process(msg) end end Workers (consumer threads) Sábado, 18 de Agosto de 12
63. Wildfire Interactive, Inc. | 1600 Seaport Boulevard, Suite 500, Redwood City, CA 94063 | (888) 274-0929 Mongrel2 Case study Sábado, 18 de Agosto de 12
64. Wildfire Interactive, Inc. | 1600 Seaport Boulevard, Suite 500, Redwood City, CA 94063 | (888) 274-0929 How it works Multi-protocol ZeroMQ for backends Async backend requests Language agnostic (bindings) Sábado, 18 de Agosto de 12
65. Wildfire Interactive, Inc. | 1600 Seaport Boulevard, Suite 500, Redwood City, CA 94063 | (888) 274-0929 Mongrel2 topology PULL PUB PULL PUB PUSH SUB BACKEND A BACKEND B MONGREL2 CLIENT Sábado, 18 de Agosto de 12
66. Wildfire Interactive, Inc. | 1600 Seaport Boulevard, Suite 500, Redwood City, CA 94063 | (888) 274-0929 Best practices Patterns - learn them Stable bind() Unstable connect() Interjection principle Set water marks Read http://zguide.zeromq.org Sábado, 18 de Agosto de 12
67. Wildfire Interactive, Inc. | 1600 Seaport Boulevard, Suite 500, Redwood City, CA 94063 | (888) 274-0929 The Future UDP, DNS transports etc. Scalability Protocol (Linux Kernel) Mobile / embedded Monitoring support (beta 3.2) Sábado, 18 de Agosto de 12
68. Wildfire Interactive, Inc. | 1600 Seaport Boulevard, Suite 500, Redwood City, CA 94063 | (888) 274-0929 Questions ? (or sync up after ?) O fim Sábado, 18 de Agosto de 12
69. Wildfire Interactive, Inc. | 1600 Seaport Boulevard, Suite 500, Redwood City, CA 94063 | (888) 274-0929 fork @methodmissing follow @methodmissing Thanks! Sábado, 18 de Agosto de 12