ZeroMQ: scriptable sockets

ZeroMQ: scriptable sockets

Ruby Shift 2012, Kyiv, Ukraine

28314d64ae18c43f6b5428dc00a48e6e?s=128

Lourens Naudé

September 30, 2012
Tweet

Transcript

  1. 1.

    Wildfire Interactive, Inc. | 1600 Seaport Boulevard, Suite 500, Redwood

    City, CA 94063 | (888) 274-0929 ZeroMQ: scriptable sockets Lourens Naudé Kyiv Sunday, September 30, 12
  2. 2.

    Wildfire Interactive, Inc. | 1600 Seaport Boulevard, Suite 500, Redwood

    City, CA 94063 | (888) 274-0929 Ops @ WildfireApp Sunday, September 30, 12
  3. 3.

    Wildfire Interactive, Inc. | 1600 Seaport Boulevard, Suite 500, Redwood

    City, CA 94063 | (888) 274-0929 In 2012 it’s still too “expensive” to create and maintain integration points. Why this talk ? Sunday, September 30, 12
  4. 4.

    Wildfire Interactive, Inc. | 1600 Seaport Boulevard, Suite 500, Redwood

    City, CA 94063 | (888) 274-0929 Agenda What it is and isn’t Messages and patterns Transports and devices Resiliency Rewby Sunday, September 30, 12
  5. 5.

    Wildfire Interactive, Inc. | 1600 Seaport Boulevard, Suite 500, Redwood

    City, CA 94063 | (888) 274-0929 What’s wrong with socket I/O ? Sunday, September 30, 12
  6. 6.

    Wildfire Interactive, Inc. | 1600 Seaport Boulevard, Suite 500, Redwood

    City, CA 94063 | (888) 274-0929 Socket I/O problems Strong dev profile required Portability Flexibility Sunday, September 30, 12
  7. 7.

    Wildfire Interactive, Inc. | 1600 Seaport Boulevard, Suite 500, Redwood

    City, CA 94063 | (888) 274-0929 Zero broker, admin, cost, waste, price tag and (almost) zero latency What is ZeroMQ ? Sunday, September 30, 12
  8. 8.

    Wildfire Interactive, Inc. | 1600 Seaport Boulevard, Suite 500, Redwood

    City, CA 94063 | (888) 274-0929 Turnkey message queue solution. What it isn’t Sunday, September 30, 12
  9. 9.

    Wildfire Interactive, Inc. | 1600 Seaport Boulevard, Suite 500, Redwood

    City, CA 94063 | (888) 274-0929 /* Copyright (c) 2007-2012 iMatix Corporation Copyright (c) 2009-2011 250bpm s.r.o. Copyright (c) 2011 VMware, Inc. Copyright (c) 2007-2011 Other contributors as noted in the AUTHORS file However ... Sunday, September 30, 12
  10. 10.

    Wildfire Interactive, Inc. | 1600 Seaport Boulevard, Suite 500, Redwood

    City, CA 94063 | (888) 274-0929 IM for apps Not a server or message queue Communication / concurrency library Messaging semantics for sockets BSD sockets API Async by design Sunday, September 30, 12
  11. 11.

    Wildfire Interactive, Inc. | 1600 Seaport Boulevard, Suite 500, Redwood

    City, CA 94063 | (888) 274-0929 How did we get here ? Sunday, September 30, 12
  12. 12.

    Wildfire Interactive, Inc. | 1600 Seaport Boulevard, Suite 500, Redwood

    City, CA 94063 | (888) 274-0929 No really ... Martin Sustrik + Pieter Hintjens A lot AMQP isn’t Messages, not byte streams Financial services - optimization iMatix && community aka Crossroads I/O (fork) Sunday, September 30, 12
  13. 13.

    Wildfire Interactive, Inc. | 1600 Seaport Boulevard, Suite 500, Redwood

    City, CA 94063 | (888) 274-0929 Rich bindings ecosystem 30+ language bindings ... Sunday, September 30, 12
  14. 14.

    Wildfire Interactive, Inc. | 1600 Seaport Boulevard, Suite 500, Redwood

    City, CA 94063 | (888) 274-0929 In the wild ... and financial industry + startups Sunday, September 30, 12
  15. 15.

    Wildfire Interactive, Inc. | 1600 Seaport Boulevard, Suite 500, Redwood

    City, CA 94063 | (888) 274-0929 Super (scriptable) sockets What is a ZeroMQ socket ? Sunday, September 30, 12
  16. 16.

    Wildfire Interactive, Inc. | 1600 Seaport Boulevard, Suite 500, Redwood

    City, CA 94063 | (888) 274-0929 How ? An API model Embeddable servers Multiple transports Socket metaphor Familiar BSD socket API Multiplexing, framing etc. Sunday, September 30, 12
  17. 17.

    Wildfire Interactive, Inc. | 1600 Seaport Boulevard, Suite 500, Redwood

    City, CA 94063 | (888) 274-0929 bind() or connect() to multiple endpoints, simultaneously Multiple endpoints Sunday, September 30, 12
  18. 18.

    Wildfire Interactive, Inc. | 1600 Seaport Boulevard, Suite 500, Redwood

    City, CA 94063 | (888) 274-0929 bind() or connect() in any order Async connect Sunday, September 30, 12
  19. 19.

    Wildfire Interactive, Inc. | 1600 Seaport Boulevard, Suite 500, Redwood

    City, CA 94063 | (888) 274-0929 Socket types PUB SUB REQ REP PUSH PULL Sunday, September 30, 12
  20. 20.

    Wildfire Interactive, Inc. | 1600 Seaport Boulevard, Suite 500, Redwood

    City, CA 94063 | (888) 274-0929 Touching the water PUSH PULL PUSH PULL PULL PUSH PULL PUSH VENTILATOR SINK Sunday, September 30, 12
  21. 21.

    Wildfire Interactive, Inc. | 1600 Seaport Boulevard, Suite 500, Redwood

    City, CA 94063 | (888) 274-0929 PUSH PULL PUSH PULL PULL PUSH PULL PUSH VENTILATOR SINK PUSH PULL Sunday, September 30, 12
  22. 22.

    Wildfire Interactive, Inc. | 1600 Seaport Boulevard, Suite 500, Redwood

    City, CA 94063 | (888) 274-0929 Opaque binary data (blobs) Messages Sunday, September 30, 12
  23. 23.

    Wildfire Interactive, Inc. | 1600 Seaport Boulevard, Suite 500, Redwood

    City, CA 94063 | (888) 274-0929 Message format Framed wire protocol No contracts Serialize as you wish No auth / security Sunday, September 30, 12
  24. 24.

    Wildfire Interactive, Inc. | 1600 Seaport Boulevard, Suite 500, Redwood

    City, CA 94063 | (888) 274-0929 Message delivery Multipart messages supported Send 5GB if you like Atomic delivery NO PERSISTENCE Sunday, September 30, 12
  25. 25.

    Wildfire Interactive, Inc. | 1600 Seaport Boulevard, Suite 500, Redwood

    City, CA 94063 | (888) 274-0929 ENOTAMQ Sunday, September 30, 12
  26. 26.

    Wildfire Interactive, Inc. | 1600 Seaport Boulevard, Suite 500, Redwood

    City, CA 94063 | (888) 274-0929 Request / Reply Supported patterns Sunday, September 30, 12
  27. 27.

    Wildfire Interactive, Inc. | 1600 Seaport Boulevard, Suite 500, Redwood

    City, CA 94063 | (888) 274-0929 Request / reply REQ REP REQ REP REP REP REQ REQ Sunday, September 30, 12
  28. 28.

    Wildfire Interactive, Inc. | 1600 Seaport Boulevard, Suite 500, Redwood

    City, CA 94063 | (888) 274-0929 Request / Reply usage REQ socket load balances Lock step Bi-directional Custom routing Downstream topology flow control Sunday, September 30, 12
  29. 29.

    Wildfire Interactive, Inc. | 1600 Seaport Boulevard, Suite 500, Redwood

    City, CA 94063 | (888) 274-0929 Publish / subscribe Supported patterns Sunday, September 30, 12
  30. 30.

    Wildfire Interactive, Inc. | 1600 Seaport Boulevard, Suite 500, Redwood

    City, CA 94063 | (888) 274-0929 Publish / subscribe PUB SUB SUB SUB Sunday, September 30, 12
  31. 31.

    Wildfire Interactive, Inc. | 1600 Seaport Boulevard, Suite 500, Redwood

    City, CA 94063 | (888) 274-0929 Logging SUB PUB PUB PUB Sunday, September 30, 12
  32. 32.

    Wildfire Interactive, Inc. | 1600 Seaport Boulevard, Suite 500, Redwood

    City, CA 94063 | (888) 274-0929 Publish / subscribe usage Market data, game state etc. PUB socket distributes SUB socket filters Subscription forwarding (3.x) Sunday, September 30, 12
  33. 33.

    Wildfire Interactive, Inc. | 1600 Seaport Boulevard, Suite 500, Redwood

    City, CA 94063 | (888) 274-0929 Pipeline Supported patterns Sunday, September 30, 12
  34. 34.

    Wildfire Interactive, Inc. | 1600 Seaport Boulevard, Suite 500, Redwood

    City, CA 94063 | (888) 274-0929 Pipeline PUSH PULL PULL PULL Sunday, September 30, 12
  35. 35.

    Wildfire Interactive, Inc. | 1600 Seaport Boulevard, Suite 500, Redwood

    City, CA 94063 | (888) 274-0929 Pipeline usage Like REQ/REP, downstream reply Extendable - UNIX pipes Round-robin distribution Tasks with variable processing Sunday, September 30, 12
  36. 36.

    Wildfire Interactive, Inc. | 1600 Seaport Boulevard, Suite 500, Redwood

    City, CA 94063 | (888) 274-0929 Transport agnostic messaging Transports Sunday, September 30, 12
  37. 37.

    Wildfire Interactive, Inc. | 1600 Seaport Boulevard, Suite 500, Redwood

    City, CA 94063 | (888) 274-0929 Supported transports inproc:// - threads in a process ipc:// - processes on a box tcp:// - nodes in a network Uniform API - it just works Sunday, September 30, 12
  38. 38.

    Wildfire Interactive, Inc. | 1600 Seaport Boulevard, Suite 500, Redwood

    City, CA 94063 | (888) 274-0929 Lightweight switches Devices Sunday, September 30, 12
  39. 39.

    Wildfire Interactive, Inc. | 1600 Seaport Boulevard, Suite 500, Redwood

    City, CA 94063 | (888) 274-0929 eg. pub/sub persistence SUB PUB PUB SUB SUB Sunday, September 30, 12
  40. 40.

    Wildfire Interactive, Inc. | 1600 Seaport Boulevard, Suite 500, Redwood

    City, CA 94063 | (888) 274-0929 Device main loop f = ctx.connect(:SUB, "tcp:// 0.0.0.0:5110") b = ctx.bind(:PUB, “tcp:// 0.0.0.0:5111”) loop do m = f.recv process(m) b.send(m) end Sunday, September 30, 12
  41. 41.

    Wildfire Interactive, Inc. | 1600 Seaport Boulevard, Suite 500, Redwood

    City, CA 94063 | (888) 274-0929 Uses Proxy Routing (LRU) Filtering / de-dup Translation Persistence Sunday, September 30, 12
  42. 42.

    Wildfire Interactive, Inc. | 1600 Seaport Boulevard, Suite 500, Redwood

    City, CA 94063 | (888) 274-0929 Interjection Principle Scalable topologies Sunday, September 30, 12
  43. 43.

    Wildfire Interactive, Inc. | 1600 Seaport Boulevard, Suite 500, Redwood

    City, CA 94063 | (888) 274-0929 eg. saving bandwidth SUB PUB PUB SUB SUB SUB SUB Sunday, September 30, 12
  44. 44.

    Wildfire Interactive, Inc. | 1600 Seaport Boulevard, Suite 500, Redwood

    City, CA 94063 | (888) 274-0929 Asynchronous external dependencies. Resiliency Sunday, September 30, 12
  45. 45.

    Wildfire Interactive, Inc. | 1600 Seaport Boulevard, Suite 500, Redwood

    City, CA 94063 | (888) 274-0929 ZeroMQ resiliency Atomic message delivery Delayed retry on failure Multi bind() and connect() Unordered bind() or connect() SOA - easy upgrades Sunday, September 30, 12
  46. 46.

    Wildfire Interactive, Inc. | 1600 Seaport Boulevard, Suite 500, Redwood

    City, CA 94063 | (888) 274-0929 Water marks Throttling / flow control High Water Mark (HWM) Socket option DROPS - REP, PUB, SUB BLOCKS - REQ, PUSH Sunday, September 30, 12
  47. 47.

    Wildfire Interactive, Inc. | 1600 Seaport Boulevard, Suite 500, Redwood

    City, CA 94063 | (888) 274-0929 99% of deployments don’t need enterprise reliability Of reliability Sunday, September 30, 12
  48. 48.

    Wildfire Interactive, Inc. | 1600 Seaport Boulevard, Suite 500, Redwood

    City, CA 94063 | (888) 274-0929 Then ... Costly transactions Small topology Large ops team Little churn Sunday, September 30, 12
  49. 49.

    Wildfire Interactive, Inc. | 1600 Seaport Boulevard, Suite 500, Redwood

    City, CA 94063 | (888) 274-0929 ... now Negligible transaction cost Large topology Small ops team Rapid churn rate Sunday, September 30, 12
  50. 50.

    Wildfire Interactive, Inc. | 1600 Seaport Boulevard, Suite 500, Redwood

    City, CA 94063 | (888) 274-0929 Reliability as patterns Lazy Pirate (REQ / REP) Majordomo (queue) Titanic (persistence) Binary Star (High Availability) See http://zguide.zeromq.org Sunday, September 30, 12
  51. 51.

    Wildfire Interactive, Inc. | 1600 Seaport Boulevard, Suite 500, Redwood

    City, CA 94063 | (888) 274-0929 Low latency, high throughput. Performance Sunday, September 30, 12
  52. 52.

    Wildfire Interactive, Inc. | 1600 Seaport Boulevard, Suite 500, Redwood

    City, CA 94063 | (888) 274-0929 Lab: 6 000 000 msgs/s Ruby: 200 000 msgs/s (16 byte payload) Numbers Sunday, September 30, 12
  53. 53.

    Wildfire Interactive, Inc. | 1600 Seaport Boulevard, Suite 500, Redwood

    City, CA 94063 | (888) 274-0929 Measured per message. Latency Sunday, September 30, 12
  54. 54.

    Wildfire Interactive, Inc. | 1600 Seaport Boulevard, Suite 500, Redwood

    City, CA 94063 | (888) 274-0929 Measured at specific points Throughput Sunday, September 30, 12
  55. 55.

    Wildfire Interactive, Inc. | 1600 Seaport Boulevard, Suite 500, Redwood

    City, CA 94063 | (888) 274-0929 ZeroMQ optimizations Dedicated I/O thread I/O model suits scripting languages Syscalls and allocations etc. Sunday, September 30, 12
  56. 56.

    Wildfire Interactive, Inc. | 1600 Seaport Boulevard, Suite 500, Redwood

    City, CA 94063 | (888) 274-0929 Opportunistic batching Latency > TCP Throughput > TCP Avoid the network stack Scales to any pipe size Sunday, September 30, 12
  57. 57.

    Wildfire Interactive, Inc. | 1600 Seaport Boulevard, Suite 500, Redwood

    City, CA 94063 | (888) 274-0929 gem install rbczmq (rzmq || ffi-rzmq) Rewby. Sunday, September 30, 12
  58. 58.

    Wildfire Interactive, Inc. | 1600 Seaport Boulevard, Suite 500, Redwood

    City, CA 94063 | (888) 274-0929 ctx = ZMQ::Context.new(1) rep = ctx.bind(:REP, “inproc://p” req = ctx.connect(:REQ, “inproc://p” req.send(“message”) rep.receive(“message”) ctx.destroy Context - sockets state Sunday, September 30, 12
  59. 59.

    Wildfire Interactive, Inc. | 1600 Seaport Boulevard, Suite 500, Redwood

    City, CA 94063 | (888) 274-0929 ctx = ZMQ::Context.new(1) sub = ctx.socket(:SUB) sub.subscribe(“APPLE”) sub.connect(“tcp://0.0.0.0:5000”) sub.connect(“ipc://other.pub”) loop do process(sub.recv) end Initialize a socket Sunday, September 30, 12
  60. 60.

    Wildfire Interactive, Inc. | 1600 Seaport Boulevard, Suite 500, Redwood

    City, CA 94063 | (888) 274-0929 ZeroMQ concurrency PUSH PULL PULL PULL MAIN THREAD THREAD 1 THREAD 2 THREAD 3 I/O THREAD Sunday, September 30, 12
  61. 61.

    Wildfire Interactive, Inc. | 1600 Seaport Boulevard, Suite 500, Redwood

    City, CA 94063 | (888) 274-0929 ctx = ZMQ::Context.new s = ctx.bind(:PUSH, "inproc://work") 3.times{ ws << spawn_worker(ctx) } loop do msg = work.shift break if msg == “exit” s.send(msg) end Master (work distributor) Sunday, September 30, 12
  62. 62.

    Wildfire Interactive, Inc. | 1600 Seaport Boulevard, Suite 500, Redwood

    City, CA 94063 | (888) 274-0929 Thread.new do s = ctx.connect(:PULL, “inproc:// work”) loop do msg = s.recv break if msg == "quit" process(msg) end end Workers (consumer threads) Sunday, September 30, 12
  63. 63.

    Wildfire Interactive, Inc. | 1600 Seaport Boulevard, Suite 500, Redwood

    City, CA 94063 | (888) 274-0929 Mongrel2 Case study Sunday, September 30, 12
  64. 64.

    Wildfire Interactive, Inc. | 1600 Seaport Boulevard, Suite 500, Redwood

    City, CA 94063 | (888) 274-0929 How it works Multi-protocol ZeroMQ for backends Async backend requests Language agnostic (bindings) Sunday, September 30, 12
  65. 65.

    Wildfire Interactive, Inc. | 1600 Seaport Boulevard, Suite 500, Redwood

    City, CA 94063 | (888) 274-0929 Mongrel2 topology PULL PUB PULL PUB PUSH SUB BACKEND A BACKEND B MONGREL2 CLIENT Sunday, September 30, 12
  66. 66.

    Wildfire Interactive, Inc. | 1600 Seaport Boulevard, Suite 500, Redwood

    City, CA 94063 | (888) 274-0929 Best practices Patterns - learn them Stable bind() Unstable connect() Interjection principle Set water marks Read http://zguide.zeromq.org Sunday, September 30, 12
  67. 67.

    Wildfire Interactive, Inc. | 1600 Seaport Boulevard, Suite 500, Redwood

    City, CA 94063 | (888) 274-0929 The Future UDP, DNS transports etc. Scalability Protocol (Linux Kernel) Mobile / embedded Monitoring support (3.2) Directory service ? Sunday, September 30, 12
  68. 68.

    Wildfire Interactive, Inc. | 1600 Seaport Boulevard, Suite 500, Redwood

    City, CA 94063 | (888) 274-0929 Questions ? (... or sync up after) O fim fork @methodmissing follow @methodmissing Sunday, September 30, 12