ZeroMQ: Scriptable Sockets

Transcript

1. Wildfire Interactive, Inc. | 1600 Seaport Boulevard, Suite 500, Redwood

   City, CA 94063 | (888) 274-0929 ZeroMQ: scriptable sockets Lourens Naudé European Ruby Camp 2012 Berlin Sábado, 18 de Agosto de 12

2. Wildfire Interactive, Inc. | 1600 Seaport Boulevard, Suite 500, Redwood

   City, CA 94063 | (888) 274-0929 In 2012 it’s still too “expensive” to create and maintain integration points. Why this talk ? Sábado, 18 de Agosto de 12

3. Wildfire Interactive, Inc. | 1600 Seaport Boulevard, Suite 500, Redwood

   City, CA 94063 | (888) 274-0929 Agenda 35k feet view Messages and patterns Transports and devices Resiliency Rewby Sábado, 18 de Agosto de 12

4. Wildfire Interactive, Inc. | 1600 Seaport Boulevard, Suite 500, Redwood

   City, CA 94063 | (888) 274-0929 What’s wrong with socket I/O ? Sábado, 18 de Agosto de 12

5. Wildfire Interactive, Inc. | 1600 Seaport Boulevard, Suite 500, Redwood

   City, CA 94063 | (888) 274-0929 Skills Strong dev profile required Multiplexed I/O knowledge Understand protocols Buffer management Sábado, 18 de Agosto de 12

6. Wildfire Interactive, Inc. | 1600 Seaport Boulevard, Suite 500, Redwood

   City, CA 94063 | (888) 274-0929 Not portable Standards compliance, meh Socket options Error codes Ruby abstracts most of these Sábado, 18 de Agosto de 12

7. Wildfire Interactive, Inc. | 1600 Seaport Boulevard, Suite 500, Redwood

   City, CA 94063 | (888) 274-0929 Not flexible 2 patterns Bi-directional or multicast Lock-step connection Fixed network topology Sábado, 18 de Agosto de 12

8. Wildfire Interactive, Inc. | 1600 Seaport Boulevard, Suite 500, Redwood

   City, CA 94063 | (888) 274-0929 Zero broker, admin, cost, waste, price tag and (almost) zero latency What is ZeroMQ ? Sábado, 18 de Agosto de 12

9. Wildfire Interactive, Inc. | 1600 Seaport Boulevard, Suite 500, Redwood

   City, CA 94063 | (888) 274-0929 IM for apps Not a server or message queue Communication / concurrency library Messaging semantics for sockets BSD sockets API Async by design Sábado, 18 de Agosto de 12

10. Wildfire Interactive, Inc. | 1600 Seaport Boulevard, Suite 500, Redwood

    City, CA 94063 | (888) 274-0929 How did we get here ? Sábado, 18 de Agosto de 12

11. Wildfire Interactive, Inc. | 1600 Seaport Boulevard, Suite 500, Redwood

    City, CA 94063 | (888) 274-0929 No really ... Martin Sustrik + Pieter Hintjens Everything AMQP isn’t Messages, not byte streams Financial services - optimization iMatix && community aka Crossroads I/O (fork) Sábado, 18 de Agosto de 12

12. Wildfire Interactive, Inc. | 1600 Seaport Boulevard, Suite 500, Redwood

    City, CA 94063 | (888) 274-0929 Rich bindings ecosystem 30+ language bindings ... Sábado, 18 de Agosto de 12

13. Wildfire Interactive, Inc. | 1600 Seaport Boulevard, Suite 500, Redwood

    City, CA 94063 | (888) 274-0929 In the wild ... and financial industry + startups Sábado, 18 de Agosto de 12

14. Wildfire Interactive, Inc. | 1600 Seaport Boulevard, Suite 500, Redwood

    City, CA 94063 | (888) 274-0929 Super (scriptable) sockets What is a ZeroMQ socket ? Sábado, 18 de Agosto de 12

15. Wildfire Interactive, Inc. | 1600 Seaport Boulevard, Suite 500, Redwood

    City, CA 94063 | (888) 274-0929 How ? An API model Embeddable servers Multiple transports Socket metaphor Familiar BSD socket API Multiplexing, framing etc. Sábado, 18 de Agosto de 12

16. Wildfire Interactive, Inc. | 1600 Seaport Boulevard, Suite 500, Redwood

    City, CA 94063 | (888) 274-0929 bind() or connect() to multiple endpoints, simultaneously Multiple endpoints Sábado, 18 de Agosto de 12

17. Wildfire Interactive, Inc. | 1600 Seaport Boulevard, Suite 500, Redwood

    City, CA 94063 | (888) 274-0929 bind() or connect() in any order Async connect Sábado, 18 de Agosto de 12

18. Wildfire Interactive, Inc. | 1600 Seaport Boulevard, Suite 500, Redwood

    City, CA 94063 | (888) 274-0929 Socket types PUB SUB REQ REP PUSH PULL Sábado, 18 de Agosto de 12

19. Wildfire Interactive, Inc. | 1600 Seaport Boulevard, Suite 500, Redwood

    City, CA 94063 | (888) 274-0929 Touching the water PUSH PULL PUSH PULL PULL PUSH PULL PUSH VENTILATOR SINK Sábado, 18 de Agosto de 12

20. Wildfire Interactive, Inc. | 1600 Seaport Boulevard, Suite 500, Redwood

    City, CA 94063 | (888) 274-0929 Opaque binary data (blobs) Messages Sábado, 18 de Agosto de 12

21. Wildfire Interactive, Inc. | 1600 Seaport Boulevard, Suite 500, Redwood

    City, CA 94063 | (888) 274-0929 Message format Framed wire protocol No contracts Serialize as you wish No auth / security Sábado, 18 de Agosto de 12

22. Wildfire Interactive, Inc. | 1600 Seaport Boulevard, Suite 500, Redwood

    City, CA 94063 | (888) 274-0929 Message delivery Multipart messages supported Send 5GB if you like Atomic delivery NO PERSISTENCE Sábado, 18 de Agosto de 12

23. Wildfire Interactive, Inc. | 1600 Seaport Boulevard, Suite 500, Redwood

    City, CA 94063 | (888) 274-0929 ENOTAMQ Sábado, 18 de Agosto de 12

24. Wildfire Interactive, Inc. | 1600 Seaport Boulevard, Suite 500, Redwood

    City, CA 94063 | (888) 274-0929 Request / Reply Supported patterns Sábado, 18 de Agosto de 12

25. Wildfire Interactive, Inc. | 1600 Seaport Boulevard, Suite 500, Redwood

    City, CA 94063 | (888) 274-0929 Request / reply REQ REP REQ REP REP REP REQ REQ Sábado, 18 de Agosto de 12

26. Wildfire Interactive, Inc. | 1600 Seaport Boulevard, Suite 500, Redwood

    City, CA 94063 | (888) 274-0929 Request / Reply usage REQ socket load balances Lock step Bi-directional Custom routing Downstream topology flow control Sábado, 18 de Agosto de 12

27. Wildfire Interactive, Inc. | 1600 Seaport Boulevard, Suite 500, Redwood

    City, CA 94063 | (888) 274-0929 Publish / subscribe Supported patterns Sábado, 18 de Agosto de 12

28. Wildfire Interactive, Inc. | 1600 Seaport Boulevard, Suite 500, Redwood

    City, CA 94063 | (888) 274-0929 Publish / subscribe PUB SUB SUB SUB Sábado, 18 de Agosto de 12

29. Wildfire Interactive, Inc. | 1600 Seaport Boulevard, Suite 500, Redwood

    City, CA 94063 | (888) 274-0929 Logging SUB PUB PUB PUB Sábado, 18 de Agosto de 12

30. Wildfire Interactive, Inc. | 1600 Seaport Boulevard, Suite 500, Redwood

    City, CA 94063 | (888) 274-0929 Publish / subscribe usage Market data, game state etc. PUB socket distributes SUB socket filters Subscription forwarding (3.x) Sábado, 18 de Agosto de 12

31. Wildfire Interactive, Inc. | 1600 Seaport Boulevard, Suite 500, Redwood

    City, CA 94063 | (888) 274-0929 Pipeline Supported patterns Sábado, 18 de Agosto de 12

32. Wildfire Interactive, Inc. | 1600 Seaport Boulevard, Suite 500, Redwood

    City, CA 94063 | (888) 274-0929 Pipeline PUSH PULL PULL PULL Sábado, 18 de Agosto de 12

33. Wildfire Interactive, Inc. | 1600 Seaport Boulevard, Suite 500, Redwood

    City, CA 94063 | (888) 274-0929 Pipeline usage Like REQ/REP, downstream reply Extendable - UNIX pipes Round-robin distribution Tasks with variable processing Sábado, 18 de Agosto de 12

34. Wildfire Interactive, Inc. | 1600 Seaport Boulevard, Suite 500, Redwood

    City, CA 94063 | (888) 274-0929 Parallel pipeline PUSH PULL PUSH PULL PULL PUSH PULL PUSH VENTILATOR SINK Sábado, 18 de Agosto de 12

35. Wildfire Interactive, Inc. | 1600 Seaport Boulevard, Suite 500, Redwood

    City, CA 94063 | (888) 274-0929 Transport agnostic messaging Transports Sábado, 18 de Agosto de 12

36. Wildfire Interactive, Inc. | 1600 Seaport Boulevard, Suite 500, Redwood

    City, CA 94063 | (888) 274-0929 Supported transports inproc:// - threads in a process ipc:// - processes on a box tcp:// - nodes in a network Uniform API - it just works Sábado, 18 de Agosto de 12

37. Wildfire Interactive, Inc. | 1600 Seaport Boulevard, Suite 500, Redwood

    City, CA 94063 | (888) 274-0929 Lightweight switches Devices Sábado, 18 de Agosto de 12

38. Wildfire Interactive, Inc. | 1600 Seaport Boulevard, Suite 500, Redwood

    City, CA 94063 | (888) 274-0929 eg. pub/sub persistence SUB PUB PUB SUB SUB Sábado, 18 de Agosto de 12

39. Wildfire Interactive, Inc. | 1600 Seaport Boulevard, Suite 500, Redwood

    City, CA 94063 | (888) 274-0929 Device main loop f = ctx.connect(:SUB, "tcp:// 0.0.0.0:5110") b = ctx.bind(:PUB, “tcp:// 0.0.0.0:5111”) loop do m = f.recv process(m) b.send(m) end Sábado, 18 de Agosto de 12

40. Wildfire Interactive, Inc. | 1600 Seaport Boulevard, Suite 500, Redwood

    City, CA 94063 | (888) 274-0929 Uses Proxy Routing (LRU) Filtering / de-dup Translation Persistence Sábado, 18 de Agosto de 12

41. Wildfire Interactive, Inc. | 1600 Seaport Boulevard, Suite 500, Redwood

    City, CA 94063 | (888) 274-0929 Interjection Principle Scalable topologies Sábado, 18 de Agosto de 12

42. Wildfire Interactive, Inc. | 1600 Seaport Boulevard, Suite 500, Redwood

    City, CA 94063 | (888) 274-0929 eg. saving bandwidth SUB PUB PUB SUB SUB SUB SUB Sábado, 18 de Agosto de 12

43. Wildfire Interactive, Inc. | 1600 Seaport Boulevard, Suite 500, Redwood

    City, CA 94063 | (888) 274-0929 Asynchronous external dependencies. Resiliency Sábado, 18 de Agosto de 12

44. Wildfire Interactive, Inc. | 1600 Seaport Boulevard, Suite 500, Redwood

    City, CA 94063 | (888) 274-0929 ZeroMQ resiliency Atomic message delivery Delayed retry on failure Multi bind() and connect() Unordered bind() or connect() SOA - easy upgrades Sábado, 18 de Agosto de 12

45. Wildfire Interactive, Inc. | 1600 Seaport Boulevard, Suite 500, Redwood

    City, CA 94063 | (888) 274-0929 Water marks Throttling / flow control High Water Mark (HWM) Socket option DROPS - REP, PUB, SUB BLOCKS - REQ, PUSH Sábado, 18 de Agosto de 12

46. Wildfire Interactive, Inc. | 1600 Seaport Boulevard, Suite 500, Redwood

    City, CA 94063 | (888) 274-0929 99% of deployments don’t need enterprise reliability Of reliability Sábado, 18 de Agosto de 12

47. Wildfire Interactive, Inc. | 1600 Seaport Boulevard, Suite 500, Redwood

    City, CA 94063 | (888) 274-0929 Then ... Costly transactions Small topology Large ops team Little churn Sábado, 18 de Agosto de 12

48. Wildfire Interactive, Inc. | 1600 Seaport Boulevard, Suite 500, Redwood

    City, CA 94063 | (888) 274-0929 ... now Negligible transaction cost Large topology Small ops team Rapid churn rate Sábado, 18 de Agosto de 12

49. Wildfire Interactive, Inc. | 1600 Seaport Boulevard, Suite 500, Redwood

    City, CA 94063 | (888) 274-0929 Reliability as patterns Lazy Pirate (REQ / REP) Majordomo (queue) Titanic (persistence) Binary Star (High Availability) See http://zguide.zeromq.org Sábado, 18 de Agosto de 12

50. Wildfire Interactive, Inc. | 1600 Seaport Boulevard, Suite 500, Redwood

    City, CA 94063 | (888) 274-0929 http://zguide.zeromq.org Best documentation. Ever. Sábado, 18 de Agosto de 12

51. Wildfire Interactive, Inc. | 1600 Seaport Boulevard, Suite 500, Redwood

    City, CA 94063 | (888) 274-0929 Low latency, high throughput. Performance Sábado, 18 de Agosto de 12

52. Wildfire Interactive, Inc. | 1600 Seaport Boulevard, Suite 500, Redwood

    City, CA 94063 | (888) 274-0929 Lab: 6 000 000 msgs/s Ruby: 200 000 msgs/s (16 byte payload) Numbers Sábado, 18 de Agosto de 12

53. Wildfire Interactive, Inc. | 1600 Seaport Boulevard, Suite 500, Redwood

    City, CA 94063 | (888) 274-0929 Measured per message. Latency Sábado, 18 de Agosto de 12

54. Wildfire Interactive, Inc. | 1600 Seaport Boulevard, Suite 500, Redwood

    City, CA 94063 | (888) 274-0929 Measured at specific points Throughput Sábado, 18 de Agosto de 12

55. Wildfire Interactive, Inc. | 1600 Seaport Boulevard, Suite 500, Redwood

    City, CA 94063 | (888) 274-0929 ZeroMQ optimizations Dedicated I/O thread I/O model suits scripting languages Syscalls and allocations etc. Sábado, 18 de Agosto de 12

56. Wildfire Interactive, Inc. | 1600 Seaport Boulevard, Suite 500, Redwood

    City, CA 94063 | (888) 274-0929 Opportunistic batching Latency > TCP Throughput > TCP Avoid the network stack Scales to any pipe size Sábado, 18 de Agosto de 12

57. Wildfire Interactive, Inc. | 1600 Seaport Boulevard, Suite 500, Redwood

    City, CA 94063 | (888) 274-0929 gem install rbczmq Rewby. Sábado, 18 de Agosto de 12

58. Wildfire Interactive, Inc. | 1600 Seaport Boulevard, Suite 500, Redwood

    City, CA 94063 | (888) 274-0929 ctx = ZMQ::Context.new(1) rep = ctx.bind(:REP, “inproc://p” req = ctx.connect(:REQ, “inproc://p” req.send(“message”) rep.receive(“message”) ctx.destroy Context - sockets state Sábado, 18 de Agosto de 12

59. Wildfire Interactive, Inc. | 1600 Seaport Boulevard, Suite 500, Redwood

    City, CA 94063 | (888) 274-0929 ctx = ZMQ::Context.new(1) sub = ctx.socket(:SUB) sub.subscribe(“APPLE”) sub.connect(“tcp://0.0.0.0:5000”) sub.connect(“ipc://other.pub”) loop do process(sub.recv) end Initialize a socket Sábado, 18 de Agosto de 12

60. Wildfire Interactive, Inc. | 1600 Seaport Boulevard, Suite 500, Redwood

    City, CA 94063 | (888) 274-0929 ZeroMQ concurrency PUSH PULL PULL PULL MAIN THREAD THREAD 1 THREAD 2 THREAD 3 I/O THREAD Sábado, 18 de Agosto de 12

61. Wildfire Interactive, Inc. | 1600 Seaport Boulevard, Suite 500, Redwood

    City, CA 94063 | (888) 274-0929 ctx = ZMQ::Context.new s = ctx.bind(:PUSH, "inproc://work") 3.times{ ws << spawn_worker(ctx) } loop do msg = work.shift break if msg == “exit” s.send(msg) end Master (work distributor) Sábado, 18 de Agosto de 12

62. Wildfire Interactive, Inc. | 1600 Seaport Boulevard, Suite 500, Redwood

    City, CA 94063 | (888) 274-0929 Thread.new do s = ctx.connect(:PULL, “inproc:// work”) loop do msg = s.recv break if msg == "quit" process(msg) end end Workers (consumer threads) Sábado, 18 de Agosto de 12

63. Wildfire Interactive, Inc. | 1600 Seaport Boulevard, Suite 500, Redwood

    City, CA 94063 | (888) 274-0929 Mongrel2 Case study Sábado, 18 de Agosto de 12

64. Wildfire Interactive, Inc. | 1600 Seaport Boulevard, Suite 500, Redwood

    City, CA 94063 | (888) 274-0929 How it works Multi-protocol ZeroMQ for backends Async backend requests Language agnostic (bindings) Sábado, 18 de Agosto de 12

65. Wildfire Interactive, Inc. | 1600 Seaport Boulevard, Suite 500, Redwood

    City, CA 94063 | (888) 274-0929 Mongrel2 topology PULL PUB PULL PUB PUSH SUB BACKEND A BACKEND B MONGREL2 CLIENT Sábado, 18 de Agosto de 12

66. Wildfire Interactive, Inc. | 1600 Seaport Boulevard, Suite 500, Redwood

    City, CA 94063 | (888) 274-0929 Best practices Patterns - learn them Stable bind() Unstable connect() Interjection principle Set water marks Read http://zguide.zeromq.org Sábado, 18 de Agosto de 12

67. Wildfire Interactive, Inc. | 1600 Seaport Boulevard, Suite 500, Redwood

    City, CA 94063 | (888) 274-0929 The Future UDP, DNS transports etc. Scalability Protocol (Linux Kernel) Mobile / embedded Monitoring support (beta 3.2) Sábado, 18 de Agosto de 12

68. Wildfire Interactive, Inc. | 1600 Seaport Boulevard, Suite 500, Redwood

    City, CA 94063 | (888) 274-0929 Questions ? (or sync up after ?) O fim Sábado, 18 de Agosto de 12

69. Wildfire Interactive, Inc. | 1600 Seaport Boulevard, Suite 500, Redwood

    City, CA 94063 | (888) 274-0929 fork @methodmissing follow @methodmissing Thanks! Sábado, 18 de Agosto de 12