NSQ - NYC Data Engineering Meetup

NSQ - NYC Data Engineering Meetup

Our talk at Ebay on 2013-05-15 for the NYC Data Engineering Meetup

Dd56a8e1de66aeedb987397511f830e7?s=128

Matt Reiferson

May 15, 2013
Tweet

Transcript

  1. 1.

    NSQ realtime distributed message processing at scale https://github.com/bitly/nsq May 15th

    2013 - NYC Data Engineering Meetup @imsnakes & @jehiah (infrastructure @bitly) Thursday, May 16, 13
  2. 3.

    PHILOSOPHY •service-oriented •avoid SPOFs •perform work asynchronously •de-couple services and

    use messaging in between •dependencies suck (make it easy to deploy) •use HTTP and JSON Thursday, May 16, 13
  3. 6.

    App ❶ ❸ ❷ DATA FLOW incoming request sync persist

    data send response Thursday, May 16, 13
  4. 7.

    App ❶ ❹ ❸ ❷ DATA FLOW incoming request sync

    persist data send response async queue message Thursday, May 16, 13
  5. 8.

    App ❶ ❹ ❸ ❷ DATA FLOW async queue message

    NSQ responsibilities Thursday, May 16, 13
  6. 10.

    PS m1 m1 m1 Producer ConsumerA ConsumerB messages duplicated to

    multiple consumers de-couple independent stream operations PUBSUB / MULTICAST Thursday, May 16, 13
  7. 11.

    Q m2 m2 m1 Producer ConsumerA ConsumerA m1 messages load

    balanced among a homogenous group of consumers horizontal scalability DISTRIBUTION Thursday, May 16, 13
  8. 12.

    Q m2 m2 m1 Producer ConsumerA ConsumerA m1 fault tolerance

    in face of consumer failure, other consumers (try to) pick up the slack DISTRIBUTION Thursday, May 16, 13
  9. 13.

    Q m1 Producer ConsumerA ConsumerA m2 if consumers cannot keep

    up with producers, the queue is able to hold onto messages so they can be processed later m3 QUEUEING X X Thursday, May 16, 13
  10. 14.

    TYPICAL (TERRIBLE) ARCHITECTURE Host A API queue queuereader •no delivery

    guarantee •SPOFs •inefficient •complicated setup •hard-coded config Thursday, May 16, 13
  11. 15.

    TYPICAL (TERRIBLE) ARCHITECTURE Host A API queue queuereader Host B

    pubsub / multicast •no delivery guarantee •SPOFs •inefficient •complicated setup •hard-coded config Thursday, May 16, 13
  12. 16.

    TYPICAL (TERRIBLE) ARCHITECTURE Host A API queue queuereader Host B

    pubsub / multicast Host C queue queuereader relay •no delivery guarantee •SPOFs •inefficient •complicated setup •hard-coded config Thursday, May 16, 13
  13. 17.

    TYPICAL (TERRIBLE) ARCHITECTURE Host A API queue queuereader Host B

    pubsub / multicast Host C queue queuereader relay SPOF SPOF COMPLEX •no delivery guarantee •SPOFs •inefficient •complicated setup •hard-coded config Thursday, May 16, 13
  14. 19.

    GOALS •provide a straightforward upgrade path •greatly simplify configuration requirements

    •promote topologies that enable high-availability and eliminate SPOFs •address the need for stronger message delivery guarantees •bound the memory footprint of a single process •improve efficiency •data format and programming language agnostic Thursday, May 16, 13
  15. 23.

    TOPICS AND CHANNELS • a topic is a distinct stream

    of messages (a single nsqd instance can have multiple topics) • a channel is an independent queue for a topic (a topic can have multiple channels) • consumers discover producers by querying nsqlookupd (a discovery service for topics) • topics and channels are created at runtime (just start publishing/subscribing) nsqd “clicks” Topics combine multicast, distribution, and queueing Thursday, May 16, 13
  16. 24.

    TOPICS AND CHANNELS • a topic is a distinct stream

    of messages (a single nsqd instance can have multiple topics) • a channel is an independent queue for a topic (a topic can have multiple channels) • consumers discover producers by querying nsqlookupd (a discovery service for topics) • topics and channels are created at runtime (just start publishing/subscribing) nsqd “metrics” Channels “clicks” Topics combine multicast, distribution, and queueing Thursday, May 16, 13
  17. 25.

    TOPICS AND CHANNELS • a topic is a distinct stream

    of messages (a single nsqd instance can have multiple topics) • a channel is an independent queue for a topic (a topic can have multiple channels) • consumers discover producers by querying nsqlookupd (a discovery service for topics) • topics and channels are created at runtime (just start publishing/subscribing) nsqd “metrics” Channels “clicks” Topics “spam_analysis” combine multicast, distribution, and queueing Thursday, May 16, 13
  18. 26.

    TOPICS AND CHANNELS • a topic is a distinct stream

    of messages (a single nsqd instance can have multiple topics) • a channel is an independent queue for a topic (a topic can have multiple channels) • consumers discover producers by querying nsqlookupd (a discovery service for topics) • topics and channels are created at runtime (just start publishing/subscribing) nsqd “metrics” Channels “clicks” Topics “spam_analysis” “archive” combine multicast, distribution, and queueing Thursday, May 16, 13
  19. 27.

    separate hosts TOPICS AND CHANNELS • a topic is a

    distinct stream of messages (a single nsqd instance can have multiple topics) • a channel is an independent queue for a topic (a topic can have multiple channels) • consumers discover producers by querying nsqlookupd (a discovery service for topics) • topics and channels are created at runtime (just start publishing/subscribing) nsqd “metrics” Channels “clicks” Topics “spam_analysis” “archive” Consumers combine multicast, distribution, and queueing Thursday, May 16, 13
  20. 28.

    separate hosts TOPICS AND CHANNELS • a topic is a

    distinct stream of messages (a single nsqd instance can have multiple topics) • a channel is an independent queue for a topic (a topic can have multiple channels) • consumers discover producers by querying nsqlookupd (a discovery service for topics) • topics and channels are created at runtime (just start publishing/subscribing) nsqd “metrics” Channels “clicks” Topics “spam_analysis” “archive” Consumers combine multicast, distribution, and queueing Thursday, May 16, 13
  21. 29.

    separate hosts TOPICS AND CHANNELS • a topic is a

    distinct stream of messages (a single nsqd instance can have multiple topics) • a channel is an independent queue for a topic (a topic can have multiple channels) • consumers discover producers by querying nsqlookupd (a discovery service for topics) • topics and channels are created at runtime (just start publishing/subscribing) nsqd “metrics” Channels “clicks” Topics “spam_analysis” “archive” Consumers combine multicast, distribution, and queueing Thursday, May 16, 13
  22. 30.

    separate hosts TOPICS AND CHANNELS • a topic is a

    distinct stream of messages (a single nsqd instance can have multiple topics) • a channel is an independent queue for a topic (a topic can have multiple channels) • consumers discover producers by querying nsqlookupd (a discovery service for topics) • topics and channels are created at runtime (just start publishing/subscribing) nsqd “metrics” Channels “clicks” Topics “spam_analysis” “archive” Consumers A A A combine multicast, distribution, and queueing Thursday, May 16, 13
  23. 31.

    separate hosts TOPICS AND CHANNELS • a topic is a

    distinct stream of messages (a single nsqd instance can have multiple topics) • a channel is an independent queue for a topic (a topic can have multiple channels) • consumers discover producers by querying nsqlookupd (a discovery service for topics) • topics and channels are created at runtime (just start publishing/subscribing) nsqd “metrics” Channels “clicks” Topics “spam_analysis” “archive” Consumers A A A combine multicast, distribution, and queueing Thursday, May 16, 13
  24. 32.

    separate hosts TOPICS AND CHANNELS • a topic is a

    distinct stream of messages (a single nsqd instance can have multiple topics) • a channel is an independent queue for a topic (a topic can have multiple channels) • consumers discover producers by querying nsqlookupd (a discovery service for topics) • topics and channels are created at runtime (just start publishing/subscribing) nsqd “metrics” Channels “clicks” Topics “spam_analysis” “archive” Consumers A A A combine multicast, distribution, and queueing Thursday, May 16, 13
  25. 33.

    separate hosts TOPICS AND CHANNELS • a topic is a

    distinct stream of messages (a single nsqd instance can have multiple topics) • a channel is an independent queue for a topic (a topic can have multiple channels) • consumers discover producers by querying nsqlookupd (a discovery service for topics) • topics and channels are created at runtime (just start publishing/subscribing) nsqd “metrics” Channels “clicks” Topics “spam_analysis” “archive” Consumers A A A combine multicast, distribution, and queueing Thursday, May 16, 13
  26. 34.

    separate hosts TOPICS AND CHANNELS • a topic is a

    distinct stream of messages (a single nsqd instance can have multiple topics) • a channel is an independent queue for a topic (a topic can have multiple channels) • consumers discover producers by querying nsqlookupd (a discovery service for topics) • topics and channels are created at runtime (just start publishing/subscribing) nsqd “metrics” Channels “clicks” Topics “spam_analysis” “archive” Consumers A A A B B B combine multicast, distribution, and queueing Thursday, May 16, 13
  27. 35.

    separate hosts TOPICS AND CHANNELS • a topic is a

    distinct stream of messages (a single nsqd instance can have multiple topics) • a channel is an independent queue for a topic (a topic can have multiple channels) • consumers discover producers by querying nsqlookupd (a discovery service for topics) • topics and channels are created at runtime (just start publishing/subscribing) nsqd “metrics” Channels “clicks” Topics “spam_analysis” “archive” Consumers A A A B B B combine multicast, distribution, and queueing Thursday, May 16, 13
  28. 36.

    separate hosts TOPICS AND CHANNELS • a topic is a

    distinct stream of messages (a single nsqd instance can have multiple topics) • a channel is an independent queue for a topic (a topic can have multiple channels) • consumers discover producers by querying nsqlookupd (a discovery service for topics) • topics and channels are created at runtime (just start publishing/subscribing) nsqd “metrics” Channels “clicks” Topics “spam_analysis” “archive” Consumers A A A B B B combine multicast, distribution, and queueing Thursday, May 16, 13
  29. 37.

    separate hosts TOPICS AND CHANNELS • a topic is a

    distinct stream of messages (a single nsqd instance can have multiple topics) • a channel is an independent queue for a topic (a topic can have multiple channels) • consumers discover producers by querying nsqlookupd (a discovery service for topics) • topics and channels are created at runtime (just start publishing/subscribing) nsqd “metrics” Channels “clicks” Topics “spam_analysis” “archive” Consumers A A A B B B combine multicast, distribution, and queueing Thursday, May 16, 13
  30. 38.

    QUEUES •topics and channels are independent queues •queues have arbitrary

    high water marks (after which messages transparently read/write to disk, bounding memory footprint) •supports channel-independent degradation and recovery buffer this channel high water mark persisted messages Thursday, May 16, 13
  31. 39.

    DISCOVERY remove the need for publishers and consumers to know

    about each other nsqlookupd nsqd producer nsqlookupd Thursday, May 16, 13
  32. 40.

    DISCOVERY remove the need for publishers and consumers to know

    about each other nsqlookupd nsqd ❶ publish msg (specifying topic) producer nsqlookupd Thursday, May 16, 13
  33. 41.

    DISCOVERY remove the need for publishers and consumers to know

    about each other nsqlookupd nsqd ❶ publish msg (specifying topic) producer ➋ IDENTIFY persistent TCP connections nsqlookupd Thursday, May 16, 13
  34. 42.

    DISCOVERY remove the need for publishers and consumers to know

    about each other nsqlookupd nsqd ❶ publish msg (specifying topic) producer ➋ IDENTIFY persistent TCP connections nsqlookupd ➌ REGISTER (topic/channel) Thursday, May 16, 13
  35. 43.

    DISCOVERY (CLIENT) remove the need for publishers and consumers to

    know about each other nsqlookupd nsqlookupd consumer Thursday, May 16, 13
  36. 44.

    DISCOVERY (CLIENT) remove the need for publishers and consumers to

    know about each other nsqlookupd nsqlookupd consumer ➊ regularly poll for topic producers HTTP requests Thursday, May 16, 13
  37. 45.

    DISCOVERY (CLIENT) remove the need for publishers and consumers to

    know about each other nsqlookupd nsqlookupd consumer ➊ regularly poll for topic producers ➋ connect to all producers HTTP requests Thursday, May 16, 13
  38. 46.

    ELIMINATE ALL THE SPOF •easily enable distributed and decentralized topologies

    •no brokers •consumers connect to all producers •messages are pushed to consumers •nsqlookupd instances are independent and require no coordination (run a few for HA) Thursday, May 16, 13
  39. 47.

    ELIMINATE ALL THE SPOF nsqd nsqd nsqd •easily enable distributed

    and decentralized topologies •no brokers •consumers connect to all producers •messages are pushed to consumers •nsqlookupd instances are independent and require no coordination (run a few for HA) Thursday, May 16, 13
  40. 48.

    ELIMINATE ALL THE SPOF nsqd nsqd nsqd consumer •easily enable

    distributed and decentralized topologies •no brokers •consumers connect to all producers •messages are pushed to consumers •nsqlookupd instances are independent and require no coordination (run a few for HA) Thursday, May 16, 13
  41. 49.

    ELIMINATE ALL THE SPOF nsqd nsqd nsqd consumer •easily enable

    distributed and decentralized topologies •no brokers •consumers connect to all producers •messages are pushed to consumers •nsqlookupd instances are independent and require no coordination (run a few for HA) Thursday, May 16, 13
  42. 50.

    ELIMINATE ALL THE SPOF nsqd nsqd nsqd consumer consumer •easily

    enable distributed and decentralized topologies •no brokers •consumers connect to all producers •messages are pushed to consumers •nsqlookupd instances are independent and require no coordination (run a few for HA) Thursday, May 16, 13
  43. 51.

    ELIMINATE ALL THE SPOF nsqd nsqd nsqd consumer consumer •easily

    enable distributed and decentralized topologies •no brokers •consumers connect to all producers •messages are pushed to consumers •nsqlookupd instances are independent and require no coordination (run a few for HA) Thursday, May 16, 13
  44. 52.

    NSQ NSQD API consumer NSQ NSQD API NSQ NSQD API

    consumer nsqlookupd nsqlookupd Thursday, May 16, 13
  45. 53.

    NSQ NSQD API consumer NSQ NSQD API NSQ NSQD API

    consumer nsqlookupd nsqlookupd PUBLISH Thursday, May 16, 13
  46. 54.

    NSQ NSQD API consumer NSQ NSQD API NSQ NSQD API

    consumer nsqlookupd nsqlookupd PUBLISH REGISTER Thursday, May 16, 13
  47. 55.

    NSQ NSQD API consumer NSQ NSQD API NSQ NSQD API

    consumer nsqlookupd nsqlookupd PUBLISH REGISTER DISCOVER Thursday, May 16, 13
  48. 56.

    NSQ NSQD API consumer NSQ NSQD API NSQ NSQD API

    consumer nsqlookupd nsqlookupd PUBLISH REGISTER DISCOVER SUBSCRIBE Thursday, May 16, 13
  49. 57.

    MESSAGE GUARANTEES •messages are delivered at least once •handling is

    guaranteed by the protocol: •nsqd sends a message and stores it temporarily •client replies FIN (finish) or REQ (re-queue) •if client does not reply message is automatically re-queued •any single nsqd instance failure can result in message loss (can be mitigated) Thursday, May 16, 13
  50. 58.

    CLIENT BEHAVIOR •messages are pushed to clients (no polling!) •clients

    manage flow via “RDY state” •clients can perform 3 actions on a message: •finish •re-queue (optionally defer by a duration of time) •touch •back off, i.e. slow down the rate of processing Thursday, May 16, 13
  51. 59.

    ONE MORE THING •#ephemeral channels - runtime introspection •no backup

    beyond channel high water mark •automatically go away when last client disconnects •server side channel pausing •administratively stop the flow of messages from a channel to its clients •no message loss (queue backs up) •really $#%^ing awesome for operations Thursday, May 16, 13
  52. 60.

    OTHER SOLUTIONS •ZeroMQ - it’s a library, not a platform

    •RabbitMQ, ActiveMQ - promotes brokered topology (and AMQP’s original authors abandoned it to build ZeroMQ) •kafka - heavyweight, complex, designed for different use case •beanstalk, kestrel - just a better queue we knew you were going to ask about this... Thursday, May 16, 13
  53. 62.

    TOOLING • nsqadmin provides a web interface to administrate and

    introspect an NSQ cluster at runtime (and empty, pause, or delete topics/ channels) • nsq_to_http - utility that helps transport an aggregate stream over HTTP • nsq_to_file - utility that safely persists an aggregated stream to disk • nsq_stat - iostat like utility for a topic/channel • nsq_tail - tail like utility for a topic/channel Thursday, May 16, 13
  54. 63.

    EXAMPLE CLIENTS •Go Client - https://gist.github.com/4039222 •Synchronous Python Client -

    https://gist.github.com/3925081 •Async Python Client - https://gist.github.com/3925092 Thursday, May 16, 13