Pro Yearly is on sale from $80 to $50! »

NSQ - NYC Golang Meetup

NSQ - NYC Golang Meetup

@bitly HQ we presented NSQ, our realtime distributed message processing system to the NYC Golang Meetup.

Dd56a8e1de66aeedb987397511f830e7?s=128

Matt Reiferson

November 08, 2012
Tweet

Transcript

  1. NSQ realtime distributed message processing at scale https://github.com/bitly/nsq November 8th

    2012 - NYC Golang Meetup @imsnakes & @jehiah (infrastructure @bitly)
  2. THE WAY OF THE BITLY

  3. PHILOSOPHY •service-oriented •perform work asynchronously •queue messages (locally) •workers process

    messages (aka “queuereader”) •scale # of workers (and backend) based on ability to handle message volume •dependencies suck (make it easy to deploy) •use HTTP and JSON
  4. App ❶ DATA FLOW incoming request

  5. App ❶ ❷ DATA FLOW incoming request sync persist data

  6. App ❶ ❸ ❷ DATA FLOW incoming request sync persist

    data send response
  7. App ❶ ❹ ❸ ❷ DATA FLOW incoming request sync

    persist data send response async queue message
  8. App ❶ ❹ ❸ ❷ DATA FLOW async queue message

    NSQ responsibilities
  9. WHY QUEUE? •try to avoid SPOFs •queue(s) and worker(s) are

    silo’d •in failure scenarios: •queues provide buffering •workers exponentially back off •messages are retried •aka no data loss because things break NSQ QUEUE API consumer
  10. WHY QUEUE? •try to avoid SPOFs •queue(s) and worker(s) are

    silo’d •in failure scenarios: •queues provide buffering •workers exponentially back off •messages are retried •aka no data loss because things break NSQ QUEUE API consumer X
  11. WHY QUEUE? •try to avoid SPOFs •queue(s) and worker(s) are

    silo’d •in failure scenarios: •queues provide buffering •workers exponentially back off •messages are retried •aka no data loss because things break NSQ QUEUE API consumer X message backlog
  12. TYPICAL (OLD) ARCHITECTURE Host A API simplequeue queuereader

  13. TYPICAL (OLD) ARCHITECTURE Host A API simplequeue queuereader Host B

    pubsub
  14. TYPICAL (OLD) ARCHITECTURE Host A API simplequeue queuereader Host B

    pubsub Host C simplequeue queuereader ps_to_http
  15. TYPICAL (OLD) ARCHITECTURE Host A API simplequeue queuereader Host B

    pubsub Host C simplequeue queuereader ps_to_http SPOF SPOF COMPLEX
  16. THE BAD •no message guarantees and clients are responsible for

    re-queueing •bottleneck and SPOF transport issue (reconnects, distribution, fault tolerance, etc.) •inefficiency - we copy streams multiple times to multiple systems •complicated service setup repeated for each stream •hard-coded knowledge of queue addresses in queuereaders •lack of internal performance metrics (clients, rate, etc.)
  17. DESIGNING A SOLUTION

  18. GOALS •provide a straightforward upgrade path •greatly simplify configuration requirements

    •provide easy topology solutions that enable high-availability and eliminate SPOFs •address the need for stronger message delivery guarantees •bound the memory footprint of a single process •improve efficiency •out of the box library support for Go and Python
  19. SIMPLIFY CONFIGURATION • a topic is a distinct stream of

    messages (a single nsqd instance can have multiple topics) • a channel is an independent queue for a topic (a topic can have multiple channels) • consumers discover producers by querying nsqlookupd (a discovery service for topics) • topics and channels are created at runtime (just start publishing/subscribing) nsqd “clicks” Topics
  20. SIMPLIFY CONFIGURATION • a topic is a distinct stream of

    messages (a single nsqd instance can have multiple topics) • a channel is an independent queue for a topic (a topic can have multiple channels) • consumers discover producers by querying nsqlookupd (a discovery service for topics) • topics and channels are created at runtime (just start publishing/subscribing) nsqd “metrics” Channels “clicks” Topics
  21. SIMPLIFY CONFIGURATION • a topic is a distinct stream of

    messages (a single nsqd instance can have multiple topics) • a channel is an independent queue for a topic (a topic can have multiple channels) • consumers discover producers by querying nsqlookupd (a discovery service for topics) • topics and channels are created at runtime (just start publishing/subscribing) nsqd “metrics” Channels “clicks” Topics “spam_analysis”
  22. SIMPLIFY CONFIGURATION • a topic is a distinct stream of

    messages (a single nsqd instance can have multiple topics) • a channel is an independent queue for a topic (a topic can have multiple channels) • consumers discover producers by querying nsqlookupd (a discovery service for topics) • topics and channels are created at runtime (just start publishing/subscribing) nsqd “metrics” Channels “clicks” Topics “spam_analysis” “archive”
  23. separate hosts SIMPLIFY CONFIGURATION • a topic is a distinct

    stream of messages (a single nsqd instance can have multiple topics) • a channel is an independent queue for a topic (a topic can have multiple channels) • consumers discover producers by querying nsqlookupd (a discovery service for topics) • topics and channels are created at runtime (just start publishing/subscribing) nsqd “metrics” Channels “clicks” Topics “spam_analysis” “archive” Consumers
  24. separate hosts SIMPLIFY CONFIGURATION • a topic is a distinct

    stream of messages (a single nsqd instance can have multiple topics) • a channel is an independent queue for a topic (a topic can have multiple channels) • consumers discover producers by querying nsqlookupd (a discovery service for topics) • topics and channels are created at runtime (just start publishing/subscribing) nsqd “metrics” Channels “clicks” Topics “spam_analysis” “archive” Consumers
  25. separate hosts SIMPLIFY CONFIGURATION • a topic is a distinct

    stream of messages (a single nsqd instance can have multiple topics) • a channel is an independent queue for a topic (a topic can have multiple channels) • consumers discover producers by querying nsqlookupd (a discovery service for topics) • topics and channels are created at runtime (just start publishing/subscribing) nsqd “metrics” Channels “clicks” Topics “spam_analysis” “archive” Consumers
  26. separate hosts SIMPLIFY CONFIGURATION • a topic is a distinct

    stream of messages (a single nsqd instance can have multiple topics) • a channel is an independent queue for a topic (a topic can have multiple channels) • consumers discover producers by querying nsqlookupd (a discovery service for topics) • topics and channels are created at runtime (just start publishing/subscribing) nsqd “metrics” Channels “clicks” Topics “spam_analysis” “archive” Consumers A A A
  27. separate hosts SIMPLIFY CONFIGURATION • a topic is a distinct

    stream of messages (a single nsqd instance can have multiple topics) • a channel is an independent queue for a topic (a topic can have multiple channels) • consumers discover producers by querying nsqlookupd (a discovery service for topics) • topics and channels are created at runtime (just start publishing/subscribing) nsqd “metrics” Channels “clicks” Topics “spam_analysis” “archive” Consumers A A A
  28. separate hosts SIMPLIFY CONFIGURATION • a topic is a distinct

    stream of messages (a single nsqd instance can have multiple topics) • a channel is an independent queue for a topic (a topic can have multiple channels) • consumers discover producers by querying nsqlookupd (a discovery service for topics) • topics and channels are created at runtime (just start publishing/subscribing) nsqd “metrics” Channels “clicks” Topics “spam_analysis” “archive” Consumers A A A
  29. separate hosts SIMPLIFY CONFIGURATION • a topic is a distinct

    stream of messages (a single nsqd instance can have multiple topics) • a channel is an independent queue for a topic (a topic can have multiple channels) • consumers discover producers by querying nsqlookupd (a discovery service for topics) • topics and channels are created at runtime (just start publishing/subscribing) nsqd “metrics” Channels “clicks” Topics “spam_analysis” “archive” Consumers A A A
  30. separate hosts SIMPLIFY CONFIGURATION • a topic is a distinct

    stream of messages (a single nsqd instance can have multiple topics) • a channel is an independent queue for a topic (a topic can have multiple channels) • consumers discover producers by querying nsqlookupd (a discovery service for topics) • topics and channels are created at runtime (just start publishing/subscribing) nsqd “metrics” Channels “clicks” Topics “spam_analysis” “archive” Consumers A A A B B B
  31. separate hosts SIMPLIFY CONFIGURATION • a topic is a distinct

    stream of messages (a single nsqd instance can have multiple topics) • a channel is an independent queue for a topic (a topic can have multiple channels) • consumers discover producers by querying nsqlookupd (a discovery service for topics) • topics and channels are created at runtime (just start publishing/subscribing) nsqd “metrics” Channels “clicks” Topics “spam_analysis” “archive” Consumers A A A B B B
  32. separate hosts SIMPLIFY CONFIGURATION • a topic is a distinct

    stream of messages (a single nsqd instance can have multiple topics) • a channel is an independent queue for a topic (a topic can have multiple channels) • consumers discover producers by querying nsqlookupd (a discovery service for topics) • topics and channels are created at runtime (just start publishing/subscribing) nsqd “metrics” Channels “clicks” Topics “spam_analysis” “archive” Consumers A A A B B B
  33. separate hosts SIMPLIFY CONFIGURATION • a topic is a distinct

    stream of messages (a single nsqd instance can have multiple topics) • a channel is an independent queue for a topic (a topic can have multiple channels) • consumers discover producers by querying nsqlookupd (a discovery service for topics) • topics and channels are created at runtime (just start publishing/subscribing) nsqd “metrics” Channels “clicks” Topics “spam_analysis” “archive” Consumers A A A B B B
  34. DISCOVERY remove the need for publishers and consumers to know

    about each other nsqlookupd nsqd producer nsqlookupd
  35. DISCOVERY remove the need for publishers and consumers to know

    about each other nsqlookupd nsqd ❶ publish msg (specifying topic) producer nsqlookupd
  36. DISCOVERY remove the need for publishers and consumers to know

    about each other nsqlookupd nsqd ❶ publish msg (specifying topic) producer ➋ IDENTIFY persistent TCP connections nsqlookupd
  37. DISCOVERY remove the need for publishers and consumers to know

    about each other nsqlookupd nsqd ❶ publish msg (specifying topic) producer ➋ IDENTIFY persistent TCP connections nsqlookupd ➌ REGISTER (topic/channel)
  38. DISCOVERY (CLIENT) remove the need for publishers and consumers to

    know about each other nsqlookupd nsqlookupd consumer
  39. DISCOVERY (CLIENT) remove the need for publishers and consumers to

    know about each other nsqlookupd nsqlookupd consumer ➊ regularly poll for topic producers HTTP requests
  40. DISCOVERY (CLIENT) remove the need for publishers and consumers to

    know about each other nsqlookupd nsqlookupd consumer ➊ regularly poll for topic producers ➋ connect to all producers HTTP requests
  41. ELIMINATE ALL THE SPOF •easily enable distributed and decentralized topologies

    •no brokers •consumers connect to all producers •messages are pushed to consumers •nsqlookupd instances are independent and require no coordination (run a few for HA)
  42. ELIMINATE ALL THE SPOF nsqd nsqd nsqd •easily enable distributed

    and decentralized topologies •no brokers •consumers connect to all producers •messages are pushed to consumers •nsqlookupd instances are independent and require no coordination (run a few for HA)
  43. ELIMINATE ALL THE SPOF nsqd nsqd nsqd consumer •easily enable

    distributed and decentralized topologies •no brokers •consumers connect to all producers •messages are pushed to consumers •nsqlookupd instances are independent and require no coordination (run a few for HA)
  44. ELIMINATE ALL THE SPOF nsqd nsqd nsqd consumer •easily enable

    distributed and decentralized topologies •no brokers •consumers connect to all producers •messages are pushed to consumers •nsqlookupd instances are independent and require no coordination (run a few for HA)
  45. ELIMINATE ALL THE SPOF nsqd nsqd nsqd consumer consumer •easily

    enable distributed and decentralized topologies •no brokers •consumers connect to all producers •messages are pushed to consumers •nsqlookupd instances are independent and require no coordination (run a few for HA)
  46. ELIMINATE ALL THE SPOF nsqd nsqd nsqd consumer consumer •easily

    enable distributed and decentralized topologies •no brokers •consumers connect to all producers •messages are pushed to consumers •nsqlookupd instances are independent and require no coordination (run a few for HA)
  47. MESSAGE GUARANTEES •messages are delivered at least once •handling is

    guaranteed by the protocol: •nsqd sends a message and stores it temporarily •client replies FIN (finish) or REQ (re-queue) •if client does not reply message is automatically re-queued •any single nsqd instance failure can result in message loss (can be mitigated)
  48. EFFICIENCY get RDY nsqd consumer

  49. EFFICIENCY get RDY nsqd consumer ➊ connect

  50. EFFICIENCY get RDY nsqd consumer ➊ connect ➋ SUB (subscribe)

  51. EFFICIENCY get RDY nsqd consumer ➊ connect ➋ SUB (subscribe)

    ➌ RDY 2
  52. EFFICIENCY get RDY nsqd consumer ➊ connect ➋ SUB (subscribe)

    Msg ➌ RDY 2
  53. EFFICIENCY get RDY nsqd consumer ➊ connect ➋ SUB (subscribe)

    Msg ➌ RDY 2 Msg
  54. EFFICIENCY get RDY nsqd consumer ➊ connect ➋ SUB (subscribe)

    Msg ➌ RDY 2 ➍ FIN (success) ➍ REQ (fail) Msg
  55. QUEUES •topics and channels are independent queues •queues have arbitrary

    high water marks (after which messages transparently read/write to disk, bounding memory footprint) •supports channel-independent degradation and recovery •10 lines of Go buffer this channel high water mark persisted messages
  56. NSQ ARCHITECTURE NSQ NSQD API consumer NSQ NSQD API NSQ

    NSQD API consumer nsqlookupd nsqlookupd
  57. NSQ ARCHITECTURE NSQ NSQD API consumer NSQ NSQD API NSQ

    NSQD API consumer nsqlookupd nsqlookupd PUBLISH
  58. NSQ ARCHITECTURE NSQ NSQD API consumer NSQ NSQD API NSQ

    NSQD API consumer nsqlookupd nsqlookupd PUBLISH REGISTER
  59. NSQ ARCHITECTURE NSQ NSQD API consumer NSQ NSQD API NSQ

    NSQD API consumer nsqlookupd nsqlookupd PUBLISH REGISTER DISCOVER
  60. NSQ ARCHITECTURE NSQ NSQD API consumer NSQ NSQD API NSQ

    NSQD API consumer nsqlookupd nsqlookupd PUBLISH REGISTER DISCOVER SUBSCRIBE
  61. TOOLING •nsqadmin provides a web interface to administrate and introspect

    an NSQ cluster at runtime •empty, pause, delete channels •nsq_to_http - utility that helps transport an aggregate stream over HTTP •nsq_to_file - utility that safely persists an aggregated stream to disk
  62. ONE MORE THING •#ephemeral channels - runtime introspection •no backup

    beyond channel high water mark •automatically go away when last client disconnects •server side channel pausing •administratively stop the flow of messages from a channel to its clients •no message loss (queue backs up) •really $#%^ing awesome for operations
  63. IMPLEMENTATION

  64. GO LESSONS LEARNED •don’t be afraid of the sync package

    •goroutines are cheap not free •watch your allocations (string() is costly, re-use buffers) •use anonymous structs for arbitrary JSON •no built-in per-request HTTP timeouts •synchronizing goroutine exit is hard - log each cleanup step in long-running goroutines •select skips nil channels
  65. IMPORT PATH github.com/bitly/nsq/nsq fork github.com/mreiferson/nsq/nsq •our git workflow involves hardcore

    forking action •re-writing import paths sucks •$ go tool install_as --import-path=github.com/bitly/nsq/nsq • https://github.com/mreiferson/go-install-as •also - relative imports are OK
  66. CLIENTS •Go Client - https://gist.github.com/4039222 •Synchronous Python Client - https://gist.github.com/3925081

    •Async Python Client - https://gist.github.com/3925092
  67. DEMO

  68. !anks @imsnakes & @jehiah https://github.com/bitly/nsq shoutouts to @danielhfrank, @ploxiln, and

    @mccutchen