$30 off During Our Annual Pro Sale. View Details »

NSQ - NYC Golang Meetup

NSQ - NYC Golang Meetup

@bitly HQ we presented NSQ, our realtime distributed message processing system to the NYC Golang Meetup.

Matt Reiferson

November 08, 2012
Tweet

More Decks by Matt Reiferson

Other Decks in Technology

Transcript

  1. NSQ
    realtime distributed message processing at scale
    https://github.com/bitly/nsq
    November 8th 2012 - NYC Golang Meetup
    @imsnakes & @jehiah (infrastructure @bitly)

    View Slide

  2. THE WAY OF THE BITLY

    View Slide

  3. PHILOSOPHY
    •service-oriented
    •perform work asynchronously
    •queue messages (locally)
    •workers process messages (aka “queuereader”)
    •scale # of workers (and backend) based on ability to handle message volume
    •dependencies suck (make it easy to deploy)
    •use HTTP and JSON

    View Slide

  4. App

    DATA FLOW
    incoming request

    View Slide

  5. App


    DATA FLOW
    incoming request
    sync persist data

    View Slide

  6. App
    ❶ ❸

    DATA FLOW
    incoming request
    sync persist data
    send response

    View Slide

  7. App




    DATA FLOW
    incoming request
    sync persist data
    send response
    async queue message

    View Slide

  8. App




    DATA FLOW
    async queue message
    NSQ responsibilities

    View Slide

  9. WHY QUEUE?
    •try to avoid SPOFs
    •queue(s) and worker(s) are silo’d
    •in failure scenarios:
    •queues provide buffering
    •workers exponentially back off
    •messages are retried
    •aka no data loss
    because things break
    NSQ
    QUEUE
    API
    consumer

    View Slide

  10. WHY QUEUE?
    •try to avoid SPOFs
    •queue(s) and worker(s) are silo’d
    •in failure scenarios:
    •queues provide buffering
    •workers exponentially back off
    •messages are retried
    •aka no data loss
    because things break
    NSQ
    QUEUE
    API
    consumer
    X

    View Slide

  11. WHY QUEUE?
    •try to avoid SPOFs
    •queue(s) and worker(s) are silo’d
    •in failure scenarios:
    •queues provide buffering
    •workers exponentially back off
    •messages are retried
    •aka no data loss
    because things break
    NSQ
    QUEUE
    API
    consumer
    X
    message backlog

    View Slide

  12. TYPICAL (OLD) ARCHITECTURE
    Host A
    API
    simplequeue
    queuereader

    View Slide

  13. TYPICAL (OLD) ARCHITECTURE
    Host A
    API
    simplequeue
    queuereader
    Host B
    pubsub

    View Slide

  14. TYPICAL (OLD) ARCHITECTURE
    Host A
    API
    simplequeue
    queuereader
    Host B
    pubsub
    Host C
    simplequeue queuereader
    ps_to_http

    View Slide

  15. TYPICAL (OLD) ARCHITECTURE
    Host A
    API
    simplequeue
    queuereader
    Host B
    pubsub
    Host C
    simplequeue queuereader
    ps_to_http
    SPOF
    SPOF
    COMPLEX

    View Slide

  16. THE BAD
    •no message guarantees and clients are responsible for re-queueing
    •bottleneck and SPOF transport issue (reconnects, distribution, fault tolerance, etc.)
    •inefficiency - we copy streams multiple times to multiple systems
    •complicated service setup repeated for each stream
    •hard-coded knowledge of queue addresses in queuereaders
    •lack of internal performance metrics (clients, rate, etc.)

    View Slide

  17. DESIGNING A SOLUTION

    View Slide

  18. GOALS
    •provide a straightforward upgrade path
    •greatly simplify configuration requirements
    •provide easy topology solutions that enable high-availability and eliminate SPOFs
    •address the need for stronger message delivery guarantees
    •bound the memory footprint of a single process
    •improve efficiency
    •out of the box library support for Go and Python

    View Slide

  19. SIMPLIFY CONFIGURATION
    • a topic is a distinct stream of messages (a
    single nsqd instance can have multiple topics)
    • a channel is an independent queue for a topic
    (a topic can have multiple channels)
    • consumers discover producers by querying
    nsqlookupd (a discovery service for topics)
    • topics and channels are created at runtime
    (just start publishing/subscribing)
    nsqd
    “clicks”
    Topics

    View Slide

  20. SIMPLIFY CONFIGURATION
    • a topic is a distinct stream of messages (a
    single nsqd instance can have multiple topics)
    • a channel is an independent queue for a topic
    (a topic can have multiple channels)
    • consumers discover producers by querying
    nsqlookupd (a discovery service for topics)
    • topics and channels are created at runtime
    (just start publishing/subscribing)
    nsqd
    “metrics”
    Channels
    “clicks”
    Topics

    View Slide

  21. SIMPLIFY CONFIGURATION
    • a topic is a distinct stream of messages (a
    single nsqd instance can have multiple topics)
    • a channel is an independent queue for a topic
    (a topic can have multiple channels)
    • consumers discover producers by querying
    nsqlookupd (a discovery service for topics)
    • topics and channels are created at runtime
    (just start publishing/subscribing)
    nsqd
    “metrics”
    Channels
    “clicks”
    Topics
    “spam_analysis”

    View Slide

  22. SIMPLIFY CONFIGURATION
    • a topic is a distinct stream of messages (a
    single nsqd instance can have multiple topics)
    • a channel is an independent queue for a topic
    (a topic can have multiple channels)
    • consumers discover producers by querying
    nsqlookupd (a discovery service for topics)
    • topics and channels are created at runtime
    (just start publishing/subscribing)
    nsqd
    “metrics”
    Channels
    “clicks”
    Topics
    “spam_analysis”
    “archive”

    View Slide

  23. separate hosts
    SIMPLIFY CONFIGURATION
    • a topic is a distinct stream of messages (a
    single nsqd instance can have multiple topics)
    • a channel is an independent queue for a topic
    (a topic can have multiple channels)
    • consumers discover producers by querying
    nsqlookupd (a discovery service for topics)
    • topics and channels are created at runtime
    (just start publishing/subscribing)
    nsqd
    “metrics”
    Channels
    “clicks”
    Topics
    “spam_analysis”
    “archive”
    Consumers

    View Slide

  24. separate hosts
    SIMPLIFY CONFIGURATION
    • a topic is a distinct stream of messages (a
    single nsqd instance can have multiple topics)
    • a channel is an independent queue for a topic
    (a topic can have multiple channels)
    • consumers discover producers by querying
    nsqlookupd (a discovery service for topics)
    • topics and channels are created at runtime
    (just start publishing/subscribing)
    nsqd
    “metrics”
    Channels
    “clicks”
    Topics
    “spam_analysis”
    “archive”
    Consumers

    View Slide

  25. separate hosts
    SIMPLIFY CONFIGURATION
    • a topic is a distinct stream of messages (a
    single nsqd instance can have multiple topics)
    • a channel is an independent queue for a topic
    (a topic can have multiple channels)
    • consumers discover producers by querying
    nsqlookupd (a discovery service for topics)
    • topics and channels are created at runtime
    (just start publishing/subscribing)
    nsqd
    “metrics”
    Channels
    “clicks”
    Topics
    “spam_analysis”
    “archive”
    Consumers

    View Slide

  26. separate hosts
    SIMPLIFY CONFIGURATION
    • a topic is a distinct stream of messages (a
    single nsqd instance can have multiple topics)
    • a channel is an independent queue for a topic
    (a topic can have multiple channels)
    • consumers discover producers by querying
    nsqlookupd (a discovery service for topics)
    • topics and channels are created at runtime
    (just start publishing/subscribing)
    nsqd
    “metrics”
    Channels
    “clicks”
    Topics
    “spam_analysis”
    “archive”
    Consumers
    A
    A
    A

    View Slide

  27. separate hosts
    SIMPLIFY CONFIGURATION
    • a topic is a distinct stream of messages (a
    single nsqd instance can have multiple topics)
    • a channel is an independent queue for a topic
    (a topic can have multiple channels)
    • consumers discover producers by querying
    nsqlookupd (a discovery service for topics)
    • topics and channels are created at runtime
    (just start publishing/subscribing)
    nsqd
    “metrics”
    Channels
    “clicks”
    Topics
    “spam_analysis”
    “archive”
    Consumers
    A
    A
    A

    View Slide

  28. separate hosts
    SIMPLIFY CONFIGURATION
    • a topic is a distinct stream of messages (a
    single nsqd instance can have multiple topics)
    • a channel is an independent queue for a topic
    (a topic can have multiple channels)
    • consumers discover producers by querying
    nsqlookupd (a discovery service for topics)
    • topics and channels are created at runtime
    (just start publishing/subscribing)
    nsqd
    “metrics”
    Channels
    “clicks”
    Topics
    “spam_analysis”
    “archive”
    Consumers
    A
    A
    A

    View Slide

  29. separate hosts
    SIMPLIFY CONFIGURATION
    • a topic is a distinct stream of messages (a
    single nsqd instance can have multiple topics)
    • a channel is an independent queue for a topic
    (a topic can have multiple channels)
    • consumers discover producers by querying
    nsqlookupd (a discovery service for topics)
    • topics and channels are created at runtime
    (just start publishing/subscribing)
    nsqd
    “metrics”
    Channels
    “clicks”
    Topics
    “spam_analysis”
    “archive”
    Consumers
    A
    A
    A

    View Slide

  30. separate hosts
    SIMPLIFY CONFIGURATION
    • a topic is a distinct stream of messages (a
    single nsqd instance can have multiple topics)
    • a channel is an independent queue for a topic
    (a topic can have multiple channels)
    • consumers discover producers by querying
    nsqlookupd (a discovery service for topics)
    • topics and channels are created at runtime
    (just start publishing/subscribing)
    nsqd
    “metrics”
    Channels
    “clicks”
    Topics
    “spam_analysis”
    “archive”
    Consumers
    A
    A
    A
    B
    B
    B

    View Slide

  31. separate hosts
    SIMPLIFY CONFIGURATION
    • a topic is a distinct stream of messages (a
    single nsqd instance can have multiple topics)
    • a channel is an independent queue for a topic
    (a topic can have multiple channels)
    • consumers discover producers by querying
    nsqlookupd (a discovery service for topics)
    • topics and channels are created at runtime
    (just start publishing/subscribing)
    nsqd
    “metrics”
    Channels
    “clicks”
    Topics
    “spam_analysis”
    “archive”
    Consumers
    A
    A
    A
    B
    B
    B

    View Slide

  32. separate hosts
    SIMPLIFY CONFIGURATION
    • a topic is a distinct stream of messages (a
    single nsqd instance can have multiple topics)
    • a channel is an independent queue for a topic
    (a topic can have multiple channels)
    • consumers discover producers by querying
    nsqlookupd (a discovery service for topics)
    • topics and channels are created at runtime
    (just start publishing/subscribing)
    nsqd
    “metrics”
    Channels
    “clicks”
    Topics
    “spam_analysis”
    “archive”
    Consumers
    A
    A
    A
    B
    B
    B

    View Slide

  33. separate hosts
    SIMPLIFY CONFIGURATION
    • a topic is a distinct stream of messages (a
    single nsqd instance can have multiple topics)
    • a channel is an independent queue for a topic
    (a topic can have multiple channels)
    • consumers discover producers by querying
    nsqlookupd (a discovery service for topics)
    • topics and channels are created at runtime
    (just start publishing/subscribing)
    nsqd
    “metrics”
    Channels
    “clicks”
    Topics
    “spam_analysis”
    “archive”
    Consumers
    A
    A
    A
    B
    B
    B

    View Slide

  34. DISCOVERY
    remove the need for publishers and consumers to know about each other
    nsqlookupd
    nsqd
    producer
    nsqlookupd

    View Slide

  35. DISCOVERY
    remove the need for publishers and consumers to know about each other
    nsqlookupd
    nsqd
    ❶ publish msg (specifying topic)
    producer
    nsqlookupd

    View Slide

  36. DISCOVERY
    remove the need for publishers and consumers to know about each other
    nsqlookupd
    nsqd
    ❶ publish msg (specifying topic)
    producer
    ➋ IDENTIFY
    persistent TCP connections
    nsqlookupd

    View Slide

  37. DISCOVERY
    remove the need for publishers and consumers to know about each other
    nsqlookupd
    nsqd
    ❶ publish msg (specifying topic)
    producer
    ➋ IDENTIFY
    persistent TCP connections
    nsqlookupd
    ➌ REGISTER (topic/channel)

    View Slide

  38. DISCOVERY (CLIENT)
    remove the need for publishers and consumers to know about each other
    nsqlookupd nsqlookupd
    consumer

    View Slide

  39. DISCOVERY (CLIENT)
    remove the need for publishers and consumers to know about each other
    nsqlookupd nsqlookupd
    consumer
    ➊ regularly poll for topic producers
    HTTP requests

    View Slide

  40. DISCOVERY (CLIENT)
    remove the need for publishers and consumers to know about each other
    nsqlookupd nsqlookupd
    consumer
    ➊ regularly poll for topic producers
    ➋ connect to all producers
    HTTP requests

    View Slide

  41. ELIMINATE ALL THE SPOF
    •easily enable distributed and
    decentralized topologies
    •no brokers
    •consumers connect to all producers
    •messages are pushed to consumers
    •nsqlookupd instances are independent
    and require no coordination (run a
    few for HA)

    View Slide

  42. ELIMINATE ALL THE SPOF
    nsqd nsqd
    nsqd
    •easily enable distributed and
    decentralized topologies
    •no brokers
    •consumers connect to all producers
    •messages are pushed to consumers
    •nsqlookupd instances are independent
    and require no coordination (run a
    few for HA)

    View Slide

  43. ELIMINATE ALL THE SPOF
    nsqd nsqd
    nsqd
    consumer
    •easily enable distributed and
    decentralized topologies
    •no brokers
    •consumers connect to all producers
    •messages are pushed to consumers
    •nsqlookupd instances are independent
    and require no coordination (run a
    few for HA)

    View Slide

  44. ELIMINATE ALL THE SPOF
    nsqd nsqd
    nsqd
    consumer
    •easily enable distributed and
    decentralized topologies
    •no brokers
    •consumers connect to all producers
    •messages are pushed to consumers
    •nsqlookupd instances are independent
    and require no coordination (run a
    few for HA)

    View Slide

  45. ELIMINATE ALL THE SPOF
    nsqd nsqd
    nsqd
    consumer consumer
    •easily enable distributed and
    decentralized topologies
    •no brokers
    •consumers connect to all producers
    •messages are pushed to consumers
    •nsqlookupd instances are independent
    and require no coordination (run a
    few for HA)

    View Slide

  46. ELIMINATE ALL THE SPOF
    nsqd nsqd
    nsqd
    consumer consumer
    •easily enable distributed and
    decentralized topologies
    •no brokers
    •consumers connect to all producers
    •messages are pushed to consumers
    •nsqlookupd instances are independent
    and require no coordination (run a
    few for HA)

    View Slide

  47. MESSAGE GUARANTEES
    •messages are delivered at least once
    •handling is guaranteed by the protocol:
    •nsqd sends a message and stores it temporarily
    •client replies FIN (finish) or REQ (re-queue)
    •if client does not reply message is automatically re-queued
    •any single nsqd instance failure can result in message loss (can be mitigated)

    View Slide

  48. EFFICIENCY
    get RDY
    nsqd
    consumer

    View Slide

  49. EFFICIENCY
    get RDY
    nsqd
    consumer
    ➊ connect

    View Slide

  50. EFFICIENCY
    get RDY
    nsqd
    consumer
    ➊ connect
    ➋ SUB
    (subscribe)

    View Slide

  51. EFFICIENCY
    get RDY
    nsqd
    consumer
    ➊ connect
    ➋ SUB
    (subscribe)
    ➌ RDY 2

    View Slide

  52. EFFICIENCY
    get RDY
    nsqd
    consumer
    ➊ connect
    ➋ SUB
    (subscribe)
    Msg
    ➌ RDY 2

    View Slide

  53. EFFICIENCY
    get RDY
    nsqd
    consumer
    ➊ connect
    ➋ SUB
    (subscribe)
    Msg
    ➌ RDY 2
    Msg

    View Slide

  54. EFFICIENCY
    get RDY
    nsqd
    consumer
    ➊ connect
    ➋ SUB
    (subscribe)
    Msg
    ➌ RDY 2
    ➍ FIN (success)
    ➍ REQ (fail)
    Msg

    View Slide

  55. QUEUES
    •topics and channels are independent queues
    •queues have arbitrary high water marks (after
    which messages transparently read/write to
    disk, bounding memory footprint)
    •supports channel-independent degradation and
    recovery
    •10 lines of Go
    buffer this
    channel
    high water mark
    persisted
    messages

    View Slide

  56. NSQ ARCHITECTURE
    NSQ
    NSQD
    API
    consumer
    NSQ
    NSQD
    API
    NSQ
    NSQD
    API
    consumer
    nsqlookupd
    nsqlookupd

    View Slide

  57. NSQ ARCHITECTURE
    NSQ
    NSQD
    API
    consumer
    NSQ
    NSQD
    API
    NSQ
    NSQD
    API
    consumer
    nsqlookupd
    nsqlookupd
    PUBLISH

    View Slide

  58. NSQ ARCHITECTURE
    NSQ
    NSQD
    API
    consumer
    NSQ
    NSQD
    API
    NSQ
    NSQD
    API
    consumer
    nsqlookupd
    nsqlookupd
    PUBLISH
    REGISTER

    View Slide

  59. NSQ ARCHITECTURE
    NSQ
    NSQD
    API
    consumer
    NSQ
    NSQD
    API
    NSQ
    NSQD
    API
    consumer
    nsqlookupd
    nsqlookupd
    PUBLISH
    REGISTER
    DISCOVER

    View Slide

  60. NSQ ARCHITECTURE
    NSQ
    NSQD
    API
    consumer
    NSQ
    NSQD
    API
    NSQ
    NSQD
    API
    consumer
    nsqlookupd
    nsqlookupd
    PUBLISH
    REGISTER
    DISCOVER
    SUBSCRIBE

    View Slide

  61. TOOLING
    •nsqadmin provides a web interface to
    administrate and introspect an NSQ
    cluster at runtime
    •empty, pause, delete channels
    •nsq_to_http - utility that helps transport
    an aggregate stream over HTTP
    •nsq_to_file - utility that safely persists an
    aggregated stream to disk

    View Slide

  62. ONE MORE THING
    •#ephemeral channels - runtime introspection
    •no backup beyond channel high water mark
    •automatically go away when last client disconnects
    •server side channel pausing
    •administratively stop the flow of messages from a channel to its clients
    •no message loss (queue backs up)
    •really $#%^ing awesome for operations

    View Slide

  63. IMPLEMENTATION

    View Slide

  64. GO LESSONS LEARNED
    •don’t be afraid of the sync package
    •goroutines are cheap not free
    •watch your allocations (string() is costly, re-use buffers)
    •use anonymous structs for arbitrary JSON
    •no built-in per-request HTTP timeouts
    •synchronizing goroutine exit is hard - log each cleanup step in long-running goroutines
    •select skips nil channels

    View Slide

  65. IMPORT PATH
    github.com/bitly/nsq/nsq
    fork
    github.com/mreiferson/nsq/nsq
    •our git workflow involves hardcore forking action
    •re-writing import paths sucks
    •$ go tool install_as --import-path=github.com/bitly/nsq/nsq
    • https://github.com/mreiferson/go-install-as
    •also - relative imports are OK

    View Slide

  66. CLIENTS
    •Go Client - https://gist.github.com/4039222
    •Synchronous Python Client - https://gist.github.com/3925081
    •Async Python Client - https://gist.github.com/3925092

    View Slide

  67. DEMO

    View Slide

  68. !anks
    @imsnakes & @jehiah
    https://github.com/bitly/nsq
    shoutouts to @danielhfrank, @ploxiln, and @mccutchen

    View Slide