•promote topologies that enable high-availability and eliminate SPOFs •address the need for stronger message delivery guarantees •bound the memory footprint of a single process •improve efficiency •data format and programming language agnostic Thursday, May 16, 13
of messages (a single nsqd instance can have multiple topics) • a channel is an independent queue for a topic (a topic can have multiple channels) • consumers discover producers by querying nsqlookupd (a discovery service for topics) • topics and channels are created at runtime (just start publishing/subscribing) nsqd “clicks” Topics combine multicast, distribution, and queueing Thursday, May 16, 13
of messages (a single nsqd instance can have multiple topics) • a channel is an independent queue for a topic (a topic can have multiple channels) • consumers discover producers by querying nsqlookupd (a discovery service for topics) • topics and channels are created at runtime (just start publishing/subscribing) nsqd “metrics” Channels “clicks” Topics combine multicast, distribution, and queueing Thursday, May 16, 13
of messages (a single nsqd instance can have multiple topics) • a channel is an independent queue for a topic (a topic can have multiple channels) • consumers discover producers by querying nsqlookupd (a discovery service for topics) • topics and channels are created at runtime (just start publishing/subscribing) nsqd “metrics” Channels “clicks” Topics “spam_analysis” combine multicast, distribution, and queueing Thursday, May 16, 13
of messages (a single nsqd instance can have multiple topics) • a channel is an independent queue for a topic (a topic can have multiple channels) • consumers discover producers by querying nsqlookupd (a discovery service for topics) • topics and channels are created at runtime (just start publishing/subscribing) nsqd “metrics” Channels “clicks” Topics “spam_analysis” “archive” combine multicast, distribution, and queueing Thursday, May 16, 13
distinct stream of messages (a single nsqd instance can have multiple topics) • a channel is an independent queue for a topic (a topic can have multiple channels) • consumers discover producers by querying nsqlookupd (a discovery service for topics) • topics and channels are created at runtime (just start publishing/subscribing) nsqd “metrics” Channels “clicks” Topics “spam_analysis” “archive” Consumers combine multicast, distribution, and queueing Thursday, May 16, 13
distinct stream of messages (a single nsqd instance can have multiple topics) • a channel is an independent queue for a topic (a topic can have multiple channels) • consumers discover producers by querying nsqlookupd (a discovery service for topics) • topics and channels are created at runtime (just start publishing/subscribing) nsqd “metrics” Channels “clicks” Topics “spam_analysis” “archive” Consumers combine multicast, distribution, and queueing Thursday, May 16, 13
distinct stream of messages (a single nsqd instance can have multiple topics) • a channel is an independent queue for a topic (a topic can have multiple channels) • consumers discover producers by querying nsqlookupd (a discovery service for topics) • topics and channels are created at runtime (just start publishing/subscribing) nsqd “metrics” Channels “clicks” Topics “spam_analysis” “archive” Consumers combine multicast, distribution, and queueing Thursday, May 16, 13
distinct stream of messages (a single nsqd instance can have multiple topics) • a channel is an independent queue for a topic (a topic can have multiple channels) • consumers discover producers by querying nsqlookupd (a discovery service for topics) • topics and channels are created at runtime (just start publishing/subscribing) nsqd “metrics” Channels “clicks” Topics “spam_analysis” “archive” Consumers A A A combine multicast, distribution, and queueing Thursday, May 16, 13
distinct stream of messages (a single nsqd instance can have multiple topics) • a channel is an independent queue for a topic (a topic can have multiple channels) • consumers discover producers by querying nsqlookupd (a discovery service for topics) • topics and channels are created at runtime (just start publishing/subscribing) nsqd “metrics” Channels “clicks” Topics “spam_analysis” “archive” Consumers A A A combine multicast, distribution, and queueing Thursday, May 16, 13
distinct stream of messages (a single nsqd instance can have multiple topics) • a channel is an independent queue for a topic (a topic can have multiple channels) • consumers discover producers by querying nsqlookupd (a discovery service for topics) • topics and channels are created at runtime (just start publishing/subscribing) nsqd “metrics” Channels “clicks” Topics “spam_analysis” “archive” Consumers A A A combine multicast, distribution, and queueing Thursday, May 16, 13
distinct stream of messages (a single nsqd instance can have multiple topics) • a channel is an independent queue for a topic (a topic can have multiple channels) • consumers discover producers by querying nsqlookupd (a discovery service for topics) • topics and channels are created at runtime (just start publishing/subscribing) nsqd “metrics” Channels “clicks” Topics “spam_analysis” “archive” Consumers A A A combine multicast, distribution, and queueing Thursday, May 16, 13
distinct stream of messages (a single nsqd instance can have multiple topics) • a channel is an independent queue for a topic (a topic can have multiple channels) • consumers discover producers by querying nsqlookupd (a discovery service for topics) • topics and channels are created at runtime (just start publishing/subscribing) nsqd “metrics” Channels “clicks” Topics “spam_analysis” “archive” Consumers A A A B B B combine multicast, distribution, and queueing Thursday, May 16, 13
distinct stream of messages (a single nsqd instance can have multiple topics) • a channel is an independent queue for a topic (a topic can have multiple channels) • consumers discover producers by querying nsqlookupd (a discovery service for topics) • topics and channels are created at runtime (just start publishing/subscribing) nsqd “metrics” Channels “clicks” Topics “spam_analysis” “archive” Consumers A A A B B B combine multicast, distribution, and queueing Thursday, May 16, 13
distinct stream of messages (a single nsqd instance can have multiple topics) • a channel is an independent queue for a topic (a topic can have multiple channels) • consumers discover producers by querying nsqlookupd (a discovery service for topics) • topics and channels are created at runtime (just start publishing/subscribing) nsqd “metrics” Channels “clicks” Topics “spam_analysis” “archive” Consumers A A A B B B combine multicast, distribution, and queueing Thursday, May 16, 13
distinct stream of messages (a single nsqd instance can have multiple topics) • a channel is an independent queue for a topic (a topic can have multiple channels) • consumers discover producers by querying nsqlookupd (a discovery service for topics) • topics and channels are created at runtime (just start publishing/subscribing) nsqd “metrics” Channels “clicks” Topics “spam_analysis” “archive” Consumers A A A B B B combine multicast, distribution, and queueing Thursday, May 16, 13
high water marks (after which messages transparently read/write to disk, bounding memory footprint) •supports channel-independent degradation and recovery buffer this channel high water mark persisted messages Thursday, May 16, 13
know about each other nsqlookupd nsqlookupd consumer ➊ regularly poll for topic producers ➋ connect to all producers HTTP requests Thursday, May 16, 13
•no brokers •consumers connect to all producers •messages are pushed to consumers •nsqlookupd instances are independent and require no coordination (run a few for HA) Thursday, May 16, 13
and decentralized topologies •no brokers •consumers connect to all producers •messages are pushed to consumers •nsqlookupd instances are independent and require no coordination (run a few for HA) Thursday, May 16, 13
distributed and decentralized topologies •no brokers •consumers connect to all producers •messages are pushed to consumers •nsqlookupd instances are independent and require no coordination (run a few for HA) Thursday, May 16, 13
distributed and decentralized topologies •no brokers •consumers connect to all producers •messages are pushed to consumers •nsqlookupd instances are independent and require no coordination (run a few for HA) Thursday, May 16, 13
enable distributed and decentralized topologies •no brokers •consumers connect to all producers •messages are pushed to consumers •nsqlookupd instances are independent and require no coordination (run a few for HA) Thursday, May 16, 13
enable distributed and decentralized topologies •no brokers •consumers connect to all producers •messages are pushed to consumers •nsqlookupd instances are independent and require no coordination (run a few for HA) Thursday, May 16, 13
guaranteed by the protocol: •nsqd sends a message and stores it temporarily •client replies FIN (finish) or REQ (re-queue) •if client does not reply message is automatically re-queued •any single nsqd instance failure can result in message loss (can be mitigated) Thursday, May 16, 13
manage flow via “RDY state” •clients can perform 3 actions on a message: •finish •re-queue (optionally defer by a duration of time) •touch •back off, i.e. slow down the rate of processing Thursday, May 16, 13
beyond channel high water mark •automatically go away when last client disconnects •server side channel pausing •administratively stop the flow of messages from a channel to its clients •no message loss (queue backs up) •really $#%^ing awesome for operations Thursday, May 16, 13
•RabbitMQ, ActiveMQ - promotes brokered topology (and AMQP’s original authors abandoned it to build ZeroMQ) •kafka - heavyweight, complex, designed for different use case •beanstalk, kestrel - just a better queue we knew you were going to ask about this... Thursday, May 16, 13
introspect an NSQ cluster at runtime (and empty, pause, or delete topics/ channels) • nsq_to_http - utility that helps transport an aggregate stream over HTTP • nsq_to_file - utility that safely persists an aggregated stream to disk • nsq_stat - iostat like utility for a topic/channel • nsq_tail - tail like utility for a topic/channel Thursday, May 16, 13
snakes
l2hyunwoo
igreenwood
iwatatomoya
shukob
yjszk
dropout009
nealle
stoneream
manfredsteyer
92thunder
dunglas
.jpeg){kind=avatar}
arthur1
addyosmani
jponch
vanstee
eileencodes
moore
nonsquared
lara
shpigford
hatefulcrawdad
samanthasiow
honnibal
