Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Spray Some NSQ On It

Spray Some NSQ On It

GopherCon 2014

Matt Reiferson

April 24, 2014
Tweet

More Decks by Matt Reiferson

Other Decks in Programming

Transcript

  1. BIG THINGS HAVE SMALL BEGINNINGS •boss says: “I want metrics!”

    •single host •synchronous writes API interweb 1 2
  2. BIG THINGS HAVE SMALL BEGINNINGS •boss says: “I want metrics!”

    •single host •synchronous writes API metrics interweb 1 2 3
  3. TOPICS AND CHANNELS •a topic is a distinct stream of

    messages •a topic has one or more channels •topics and channels are created at runtime •messages are pushed to consumers combine pubsub, distribution, and queueing nsqd “clicks” Topics
  4. TOPICS AND CHANNELS •a topic is a distinct stream of

    messages •a topic has one or more channels •topics and channels are created at runtime •messages are pushed to consumers combine pubsub, distribution, and queueing nsqd “metrics” Channels “clicks” Topics
  5. TOPICS AND CHANNELS •a topic is a distinct stream of

    messages •a topic has one or more channels •topics and channels are created at runtime •messages are pushed to consumers combine pubsub, distribution, and queueing nsqd “metrics” Channels “clicks” Topics “spam_analysis”
  6. TOPICS AND CHANNELS •a topic is a distinct stream of

    messages •a topic has one or more channels •topics and channels are created at runtime •messages are pushed to consumers combine pubsub, distribution, and queueing nsqd “metrics” Channels “clicks” Topics “spam_analysis” “archive”
  7. TOPICS AND CHANNELS •a topic is a distinct stream of

    messages •a topic has one or more channels •topics and channels are created at runtime •messages are pushed to consumers combine pubsub, distribution, and queueing nsqd “metrics” Channels “clicks” Topics “spam_analysis” “archive” Consumers
  8. TOPICS AND CHANNELS •a topic is a distinct stream of

    messages •a topic has one or more channels •topics and channels are created at runtime •messages are pushed to consumers combine pubsub, distribution, and queueing nsqd “metrics” Channels “clicks” Topics “spam_analysis” “archive” Consumers
  9. TOPICS AND CHANNELS •a topic is a distinct stream of

    messages •a topic has one or more channels •topics and channels are created at runtime •messages are pushed to consumers combine pubsub, distribution, and queueing nsqd “metrics” Channels “clicks” Topics “spam_analysis” “archive” Consumers
  10. TOPICS AND CHANNELS •a topic is a distinct stream of

    messages •a topic has one or more channels •topics and channels are created at runtime •messages are pushed to consumers combine pubsub, distribution, and queueing nsqd “metrics” Channels “clicks” Topics “spam_analysis” “archive” Consumers A A A
  11. TOPICS AND CHANNELS •a topic is a distinct stream of

    messages •a topic has one or more channels •topics and channels are created at runtime •messages are pushed to consumers combine pubsub, distribution, and queueing nsqd “metrics” Channels “clicks” Topics “spam_analysis” “archive” Consumers A A A
  12. TOPICS AND CHANNELS •a topic is a distinct stream of

    messages •a topic has one or more channels •topics and channels are created at runtime •messages are pushed to consumers combine pubsub, distribution, and queueing nsqd “metrics” Channels “clicks” Topics “spam_analysis” “archive” Consumers A A A
  13. TOPICS AND CHANNELS •a topic is a distinct stream of

    messages •a topic has one or more channels •topics and channels are created at runtime •messages are pushed to consumers combine pubsub, distribution, and queueing nsqd “metrics” Channels “clicks” Topics “spam_analysis” “archive” Consumers A A A
  14. TOPICS AND CHANNELS •a topic is a distinct stream of

    messages •a topic has one or more channels •topics and channels are created at runtime •messages are pushed to consumers combine pubsub, distribution, and queueing nsqd “metrics” Channels “clicks” Topics “spam_analysis” “archive” Consumers A A A B B B
  15. TOPICS AND CHANNELS •a topic is a distinct stream of

    messages •a topic has one or more channels •topics and channels are created at runtime •messages are pushed to consumers combine pubsub, distribution, and queueing nsqd “metrics” Channels “clicks” Topics “spam_analysis” “archive” Consumers A A A B B B
  16. TOPICS AND CHANNELS •a topic is a distinct stream of

    messages •a topic has one or more channels •topics and channels are created at runtime •messages are pushed to consumers combine pubsub, distribution, and queueing nsqd “metrics” Channels “clicks” Topics “spam_analysis” “archive” Consumers A A A B B B
  17. TOPICS AND CHANNELS •a topic is a distinct stream of

    messages •a topic has one or more channels •topics and channels are created at runtime •messages are pushed to consumers combine pubsub, distribution, and queueing nsqd “metrics” Channels “clicks” Topics “spam_analysis” “archive” Consumers A A A B B B
  18. UNDER THE HOOD •topics and channels are independent •configurable high

    water mark
 (disk persistence) •“bounded” memory footprint DiskQueue In-Memory goroutine Input Chan Output Chan
  19. DE-COUPLE •PUB locally to nsqd via HTTP •perform work async

    •co-locate everything (silo) API nsqd “metrics” channel “api_event” topic metrics consumer consumer
  20. SCALE HORIZONTALLY API nsqd “metrics” channel API nsqd metrics consumer

    consumer consumer consumer “api_event” topic “api_event” topic “metrics” channel … …
  21. NSQ NSQD API consumer NSQ NSQD API NSQ NSQD API

    consumer nsqlookupd nsqlookupd TYPICAL NSQ CLUSTER •enable distributed and decentralized topologies •no centralized broker •nsqlookupd instances are independent (no coordinatation)
  22. NSQ NSQD API consumer NSQ NSQD API NSQ NSQD API

    consumer nsqlookupd nsqlookupd PUBLISH TYPICAL NSQ CLUSTER •enable distributed and decentralized topologies •no centralized broker •nsqlookupd instances are independent (no coordinatation)
  23. NSQ NSQD API consumer NSQ NSQD API NSQ NSQD API

    consumer nsqlookupd nsqlookupd PUBLISH REGISTER TYPICAL NSQ CLUSTER •enable distributed and decentralized topologies •no centralized broker •nsqlookupd instances are independent (no coordinatation)
  24. NSQ NSQD API consumer NSQ NSQD API NSQ NSQD API

    consumer nsqlookupd nsqlookupd PUBLISH REGISTER DISCOVER TYPICAL NSQ CLUSTER •enable distributed and decentralized topologies •no centralized broker •nsqlookupd instances are independent (no coordinatation)
  25. NSQ NSQD API consumer NSQ NSQD API NSQ NSQD API

    consumer nsqlookupd nsqlookupd PUBLISH REGISTER DISCOVER SUBSCRIBE TYPICAL NSQ CLUSTER •enable distributed and decentralized topologies •no centralized broker •nsqlookupd instances are independent (no coordinatation)
  26. DISCOVERABILITY •introduce nsqlookupd •producers and consumers
 come and go •other

    services can discover and subscribe to this topic API nsqd “metrics” channel “api_event” topic API nsqd “metrics” channel “api_event” topic
  27. DISCOVERABILITY •introduce nsqlookupd •producers and consumers
 come and go •other

    services can discover and subscribe to this topic API nsqd “metrics” channel “api_event” topic API nsqd “metrics” channel “api_event” topic nsqlookupd TCP nsqlookupd TCP
  28. DISCOVERABILITY •introduce nsqlookupd •producers and consumers
 come and go •other

    services can discover and subscribe to this topic API nsqd “metrics” channel “api_event” topic API nsqd “metrics” channel “api_event” topic nsq_to_file nsq_to_file nsqlookupd TCP nsqlookupd TCP “archive” channel
  29. DISCOVERABILITY •introduce nsqlookupd •producers and consumers
 come and go •other

    services can discover and subscribe to this topic API nsqd “metrics” channel “api_event” topic API nsqd “metrics” channel “api_event” topic nsq_to_file nsq_to_file nsqlookupd TCP nsqlookupd TCP “archive” channel 1 - query nsqlookupd! for topic “api_event” HTTP
  30. DISCOVERABILITY •introduce nsqlookupd •producers and consumers
 come and go •other

    services can discover and subscribe to this topic API nsqd “metrics” channel “api_event” topic API nsqd “metrics” channel “api_event” topic 2 - connect to all discovered
 producers nsq_to_file nsq_to_file nsqlookupd TCP nsqlookupd TCP “archive” channel 1 - query nsqlookupd! for topic “api_event” HTTP
  31. OPS

  32. GUARANTEES •messages are delivered at least once •messages are not

    durable (by default) •messages received are un-ordered •consumers eventually find all topic producers
  33. BACK PRESSSSSSSSSSSSURE •clients control flow - RDY state •concurrency knob

    •processing fails? •slow down rate •bad message? •limit attempts •delay re-processing Incoming Msgs Msg Handler Timeout - - Check Timeout Full Throttle success Timeout == 0 RDY N
  34. BACK PRESSSSSSSSSSSSURE •clients control flow - RDY state •concurrency knob

    •processing fails? •slow down rate •bad message? •limit attempts •delay re-processing Incoming Msgs Msg Handler Timeout ++ Timeout - - Check Timeout Full Throttle success fail Timeout == 0 RDY N
  35. BACK PRESSSSSSSSSSSSURE •clients control flow - RDY state •concurrency knob

    •processing fails? •slow down rate •bad message? •limit attempts •delay re-processing Incoming Msgs Msg Handler Pause Timeout ++ Timeout - - Check Timeout Full Throttle success fail Timeout == 0 RDY N RDY 0
  36. BACK PRESSSSSSSSSSSSURE •clients control flow - RDY state •concurrency knob

    •processing fails? •slow down rate •bad message? •limit attempts •delay re-processing Incoming Msgs Msg Handler Backoff Expired Pause Timeout ++ Timeout - - Check Timeout Full Throttle success fail Timeout == 0 RDY N RDY 0
  37. BACK PRESSSSSSSSSSSSURE •clients control flow - RDY state •concurrency knob

    •processing fails? •slow down rate •bad message? •limit attempts •delay re-processing Incoming Msgs Msg Handler Test Water Backoff Expired Pause Timeout ++ Timeout - - Check Timeout Full Throttle success fail Timeout == 0 RDY N RDY 0 RDY 1
  38. BACK PRESSSSSSSSSSSSURE •clients control flow - RDY state •concurrency knob

    •processing fails? •slow down rate •bad message? •limit attempts •delay re-processing Incoming Msgs Msg Handler Test Water Backoff Expired Pause Timeout ++ Timeout - - Check Timeout Full Throttle success fail Timeout > 0 Timeout == 0 RDY N RDY 0 RDY 1