Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Spray Some NSQ On It

Spray Some NSQ On It

GopherCon 2014

Dd56a8e1de66aeedb987397511f830e7?s=128

Matt Reiferson

April 24, 2014
Tweet

Transcript

  1. ! NSQ a realtime distributed messaging platform https://github.com/bitly/nsq @imsnakes April

    24th 2014 - GopherCon Matt Reiferson - CTO at Torando Labs
  2. HI, I’M MATT

  3. HI, I’M MATT

  4. WHAT EVEN IS NSQ? NOT STABLE QUEUE NAMING SUCKS QUEUE

  5. NEW SIMPLE QUEUE

  6. SPRAY SOME NSQ ON IT

  7. BIG THINGS HAVE SMALL BEGINNINGS •boss says: “I want metrics!”

    •single host •synchronous writes API interweb 1 2
  8. BIG THINGS HAVE SMALL BEGINNINGS •boss says: “I want metrics!”

    •single host •synchronous writes API metrics interweb 1 2 3
  9. …AND THEN THEY EAT YOU

  10. nightmare …AND THEN THEY EAT YOU

  11. BUT THERE’S HOPE

  12. JUST USE

  13. Distributed Message Bus THE GOAL provide a unifying distributed system

    to receive and disseminate event data
  14. MESSAGING PATTERNS

  15. PS Producer ConsumerA ConsumerB BROADCAST de-couple

  16. PS Producer ConsumerA ConsumerB BROADCAST de-couple m1 m1

  17. PS Producer ConsumerA ConsumerB BROADCAST de-couple m1 m1 m2 m2

  18. PS Producer ConsumerA ConsumerB BROADCAST de-couple m1 m1 m2 m2

  19. PS Producer ConsumerA ConsumerB BROADCAST de-couple m2 m2

  20. Q Producer ConsumerA ConsumerA horizontal scalability DISTRIBUTION

  21. Q Producer ConsumerA ConsumerA m1 horizontal scalability DISTRIBUTION

  22. Q Producer ConsumerA ConsumerA m1 horizontal scalability DISTRIBUTION m2

  23. Q Producer ConsumerA ConsumerA m1 horizontal scalability DISTRIBUTION m2 m3

  24. Q Producer ConsumerA ConsumerA horizontal scalability DISTRIBUTION m2 m3

  25. Q Producer ConsumerA ConsumerA horizontal scalability DISTRIBUTION m2 m3

  26. Q Producer ConsumerA ConsumerA fault tolerance FAILURE

  27. Q Producer ConsumerA ConsumerA fault tolerance FAILURE m1

  28. Q Producer ConsumerA ConsumerA fault tolerance FAILURE m2 m1

  29. Q Producer ConsumerA ConsumerA fault tolerance FAILURE m2 m1

  30. Q Producer ConsumerA fault tolerance FAILURE

  31. Q Producer ConsumerA fault tolerance FAILURE m3

  32. Q Producer ConsumerA fault tolerance FAILURE m3

  33. Q Producer ConsumerA fault tolerance FAILURE m3

  34. Q Producer ConsumerA fault tolerance FAILURE

  35. Q Producer ConsumerA ConsumerA EVEN MORE FAILURE welp

  36. Q Producer EVEN MORE FAILURE welp m1

  37. Q Producer EVEN MORE FAILURE welp m2 m1

  38. Q Producer EVEN MORE FAILURE welp m2 m3 m1

  39. Q Producer EVEN MORE FAILURE welp m2 m3 m1

  40. Q Producer EVEN MORE FAILURE welp m2 m3 m1

  41. Q Producer ConsumerA EVEN MORE FAILURE welp m2 m3 m1

  42. Q Producer ConsumerA EVEN MORE FAILURE welp m2 m3 m1

  43. Q Producer ConsumerA EVEN MORE FAILURE welp m2 m3

  44. Q Producer ConsumerA EVEN MORE FAILURE welp m3

  45. Q Producer ConsumerA EVEN MORE FAILURE welp

  46. NSQD

  47. TOPICS AND CHANNELS •a topic is a distinct stream of

    messages •a topic has one or more channels •topics and channels are created at runtime •messages are pushed to consumers combine pubsub, distribution, and queueing nsqd “clicks” Topics
  48. TOPICS AND CHANNELS •a topic is a distinct stream of

    messages •a topic has one or more channels •topics and channels are created at runtime •messages are pushed to consumers combine pubsub, distribution, and queueing nsqd “metrics” Channels “clicks” Topics
  49. TOPICS AND CHANNELS •a topic is a distinct stream of

    messages •a topic has one or more channels •topics and channels are created at runtime •messages are pushed to consumers combine pubsub, distribution, and queueing nsqd “metrics” Channels “clicks” Topics “spam_analysis”
  50. TOPICS AND CHANNELS •a topic is a distinct stream of

    messages •a topic has one or more channels •topics and channels are created at runtime •messages are pushed to consumers combine pubsub, distribution, and queueing nsqd “metrics” Channels “clicks” Topics “spam_analysis” “archive”
  51. TOPICS AND CHANNELS •a topic is a distinct stream of

    messages •a topic has one or more channels •topics and channels are created at runtime •messages are pushed to consumers combine pubsub, distribution, and queueing nsqd “metrics” Channels “clicks” Topics “spam_analysis” “archive” Consumers
  52. TOPICS AND CHANNELS •a topic is a distinct stream of

    messages •a topic has one or more channels •topics and channels are created at runtime •messages are pushed to consumers combine pubsub, distribution, and queueing nsqd “metrics” Channels “clicks” Topics “spam_analysis” “archive” Consumers
  53. TOPICS AND CHANNELS •a topic is a distinct stream of

    messages •a topic has one or more channels •topics and channels are created at runtime •messages are pushed to consumers combine pubsub, distribution, and queueing nsqd “metrics” Channels “clicks” Topics “spam_analysis” “archive” Consumers
  54. TOPICS AND CHANNELS •a topic is a distinct stream of

    messages •a topic has one or more channels •topics and channels are created at runtime •messages are pushed to consumers combine pubsub, distribution, and queueing nsqd “metrics” Channels “clicks” Topics “spam_analysis” “archive” Consumers A A A
  55. TOPICS AND CHANNELS •a topic is a distinct stream of

    messages •a topic has one or more channels •topics and channels are created at runtime •messages are pushed to consumers combine pubsub, distribution, and queueing nsqd “metrics” Channels “clicks” Topics “spam_analysis” “archive” Consumers A A A
  56. TOPICS AND CHANNELS •a topic is a distinct stream of

    messages •a topic has one or more channels •topics and channels are created at runtime •messages are pushed to consumers combine pubsub, distribution, and queueing nsqd “metrics” Channels “clicks” Topics “spam_analysis” “archive” Consumers A A A
  57. TOPICS AND CHANNELS •a topic is a distinct stream of

    messages •a topic has one or more channels •topics and channels are created at runtime •messages are pushed to consumers combine pubsub, distribution, and queueing nsqd “metrics” Channels “clicks” Topics “spam_analysis” “archive” Consumers A A A
  58. TOPICS AND CHANNELS •a topic is a distinct stream of

    messages •a topic has one or more channels •topics and channels are created at runtime •messages are pushed to consumers combine pubsub, distribution, and queueing nsqd “metrics” Channels “clicks” Topics “spam_analysis” “archive” Consumers A A A B B B
  59. TOPICS AND CHANNELS •a topic is a distinct stream of

    messages •a topic has one or more channels •topics and channels are created at runtime •messages are pushed to consumers combine pubsub, distribution, and queueing nsqd “metrics” Channels “clicks” Topics “spam_analysis” “archive” Consumers A A A B B B
  60. TOPICS AND CHANNELS •a topic is a distinct stream of

    messages •a topic has one or more channels •topics and channels are created at runtime •messages are pushed to consumers combine pubsub, distribution, and queueing nsqd “metrics” Channels “clicks” Topics “spam_analysis” “archive” Consumers A A A B B B
  61. TOPICS AND CHANNELS •a topic is a distinct stream of

    messages •a topic has one or more channels •topics and channels are created at runtime •messages are pushed to consumers combine pubsub, distribution, and queueing nsqd “metrics” Channels “clicks” Topics “spam_analysis” “archive” Consumers A A A B B B
  62. UNDER THE HOOD •topics and channels are independent •configurable high

    water mark
 (disk persistence) •“bounded” memory footprint DiskQueue In-Memory goroutine Input Chan Output Chan
  63. THERE AND BACK AGAIN

  64. DE-COUPLE •PUB locally to nsqd via HTTP •perform work async

    •co-locate everything (silo) API nsqd “metrics” channel “api_event” topic metrics consumer consumer
  65. SCALE HORIZONTALLY API nsqd “metrics” channel API nsqd metrics consumer

    consumer consumer consumer “api_event” topic “api_event” topic “metrics” channel … …
  66. A MESSAGE QUEUE IS BORING (IN ISOLATION)

  67. NSQLOOKUPD

  68. NSQ NSQD API consumer NSQ NSQD API NSQ NSQD API

    consumer nsqlookupd nsqlookupd TYPICAL NSQ CLUSTER •enable distributed and decentralized topologies •no centralized broker •nsqlookupd instances are independent (no coordinatation)
  69. NSQ NSQD API consumer NSQ NSQD API NSQ NSQD API

    consumer nsqlookupd nsqlookupd PUBLISH TYPICAL NSQ CLUSTER •enable distributed and decentralized topologies •no centralized broker •nsqlookupd instances are independent (no coordinatation)
  70. NSQ NSQD API consumer NSQ NSQD API NSQ NSQD API

    consumer nsqlookupd nsqlookupd PUBLISH REGISTER TYPICAL NSQ CLUSTER •enable distributed and decentralized topologies •no centralized broker •nsqlookupd instances are independent (no coordinatation)
  71. NSQ NSQD API consumer NSQ NSQD API NSQ NSQD API

    consumer nsqlookupd nsqlookupd PUBLISH REGISTER DISCOVER TYPICAL NSQ CLUSTER •enable distributed and decentralized topologies •no centralized broker •nsqlookupd instances are independent (no coordinatation)
  72. NSQ NSQD API consumer NSQ NSQD API NSQ NSQD API

    consumer nsqlookupd nsqlookupd PUBLISH REGISTER DISCOVER SUBSCRIBE TYPICAL NSQ CLUSTER •enable distributed and decentralized topologies •no centralized broker •nsqlookupd instances are independent (no coordinatation)
  73. … AND AGAIN

  74. DISCOVERABILITY •introduce nsqlookupd •producers and consumers
 come and go •other

    services can discover and subscribe to this topic API nsqd “metrics” channel “api_event” topic API nsqd “metrics” channel “api_event” topic
  75. DISCOVERABILITY •introduce nsqlookupd •producers and consumers
 come and go •other

    services can discover and subscribe to this topic API nsqd “metrics” channel “api_event” topic API nsqd “metrics” channel “api_event” topic nsqlookupd TCP nsqlookupd TCP
  76. DISCOVERABILITY •introduce nsqlookupd •producers and consumers
 come and go •other

    services can discover and subscribe to this topic API nsqd “metrics” channel “api_event” topic API nsqd “metrics” channel “api_event” topic nsq_to_file nsq_to_file nsqlookupd TCP nsqlookupd TCP “archive” channel
  77. DISCOVERABILITY •introduce nsqlookupd •producers and consumers
 come and go •other

    services can discover and subscribe to this topic API nsqd “metrics” channel “api_event” topic API nsqd “metrics” channel “api_event” topic nsq_to_file nsq_to_file nsqlookupd TCP nsqlookupd TCP “archive” channel 1 - query nsqlookupd! for topic “api_event” HTTP
  78. DISCOVERABILITY •introduce nsqlookupd •producers and consumers
 come and go •other

    services can discover and subscribe to this topic API nsqd “metrics” channel “api_event” topic API nsqd “metrics” channel “api_event” topic 2 - connect to all discovered
 producers nsq_to_file nsq_to_file nsqlookupd TCP nsqlookupd TCP “archive” channel 1 - query nsqlookupd! for topic “api_event” HTTP
  79. TO INFINITY AND BEYOND nsqlookupd nsqd Datacenter A

  80. TO INFINITY AND BEYOND nsqlookupd nsqd Datacenter A nsq_to_nsq

  81. TO INFINITY AND BEYOND nsqlookupd nsqlookupd nsqd nsqd Datacenter A

    Datacenter B nsq_to_nsq
  82. TO INFINITY AND BEYOND nsqlookupd nsqlookupd nsqd nsqd Datacenter A

    Datacenter B nsq_to_nsq
  83. TO INFINITY AND BEYOND nsqlookupd nsqlookupd nsqd nsqd Datacenter A

    Datacenter B nsq_to_nsq
  84. OPS

  85. GUARANTEES •messages are delivered at least once •messages are not

    durable (by default) •messages received are un-ordered •consumers eventually find all topic producers
  86. BACK PRESSSSSSSSSSSSURE •clients control flow - RDY state •concurrency knob

    •processing fails? •slow down rate •bad message? •limit attempts •delay re-processing Incoming Msgs Msg Handler Timeout - - Check Timeout Full Throttle success Timeout == 0 RDY N
  87. BACK PRESSSSSSSSSSSSURE •clients control flow - RDY state •concurrency knob

    •processing fails? •slow down rate •bad message? •limit attempts •delay re-processing Incoming Msgs Msg Handler Timeout ++ Timeout - - Check Timeout Full Throttle success fail Timeout == 0 RDY N
  88. BACK PRESSSSSSSSSSSSURE •clients control flow - RDY state •concurrency knob

    •processing fails? •slow down rate •bad message? •limit attempts •delay re-processing Incoming Msgs Msg Handler Pause Timeout ++ Timeout - - Check Timeout Full Throttle success fail Timeout == 0 RDY N RDY 0
  89. BACK PRESSSSSSSSSSSSURE •clients control flow - RDY state •concurrency knob

    •processing fails? •slow down rate •bad message? •limit attempts •delay re-processing Incoming Msgs Msg Handler Backoff Expired Pause Timeout ++ Timeout - - Check Timeout Full Throttle success fail Timeout == 0 RDY N RDY 0
  90. BACK PRESSSSSSSSSSSSURE •clients control flow - RDY state •concurrency knob

    •processing fails? •slow down rate •bad message? •limit attempts •delay re-processing Incoming Msgs Msg Handler Test Water Backoff Expired Pause Timeout ++ Timeout - - Check Timeout Full Throttle success fail Timeout == 0 RDY N RDY 0 RDY 1
  91. BACK PRESSSSSSSSSSSSURE •clients control flow - RDY state •concurrency knob

    •processing fails? •slow down rate •bad message? •limit attempts •delay re-processing Incoming Msgs Msg Handler Test Water Backoff Expired Pause Timeout ++ Timeout - - Check Timeout Full Throttle success fail Timeout > 0 Timeout == 0 RDY N RDY 0 RDY 1
  92. TOPIC & CHANNEL PAUSING “clicks” “metrics” “reporting” “achrive” C
 A


    C
 H
 E DB
  93. TOPIC & CHANNEL PAUSING “clicks” “metrics” “reporting” “achrive” C
 A


    C
 H
 E DB A A A
  94. TOPIC & CHANNEL PAUSING “clicks” “metrics” “reporting” “achrive” C
 A


    C
 H
 E DB A A A B B B
  95. TOPIC & CHANNEL PAUSING “clicks” “metrics” “reporting” “achrive” C
 A


    C
 H
 E DB A A A B B B
  96. TOPIC & CHANNEL PAUSING “clicks” “metrics” “reporting” “achrive” C
 A


    C
 H
 E DB B B B
  97. TOPIC & CHANNEL PAUSING “clicks” “metrics” “reporting” “achrive” C
 A


    C
 H
 E DB
  98. TOPIC & CHANNEL PAUSING “clicks” “metrics” “reporting” “achrive” paused C


    A
 C
 H
 E DB C D C C D D
  99. TOPIC & CHANNEL PAUSING “clicks” “metrics” “reporting” paused C
 A


    C
 H
 E DB “archive” C D C C D D
  100. TOPIC & CHANNEL PAUSING “clicks” “metrics” “reporting” C
 A
 C


    H
 E DB “archive” C D C C D D
  101. TOPIC & CHANNEL PAUSING “clicks” “metrics” “reporting” C
 A
 C


    H
 E DB “archive” C D C C D D
  102. TOPIC & CHANNEL PAUSING “clicks” “metrics” “reporting” C
 A
 C


    H
 E DB “archive” C D C C D D
  103. TOPIC & CHANNEL PAUSING “clicks” “metrics” “reporting” C
 A
 C


    H
 E DB “archive” D D D
  104. TOPIC & CHANNEL PAUSING “clicks” “metrics” “reporting” C
 A
 C


    H
 E DB “archive”
  105. TOPIC & CHANNEL PAUSING “clicks” “metrics” “reporting” paused F F

    F C
 A
 C
 H
 E DB “archive”
  106. TOPIC & CHANNEL PAUSING “clicks” “metrics” “reporting” paused F F

    F C
 A
 C
 H
 E DB “archive” G G G
  107. TOPIC & CHANNEL PAUSING “clicks” “metrics” “reporting” paused F F

    F C
 A
 C
 H
 E DB “archive” G G G
  108. TOPIC & CHANNEL PAUSING “clicks” “metrics” “reporting” paused F C


    A
 C
 H
 E DB “archive” G G G
  109. TOPIC & CHANNEL PAUSING “clicks” “metrics” “reporting” paused F C


    A
 C
 H
 E DB “archive” G
  110. TOPIC & CHANNEL PAUSING “clicks” “metrics” “reporting” C
 A
 C


    H
 E DB “archive”
  111. ONE MORE THING •#ephemeral channels •channel sampling •TLS / Snappy

    •telemetry over statsd / HTTP
  112. SUMMARY •github.com/bitly/nsq •10,300 lines of Go •19 client libraries, 11

    languages •over 2 years in production
  113. SUMMARY •github.com/bitly/nsq •10,300 lines of Go •19 client libraries, 11

    languages •over 2 years in production
  114. IN PRODUCTION heavy water

  115. Thanks! @imsnakes ! https://github.com/bitly/nsq ! shoutout to @jehiah (co-author of

    NSQ)