Spray Some NSQ On It

GopherCon 2014

1. ! NSQ a realtime distributed messaging platform https://github.com/bitly/nsq @imsnakes April 24th 2014 - GopherCon Matt Reiferson - CTO at Torando Labs
2. HI, I’M MATT
3. HI, I’M MATT
4. WHAT EVEN IS NSQ? NOT STABLE QUEUE NAMING SUCKS QUEUE
5. NEW SIMPLE QUEUE
6. SPRAY SOME NSQ ON IT
7. BIG THINGS HAVE SMALL BEGINNINGS •boss says: “I want metrics!” •single host •synchronous writes API interweb 1 2
8. BIG THINGS HAVE SMALL BEGINNINGS •boss says: “I want metrics!” •single host •synchronous writes API metrics interweb 1 2 3
9. …AND THEN THEY EAT YOU
10. nightmare …AND THEN THEY EAT YOU
11. BUT THERE’S HOPE
12. JUST USE
13. Distributed Message Bus THE GOAL provide a unifying distributed system to receive and disseminate event data
14. MESSAGING PATTERNS
15. PS Producer ConsumerA ConsumerB BROADCAST de-couple
16. PS Producer ConsumerA ConsumerB BROADCAST de-couple m1 m1
17. PS Producer ConsumerA ConsumerB BROADCAST de-couple m1 m1 m2 m2
18. PS Producer ConsumerA ConsumerB BROADCAST de-couple m1 m1 m2 m2
19. PS Producer ConsumerA ConsumerB BROADCAST de-couple m2 m2
20. Q Producer ConsumerA ConsumerA horizontal scalability DISTRIBUTION
21. Q Producer ConsumerA ConsumerA m1 horizontal scalability DISTRIBUTION
22. Q Producer ConsumerA ConsumerA m1 horizontal scalability DISTRIBUTION m2
23. Q Producer ConsumerA ConsumerA m1 horizontal scalability DISTRIBUTION m2 m3
24. Q Producer ConsumerA ConsumerA horizontal scalability DISTRIBUTION m2 m3
25. Q Producer ConsumerA ConsumerA horizontal scalability DISTRIBUTION m2 m3
26. Q Producer ConsumerA ConsumerA fault tolerance FAILURE
27. Q Producer ConsumerA ConsumerA fault tolerance FAILURE m1
28. Q Producer ConsumerA ConsumerA fault tolerance FAILURE m2 m1
29. Q Producer ConsumerA ConsumerA fault tolerance FAILURE m2 m1
30. Q Producer ConsumerA fault tolerance FAILURE
31. Q Producer ConsumerA fault tolerance FAILURE m3
32. Q Producer ConsumerA fault tolerance FAILURE m3
33. Q Producer ConsumerA fault tolerance FAILURE m3
34. Q Producer ConsumerA fault tolerance FAILURE
35. Q Producer ConsumerA ConsumerA EVEN MORE FAILURE welp
36. Q Producer EVEN MORE FAILURE welp m1
37. Q Producer EVEN MORE FAILURE welp m2 m1
38. Q Producer EVEN MORE FAILURE welp m2 m3 m1
39. Q Producer EVEN MORE FAILURE welp m2 m3 m1
40. Q Producer EVEN MORE FAILURE welp m2 m3 m1
41. Q Producer ConsumerA EVEN MORE FAILURE welp m2 m3 m1
42. Q Producer ConsumerA EVEN MORE FAILURE welp m2 m3 m1
43. Q Producer ConsumerA EVEN MORE FAILURE welp m2 m3
44. Q Producer ConsumerA EVEN MORE FAILURE welp m3
45. Q Producer ConsumerA EVEN MORE FAILURE welp
46. NSQD
47. TOPICS AND CHANNELS •a topic is a distinct stream of messages •a topic has one or more channels •topics and channels are created at runtime •messages are pushed to consumers combine pubsub, distribution, and queueing nsqd “clicks” Topics
48. TOPICS AND CHANNELS •a topic is a distinct stream of messages •a topic has one or more channels •topics and channels are created at runtime •messages are pushed to consumers combine pubsub, distribution, and queueing nsqd “metrics” Channels “clicks” Topics
49. TOPICS AND CHANNELS •a topic is a distinct stream of messages •a topic has one or more channels •topics and channels are created at runtime •messages are pushed to consumers combine pubsub, distribution, and queueing nsqd “metrics” Channels “clicks” Topics “spam_analysis”
50. TOPICS AND CHANNELS •a topic is a distinct stream of messages •a topic has one or more channels •topics and channels are created at runtime •messages are pushed to consumers combine pubsub, distribution, and queueing nsqd “metrics” Channels “clicks” Topics “spam_analysis” “archive”
51. TOPICS AND CHANNELS •a topic is a distinct stream of messages •a topic has one or more channels •topics and channels are created at runtime •messages are pushed to consumers combine pubsub, distribution, and queueing nsqd “metrics” Channels “clicks” Topics “spam_analysis” “archive” Consumers
52. TOPICS AND CHANNELS •a topic is a distinct stream of messages •a topic has one or more channels •topics and channels are created at runtime •messages are pushed to consumers combine pubsub, distribution, and queueing nsqd “metrics” Channels “clicks” Topics “spam_analysis” “archive” Consumers
53. TOPICS AND CHANNELS •a topic is a distinct stream of messages •a topic has one or more channels •topics and channels are created at runtime •messages are pushed to consumers combine pubsub, distribution, and queueing nsqd “metrics” Channels “clicks” Topics “spam_analysis” “archive” Consumers
54. TOPICS AND CHANNELS •a topic is a distinct stream of messages •a topic has one or more channels •topics and channels are created at runtime •messages are pushed to consumers combine pubsub, distribution, and queueing nsqd “metrics” Channels “clicks” Topics “spam_analysis” “archive” Consumers A A A
55. TOPICS AND CHANNELS •a topic is a distinct stream of messages •a topic has one or more channels •topics and channels are created at runtime •messages are pushed to consumers combine pubsub, distribution, and queueing nsqd “metrics” Channels “clicks” Topics “spam_analysis” “archive” Consumers A A A
56. TOPICS AND CHANNELS •a topic is a distinct stream of messages •a topic has one or more channels •topics and channels are created at runtime •messages are pushed to consumers combine pubsub, distribution, and queueing nsqd “metrics” Channels “clicks” Topics “spam_analysis” “archive” Consumers A A A
57. TOPICS AND CHANNELS •a topic is a distinct stream of messages •a topic has one or more channels •topics and channels are created at runtime •messages are pushed to consumers combine pubsub, distribution, and queueing nsqd “metrics” Channels “clicks” Topics “spam_analysis” “archive” Consumers A A A
58. TOPICS AND CHANNELS •a topic is a distinct stream of messages •a topic has one or more channels •topics and channels are created at runtime •messages are pushed to consumers combine pubsub, distribution, and queueing nsqd “metrics” Channels “clicks” Topics “spam_analysis” “archive” Consumers A A A B B B
59. TOPICS AND CHANNELS •a topic is a distinct stream of messages •a topic has one or more channels •topics and channels are created at runtime •messages are pushed to consumers combine pubsub, distribution, and queueing nsqd “metrics” Channels “clicks” Topics “spam_analysis” “archive” Consumers A A A B B B
60. TOPICS AND CHANNELS •a topic is a distinct stream of messages •a topic has one or more channels •topics and channels are created at runtime •messages are pushed to consumers combine pubsub, distribution, and queueing nsqd “metrics” Channels “clicks” Topics “spam_analysis” “archive” Consumers A A A B B B
61. TOPICS AND CHANNELS •a topic is a distinct stream of messages •a topic has one or more channels •topics and channels are created at runtime •messages are pushed to consumers combine pubsub, distribution, and queueing nsqd “metrics” Channels “clicks” Topics “spam_analysis” “archive” Consumers A A A B B B
62. UNDER THE HOOD •topics and channels are independent •configurable high water mark
 (disk persistence) •“bounded” memory footprint DiskQueue In-Memory goroutine Input Chan Output Chan
63. THERE AND BACK AGAIN
64. DE-COUPLE •PUB locally to nsqd via HTTP •perform work async •co-locate everything (silo) API nsqd “metrics” channel “api_event” topic metrics consumer consumer
65. SCALE HORIZONTALLY API nsqd “metrics” channel API nsqd metrics consumer consumer consumer consumer “api_event” topic “api_event” topic “metrics” channel … …
66. A MESSAGE QUEUE IS BORING (IN ISOLATION)
67. NSQLOOKUPD
68. NSQ NSQD API consumer NSQ NSQD API NSQ NSQD API consumer nsqlookupd nsqlookupd TYPICAL NSQ CLUSTER •enable distributed and decentralized topologies •no centralized broker •nsqlookupd instances are independent (no coordinatation)
69. NSQ NSQD API consumer NSQ NSQD API NSQ NSQD API consumer nsqlookupd nsqlookupd PUBLISH TYPICAL NSQ CLUSTER •enable distributed and decentralized topologies •no centralized broker •nsqlookupd instances are independent (no coordinatation)
70. NSQ NSQD API consumer NSQ NSQD API NSQ NSQD API consumer nsqlookupd nsqlookupd PUBLISH REGISTER TYPICAL NSQ CLUSTER •enable distributed and decentralized topologies •no centralized broker •nsqlookupd instances are independent (no coordinatation)
71. NSQ NSQD API consumer NSQ NSQD API NSQ NSQD API consumer nsqlookupd nsqlookupd PUBLISH REGISTER DISCOVER TYPICAL NSQ CLUSTER •enable distributed and decentralized topologies •no centralized broker •nsqlookupd instances are independent (no coordinatation)
72. NSQ NSQD API consumer NSQ NSQD API NSQ NSQD API consumer nsqlookupd nsqlookupd PUBLISH REGISTER DISCOVER SUBSCRIBE TYPICAL NSQ CLUSTER •enable distributed and decentralized topologies •no centralized broker •nsqlookupd instances are independent (no coordinatation)
73. … AND AGAIN
74. DISCOVERABILITY •introduce nsqlookupd •producers and consumers
 come and go •other services can discover and subscribe to this topic API nsqd “metrics” channel “api_event” topic API nsqd “metrics” channel “api_event” topic
75. DISCOVERABILITY •introduce nsqlookupd •producers and consumers
 come and go •other services can discover and subscribe to this topic API nsqd “metrics” channel “api_event” topic API nsqd “metrics” channel “api_event” topic nsqlookupd TCP nsqlookupd TCP
76. DISCOVERABILITY •introduce nsqlookupd •producers and consumers
 come and go •other services can discover and subscribe to this topic API nsqd “metrics” channel “api_event” topic API nsqd “metrics” channel “api_event” topic nsq_to_file nsq_to_file nsqlookupd TCP nsqlookupd TCP “archive” channel
77. DISCOVERABILITY •introduce nsqlookupd •producers and consumers
 come and go •other services can discover and subscribe to this topic API nsqd “metrics” channel “api_event” topic API nsqd “metrics” channel “api_event” topic nsq_to_file nsq_to_file nsqlookupd TCP nsqlookupd TCP “archive” channel 1 - query nsqlookupd! for topic “api_event” HTTP
78. DISCOVERABILITY •introduce nsqlookupd •producers and consumers
 come and go •other services can discover and subscribe to this topic API nsqd “metrics” channel “api_event” topic API nsqd “metrics” channel “api_event” topic 2 - connect to all discovered
 producers nsq_to_file nsq_to_file nsqlookupd TCP nsqlookupd TCP “archive” channel 1 - query nsqlookupd! for topic “api_event” HTTP
79. TO INFINITY AND BEYOND nsqlookupd nsqd Datacenter A
80. TO INFINITY AND BEYOND nsqlookupd nsqd Datacenter A nsq_to_nsq
81. TO INFINITY AND BEYOND nsqlookupd nsqlookupd nsqd nsqd Datacenter A Datacenter B nsq_to_nsq
82. TO INFINITY AND BEYOND nsqlookupd nsqlookupd nsqd nsqd Datacenter A Datacenter B nsq_to_nsq
83. TO INFINITY AND BEYOND nsqlookupd nsqlookupd nsqd nsqd Datacenter A Datacenter B nsq_to_nsq
84. OPS
85. GUARANTEES •messages are delivered at least once •messages are not durable (by default) •messages received are un-ordered •consumers eventually find all topic producers
86. BACK PRESSSSSSSSSSSSURE •clients control flow - RDY state •concurrency knob •processing fails? •slow down rate •bad message? •limit attempts •delay re-processing Incoming Msgs Msg Handler Timeout - - Check Timeout Full Throttle success Timeout == 0 RDY N
87. BACK PRESSSSSSSSSSSSURE •clients control flow - RDY state •concurrency knob •processing fails? •slow down rate •bad message? •limit attempts •delay re-processing Incoming Msgs Msg Handler Timeout ++ Timeout - - Check Timeout Full Throttle success fail Timeout == 0 RDY N
88. BACK PRESSSSSSSSSSSSURE •clients control flow - RDY state •concurrency knob •processing fails? •slow down rate •bad message? •limit attempts •delay re-processing Incoming Msgs Msg Handler Pause Timeout ++ Timeout - - Check Timeout Full Throttle success fail Timeout == 0 RDY N RDY 0
89. BACK PRESSSSSSSSSSSSURE •clients control flow - RDY state •concurrency knob •processing fails? •slow down rate •bad message? •limit attempts •delay re-processing Incoming Msgs Msg Handler Backoff Expired Pause Timeout ++ Timeout - - Check Timeout Full Throttle success fail Timeout == 0 RDY N RDY 0
90. BACK PRESSSSSSSSSSSSURE •clients control flow - RDY state •concurrency knob •processing fails? •slow down rate •bad message? •limit attempts •delay re-processing Incoming Msgs Msg Handler Test Water Backoff Expired Pause Timeout ++ Timeout - - Check Timeout Full Throttle success fail Timeout == 0 RDY N RDY 0 RDY 1
91. BACK PRESSSSSSSSSSSSURE •clients control flow - RDY state •concurrency knob •processing fails? •slow down rate •bad message? •limit attempts •delay re-processing Incoming Msgs Msg Handler Test Water Backoff Expired Pause Timeout ++ Timeout - - Check Timeout Full Throttle success fail Timeout > 0 Timeout == 0 RDY N RDY 0 RDY 1
92. TOPIC & CHANNEL PAUSING “clicks” “metrics” “reporting” “achrive” C
 A
 C
 H
 E DB
93. TOPIC & CHANNEL PAUSING “clicks” “metrics” “reporting” “achrive” C
 A
 C
 H
 E DB A A A
94. TOPIC & CHANNEL PAUSING “clicks” “metrics” “reporting” “achrive” C
 A
 C
 H
 E DB A A A B B B
95. TOPIC & CHANNEL PAUSING “clicks” “metrics” “reporting” “achrive” C
 A
 C
 H
 E DB A A A B B B
96. TOPIC & CHANNEL PAUSING “clicks” “metrics” “reporting” “achrive” C
 A
 C
 H
 E DB B B B
97. TOPIC & CHANNEL PAUSING “clicks” “metrics” “reporting” “achrive” C
 A
 C
 H
 E DB
98. TOPIC & CHANNEL PAUSING “clicks” “metrics” “reporting” “achrive” paused C
 A
 C
 H
 E DB C D C C D D
99. TOPIC & CHANNEL PAUSING “clicks” “metrics” “reporting” paused C
 A
 C
 H
 E DB “archive” C D C C D D
100. TOPIC & CHANNEL PAUSING “clicks” “metrics” “reporting” C
 A
 C
 H
 E DB “archive” C D C C D D
101. TOPIC & CHANNEL PAUSING “clicks” “metrics” “reporting” C
 A
 C
 H
 E DB “archive” C D C C D D
102. TOPIC & CHANNEL PAUSING “clicks” “metrics” “reporting” C
 A
 C
 H
 E DB “archive” C D C C D D
103. TOPIC & CHANNEL PAUSING “clicks” “metrics” “reporting” C
 A
 C
 H
 E DB “archive” D D D
104. TOPIC & CHANNEL PAUSING “clicks” “metrics” “reporting” C
 A
 C
 H
 E DB “archive”
105. TOPIC & CHANNEL PAUSING “clicks” “metrics” “reporting” paused F F F C
 A
 C
 H
 E DB “archive”
106. TOPIC & CHANNEL PAUSING “clicks” “metrics” “reporting” paused F F F C
 A
 C
 H
 E DB “archive” G G G
107. TOPIC & CHANNEL PAUSING “clicks” “metrics” “reporting” paused F F F C
 A
 C
 H
 E DB “archive” G G G
108. TOPIC & CHANNEL PAUSING “clicks” “metrics” “reporting” paused F C
 A
 C
 H
 E DB “archive” G G G
109. TOPIC & CHANNEL PAUSING “clicks” “metrics” “reporting” paused F C
 A
 C
 H
 E DB “archive” G
110. TOPIC & CHANNEL PAUSING “clicks” “metrics” “reporting” C
 A
 C
 H
 E DB “archive”
111. ONE MORE THING •#ephemeral channels •channel sampling •TLS / Snappy •telemetry over statsd / HTTP
112. SUMMARY •github.com/bitly/nsq •10,300 lines of Go •19 client libraries, 11 languages •over 2 years in production
113. SUMMARY •github.com/bitly/nsq •10,300 lines of Go •19 client libraries, 11 languages •over 2 years in production
114. IN PRODUCTION heavy water
115. Thanks! @imsnakes ! https://github.com/bitly/nsq ! shoutout to @jehiah (co-author of NSQ)