Spray Some NSQ On It

Transcript

1. ! NSQ a realtime distributed messaging platform https://github.com/bitly/nsq @imsnakes April

   24th 2014 - GopherCon Matt Reiferson - CTO at Torando Labs

2. HI, I’M MATT

3. HI, I’M MATT

4. WHAT EVEN IS NSQ? NOT STABLE QUEUE NAMING SUCKS QUEUE

5. NEW SIMPLE QUEUE

6. SPRAY SOME NSQ ON IT

7. BIG THINGS HAVE SMALL BEGINNINGS •boss says: “I want metrics!”

   •single host •synchronous writes API interweb 1 2

8. BIG THINGS HAVE SMALL BEGINNINGS •boss says: “I want metrics!”

   •single host •synchronous writes API metrics interweb 1 2 3

9. …AND THEN THEY EAT YOU

10. nightmare …AND THEN THEY EAT YOU

11. BUT THERE’S HOPE

12. JUST USE

13. Distributed Message Bus THE GOAL provide a unifying distributed system

    to receive and disseminate event data

14. MESSAGING PATTERNS

15. PS Producer ConsumerA ConsumerB BROADCAST de-couple

16. PS Producer ConsumerA ConsumerB BROADCAST de-couple m1 m1

17. PS Producer ConsumerA ConsumerB BROADCAST de-couple m1 m1 m2 m2

18. PS Producer ConsumerA ConsumerB BROADCAST de-couple m1 m1 m2 m2

19. PS Producer ConsumerA ConsumerB BROADCAST de-couple m2 m2

20. Q Producer ConsumerA ConsumerA horizontal scalability DISTRIBUTION

21. Q Producer ConsumerA ConsumerA m1 horizontal scalability DISTRIBUTION

22. Q Producer ConsumerA ConsumerA m1 horizontal scalability DISTRIBUTION m2

23. Q Producer ConsumerA ConsumerA m1 horizontal scalability DISTRIBUTION m2 m3

24. Q Producer ConsumerA ConsumerA horizontal scalability DISTRIBUTION m2 m3

25. Q Producer ConsumerA ConsumerA horizontal scalability DISTRIBUTION m2 m3

26. Q Producer ConsumerA ConsumerA fault tolerance FAILURE

27. Q Producer ConsumerA ConsumerA fault tolerance FAILURE m1

28. Q Producer ConsumerA ConsumerA fault tolerance FAILURE m2 m1

29. Q Producer ConsumerA ConsumerA fault tolerance FAILURE m2 m1

30. Q Producer ConsumerA fault tolerance FAILURE

31. Q Producer ConsumerA fault tolerance FAILURE m3

32. Q Producer ConsumerA fault tolerance FAILURE m3

33. Q Producer ConsumerA fault tolerance FAILURE m3

34. Q Producer ConsumerA fault tolerance FAILURE

35. Q Producer ConsumerA ConsumerA EVEN MORE FAILURE welp

36. Q Producer EVEN MORE FAILURE welp m1

37. Q Producer EVEN MORE FAILURE welp m2 m1

38. Q Producer EVEN MORE FAILURE welp m2 m3 m1

39. Q Producer EVEN MORE FAILURE welp m2 m3 m1

40. Q Producer EVEN MORE FAILURE welp m2 m3 m1

41. Q Producer ConsumerA EVEN MORE FAILURE welp m2 m3 m1

42. Q Producer ConsumerA EVEN MORE FAILURE welp m2 m3 m1

43. Q Producer ConsumerA EVEN MORE FAILURE welp m2 m3

44. Q Producer ConsumerA EVEN MORE FAILURE welp m3

45. Q Producer ConsumerA EVEN MORE FAILURE welp

46. NSQD

47. TOPICS AND CHANNELS •a topic is a distinct stream of

    messages •a topic has one or more channels •topics and channels are created at runtime •messages are pushed to consumers combine pubsub, distribution, and queueing nsqd “clicks” Topics

48. TOPICS AND CHANNELS •a topic is a distinct stream of

    messages •a topic has one or more channels •topics and channels are created at runtime •messages are pushed to consumers combine pubsub, distribution, and queueing nsqd “metrics” Channels “clicks” Topics

49. TOPICS AND CHANNELS •a topic is a distinct stream of

    messages •a topic has one or more channels •topics and channels are created at runtime •messages are pushed to consumers combine pubsub, distribution, and queueing nsqd “metrics” Channels “clicks” Topics “spam_analysis”

50. TOPICS AND CHANNELS •a topic is a distinct stream of

    messages •a topic has one or more channels •topics and channels are created at runtime •messages are pushed to consumers combine pubsub, distribution, and queueing nsqd “metrics” Channels “clicks” Topics “spam_analysis” “archive”

51. TOPICS AND CHANNELS •a topic is a distinct stream of

    messages •a topic has one or more channels •topics and channels are created at runtime •messages are pushed to consumers combine pubsub, distribution, and queueing nsqd “metrics” Channels “clicks” Topics “spam_analysis” “archive” Consumers

52. TOPICS AND CHANNELS •a topic is a distinct stream of

    messages •a topic has one or more channels •topics and channels are created at runtime •messages are pushed to consumers combine pubsub, distribution, and queueing nsqd “metrics” Channels “clicks” Topics “spam_analysis” “archive” Consumers

53. TOPICS AND CHANNELS •a topic is a distinct stream of

    messages •a topic has one or more channels •topics and channels are created at runtime •messages are pushed to consumers combine pubsub, distribution, and queueing nsqd “metrics” Channels “clicks” Topics “spam_analysis” “archive” Consumers

54. TOPICS AND CHANNELS •a topic is a distinct stream of

    messages •a topic has one or more channels •topics and channels are created at runtime •messages are pushed to consumers combine pubsub, distribution, and queueing nsqd “metrics” Channels “clicks” Topics “spam_analysis” “archive” Consumers A A A

55. TOPICS AND CHANNELS •a topic is a distinct stream of

    messages •a topic has one or more channels •topics and channels are created at runtime •messages are pushed to consumers combine pubsub, distribution, and queueing nsqd “metrics” Channels “clicks” Topics “spam_analysis” “archive” Consumers A A A

56. TOPICS AND CHANNELS •a topic is a distinct stream of

    messages •a topic has one or more channels •topics and channels are created at runtime •messages are pushed to consumers combine pubsub, distribution, and queueing nsqd “metrics” Channels “clicks” Topics “spam_analysis” “archive” Consumers A A A

57. TOPICS AND CHANNELS •a topic is a distinct stream of

    messages •a topic has one or more channels •topics and channels are created at runtime •messages are pushed to consumers combine pubsub, distribution, and queueing nsqd “metrics” Channels “clicks” Topics “spam_analysis” “archive” Consumers A A A

58. TOPICS AND CHANNELS •a topic is a distinct stream of

    messages •a topic has one or more channels •topics and channels are created at runtime •messages are pushed to consumers combine pubsub, distribution, and queueing nsqd “metrics” Channels “clicks” Topics “spam_analysis” “archive” Consumers A A A B B B

59. TOPICS AND CHANNELS •a topic is a distinct stream of

    messages •a topic has one or more channels •topics and channels are created at runtime •messages are pushed to consumers combine pubsub, distribution, and queueing nsqd “metrics” Channels “clicks” Topics “spam_analysis” “archive” Consumers A A A B B B

60. TOPICS AND CHANNELS •a topic is a distinct stream of

    messages •a topic has one or more channels •topics and channels are created at runtime •messages are pushed to consumers combine pubsub, distribution, and queueing nsqd “metrics” Channels “clicks” Topics “spam_analysis” “archive” Consumers A A A B B B

61. TOPICS AND CHANNELS •a topic is a distinct stream of

    messages •a topic has one or more channels •topics and channels are created at runtime •messages are pushed to consumers combine pubsub, distribution, and queueing nsqd “metrics” Channels “clicks” Topics “spam_analysis” “archive” Consumers A A A B B B

62. UNDER THE HOOD •topics and channels are independent •configurable high

    water mark
 (disk persistence) •“bounded” memory footprint DiskQueue In-Memory goroutine Input Chan Output Chan

63. THERE AND BACK AGAIN

64. DE-COUPLE •PUB locally to nsqd via HTTP •perform work async

    •co-locate everything (silo) API nsqd “metrics” channel “api_event” topic metrics consumer consumer

65. SCALE HORIZONTALLY API nsqd “metrics” channel API nsqd metrics consumer

    consumer consumer consumer “api_event” topic “api_event” topic “metrics” channel … …

66. A MESSAGE QUEUE IS BORING (IN ISOLATION)

67. NSQLOOKUPD

68. NSQ NSQD API consumer NSQ NSQD API NSQ NSQD API

    consumer nsqlookupd nsqlookupd TYPICAL NSQ CLUSTER •enable distributed and decentralized topologies •no centralized broker •nsqlookupd instances are independent (no coordinatation)

69. NSQ NSQD API consumer NSQ NSQD API NSQ NSQD API

    consumer nsqlookupd nsqlookupd PUBLISH TYPICAL NSQ CLUSTER •enable distributed and decentralized topologies •no centralized broker •nsqlookupd instances are independent (no coordinatation)

70. NSQ NSQD API consumer NSQ NSQD API NSQ NSQD API

    consumer nsqlookupd nsqlookupd PUBLISH REGISTER TYPICAL NSQ CLUSTER •enable distributed and decentralized topologies •no centralized broker •nsqlookupd instances are independent (no coordinatation)

71. NSQ NSQD API consumer NSQ NSQD API NSQ NSQD API

    consumer nsqlookupd nsqlookupd PUBLISH REGISTER DISCOVER TYPICAL NSQ CLUSTER •enable distributed and decentralized topologies •no centralized broker •nsqlookupd instances are independent (no coordinatation)

72. NSQ NSQD API consumer NSQ NSQD API NSQ NSQD API

    consumer nsqlookupd nsqlookupd PUBLISH REGISTER DISCOVER SUBSCRIBE TYPICAL NSQ CLUSTER •enable distributed and decentralized topologies •no centralized broker •nsqlookupd instances are independent (no coordinatation)

73. … AND AGAIN

74. DISCOVERABILITY •introduce nsqlookupd •producers and consumers
 come and go •other

    services can discover and subscribe to this topic API nsqd “metrics” channel “api_event” topic API nsqd “metrics” channel “api_event” topic

75. DISCOVERABILITY •introduce nsqlookupd •producers and consumers
 come and go •other

    services can discover and subscribe to this topic API nsqd “metrics” channel “api_event” topic API nsqd “metrics” channel “api_event” topic nsqlookupd TCP nsqlookupd TCP

76. DISCOVERABILITY •introduce nsqlookupd •producers and consumers
 come and go •other

    services can discover and subscribe to this topic API nsqd “metrics” channel “api_event” topic API nsqd “metrics” channel “api_event” topic nsq_to_file nsq_to_file nsqlookupd TCP nsqlookupd TCP “archive” channel

77. DISCOVERABILITY •introduce nsqlookupd •producers and consumers
 come and go •other

    services can discover and subscribe to this topic API nsqd “metrics” channel “api_event” topic API nsqd “metrics” channel “api_event” topic nsq_to_file nsq_to_file nsqlookupd TCP nsqlookupd TCP “archive” channel 1 - query nsqlookupd! for topic “api_event” HTTP

78. DISCOVERABILITY •introduce nsqlookupd •producers and consumers
 come and go •other

    services can discover and subscribe to this topic API nsqd “metrics” channel “api_event” topic API nsqd “metrics” channel “api_event” topic 2 - connect to all discovered
 producers nsq_to_file nsq_to_file nsqlookupd TCP nsqlookupd TCP “archive” channel 1 - query nsqlookupd! for topic “api_event” HTTP

79. TO INFINITY AND BEYOND nsqlookupd nsqd Datacenter A

80. TO INFINITY AND BEYOND nsqlookupd nsqd Datacenter A nsq_to_nsq

81. TO INFINITY AND BEYOND nsqlookupd nsqlookupd nsqd nsqd Datacenter A

    Datacenter B nsq_to_nsq

82. TO INFINITY AND BEYOND nsqlookupd nsqlookupd nsqd nsqd Datacenter A

    Datacenter B nsq_to_nsq

83. TO INFINITY AND BEYOND nsqlookupd nsqlookupd nsqd nsqd Datacenter A

    Datacenter B nsq_to_nsq

84. OPS

85. GUARANTEES •messages are delivered at least once •messages are not

    durable (by default) •messages received are un-ordered •consumers eventually find all topic producers

86. BACK PRESSSSSSSSSSSSURE •clients control flow - RDY state •concurrency knob

    •processing fails? •slow down rate •bad message? •limit attempts •delay re-processing Incoming Msgs Msg Handler Timeout - - Check Timeout Full Throttle success Timeout == 0 RDY N

87. BACK PRESSSSSSSSSSSSURE •clients control flow - RDY state •concurrency knob

    •processing fails? •slow down rate •bad message? •limit attempts •delay re-processing Incoming Msgs Msg Handler Timeout ++ Timeout - - Check Timeout Full Throttle success fail Timeout == 0 RDY N

88. BACK PRESSSSSSSSSSSSURE •clients control flow - RDY state •concurrency knob

    •processing fails? •slow down rate •bad message? •limit attempts •delay re-processing Incoming Msgs Msg Handler Pause Timeout ++ Timeout - - Check Timeout Full Throttle success fail Timeout == 0 RDY N RDY 0

89. BACK PRESSSSSSSSSSSSURE •clients control flow - RDY state •concurrency knob

    •processing fails? •slow down rate •bad message? •limit attempts •delay re-processing Incoming Msgs Msg Handler Backoff Expired Pause Timeout ++ Timeout - - Check Timeout Full Throttle success fail Timeout == 0 RDY N RDY 0

90. BACK PRESSSSSSSSSSSSURE •clients control flow - RDY state •concurrency knob

    •processing fails? •slow down rate •bad message? •limit attempts •delay re-processing Incoming Msgs Msg Handler Test Water Backoff Expired Pause Timeout ++ Timeout - - Check Timeout Full Throttle success fail Timeout == 0 RDY N RDY 0 RDY 1

91. BACK PRESSSSSSSSSSSSURE •clients control flow - RDY state •concurrency knob

    •processing fails? •slow down rate •bad message? •limit attempts •delay re-processing Incoming Msgs Msg Handler Test Water Backoff Expired Pause Timeout ++ Timeout - - Check Timeout Full Throttle success fail Timeout > 0 Timeout == 0 RDY N RDY 0 RDY 1

92. TOPIC & CHANNEL PAUSING “clicks” “metrics” “reporting” “achrive” C
 A


    C
 H
 E DB

93. TOPIC & CHANNEL PAUSING “clicks” “metrics” “reporting” “achrive” C
 A


    C
 H
 E DB A A A

94. TOPIC & CHANNEL PAUSING “clicks” “metrics” “reporting” “achrive” C
 A


    C
 H
 E DB A A A B B B

95. TOPIC & CHANNEL PAUSING “clicks” “metrics” “reporting” “achrive” C
 A


    C
 H
 E DB A A A B B B

96. TOPIC & CHANNEL PAUSING “clicks” “metrics” “reporting” “achrive” C
 A


    C
 H
 E DB B B B

97. TOPIC & CHANNEL PAUSING “clicks” “metrics” “reporting” “achrive” C
 A


    C
 H
 E DB

98. TOPIC & CHANNEL PAUSING “clicks” “metrics” “reporting” “achrive” paused C


    A
 C
 H
 E DB C D C C D D

99. TOPIC & CHANNEL PAUSING “clicks” “metrics” “reporting” paused C
 A


    C
 H
 E DB “archive” C D C C D D

100. TOPIC & CHANNEL PAUSING “clicks” “metrics” “reporting” C
 A
 C


     H
 E DB “archive” C D C C D D

101. TOPIC & CHANNEL PAUSING “clicks” “metrics” “reporting” C
 A
 C


     H
 E DB “archive” C D C C D D

102. TOPIC & CHANNEL PAUSING “clicks” “metrics” “reporting” C
 A
 C


     H
 E DB “archive” C D C C D D

103. TOPIC & CHANNEL PAUSING “clicks” “metrics” “reporting” C
 A
 C


     H
 E DB “archive” D D D

104. TOPIC & CHANNEL PAUSING “clicks” “metrics” “reporting” C
 A
 C


     H
 E DB “archive”

105. TOPIC & CHANNEL PAUSING “clicks” “metrics” “reporting” paused F F

     F C
 A
 C
 H
 E DB “archive”

106. TOPIC & CHANNEL PAUSING “clicks” “metrics” “reporting” paused F F

     F C
 A
 C
 H
 E DB “archive” G G G

107. TOPIC & CHANNEL PAUSING “clicks” “metrics” “reporting” paused F F

     F C
 A
 C
 H
 E DB “archive” G G G

108. TOPIC & CHANNEL PAUSING “clicks” “metrics” “reporting” paused F C


     A
 C
 H
 E DB “archive” G G G

109. TOPIC & CHANNEL PAUSING “clicks” “metrics” “reporting” paused F C


     A
 C
 H
 E DB “archive” G

110. TOPIC & CHANNEL PAUSING “clicks” “metrics” “reporting” C
 A
 C


     H
 E DB “archive”

111. ONE MORE THING •#ephemeral channels •channel sampling •TLS / Snappy

     •telemetry over statsd / HTTP

112. SUMMARY •github.com/bitly/nsq •10,300 lines of Go •19 client libraries, 11

     languages •over 2 years in production

113. SUMMARY •github.com/bitly/nsq •10,300 lines of Go •19 client libraries, 11

     languages •over 2 years in production

114. IN PRODUCTION heavy water

115. Thanks! @imsnakes ! https://github.com/bitly/nsq ! shoutout to @jehiah (co-author of

     NSQ)