Scale with NSQ: a realtime distributed messaging system

Talk given at Berlin Buzzwords 2015

Transcript

1. SCALE WITH NSQ a realtime distributed messaging platform @GeorgiCodes Georgi Knox - Software Engineer @ Bitly Berlin Buzzwords June 2, 2015
2. G’DAY, MY NAME IS GEORGI @GeorgiCodes
3. WHAT IS BITLY? •Popular URL shortener •analytics around how and where those links were shared •10 billion clicks per month •8000 requests per second
4. WHAT IS NSQ? • At version: 0.3.5 •Open-sourced by Bitly •written in Go •19 client libraries, 11 languages • > 3 years in production
5. GROWING PAINS •single host •synchronous writes •new feature: track metrics API metrics interweb 1 2 3
6. MESSAGING PATTERNS
7. PS Producer ConsumerA ConsumerB BROADCAST de-coupling of producers and consumers multicast: message copied and delivered to n consumers m1 m1 m2 m2
8. queue Producer ConsumerA ConsumerA m1 horizontal scalability DISTRIBUTION m2 m3
9. queue Producer ConsumerA ConsumerA fault tolerance FAILURE m2 m1 m3
10. queue Producer ConsumerA ConsumerA EVEN MORE FAILURE when all the things fail m2 m3 m1
11. NSQD
12. TOPICS AND CHANNELS •a topic is a distinct stream of messages •a topic has one or more channels •topics and channels are created at runtime •messages are pushed to consumers combine pubsub, distribution, and queueing nsqd “metrics” Channels “clicks” Topics “spam_analysis” “archive” Consumers A A A B B B
13. UNDER THE HOOD •topics and channels are independent •configurable high water mark
 (disk persistence) DiskQueue In-Memory goroutine Input Chan Output Chan
14. OUR SIMPLE EXAMPLE + NSQD •introduce nsqd •de-coupled production and consumption of data •PUB locally to nsqd via HTTP •perform work async •co-locate everything (silo) API nsqd “metrics” channel “api_event” topic metrics consumer consumer
15. SCALE HORIZONTALLY API nsqd “metrics” channel API nsqd metrics consumer consumer consumer consumer “api_event” topic “api_event” topic “metrics” channel … …
16. NSQLOOKUPD
17. NSQ NSQD API consumer NSQ NSQD API NSQ NSQD API consumer nsqlookupd nsqlookupd PUBLISH REGISTER DISCOVER SUBSCRIBE TYPICAL NSQ CLUSTER •enable distributed and decentralized topologies •no centralized broker •nsqlookupd instances are independent (no coordinatation)
18. OUR SIMPLE EXAMPLE + NSQLOOKUPD •introduce nsqlookupd •discoverability •producers and consumers
 come and go •other services can discover and subscribe to this topic API nsqd “metrics” channel “api_event” topic API nsqd “metrics” channel “api_event” topic 2 - connect to all discovered
 producers nsq_to_file nsq_to_file nsqlookupd TCP nsqlookupd TCP “archive” channel 1 - query nsqlookupd for topic “api_event” HTTP
19. GUARANTEES •messages are delivered at least once •messages are not durable (by default) •messages received are un-ordered •consumers eventually find all topic producers
20. TOPIC & CHANNEL PAUSING “clicks” “metrics” “reporting” “achrive” paused paused F F F C
 A
 C
 H
 E DB “archive” A A A B B B C D C C D D G G G
21. NSQ ADMIN •#ephemeral channels
22. THANK-YOU!
 @GeorgiCodes big thanks to @jehiah and @imsnakes (authors of NSQ)