Streaming Your Way to Web Scale: Scaling Bitly via Stream-Based Processing

In this talk, I will cover how a web site naturally evolves from a simple web application, through the process of adding an asynchronous job queue, to a fully stream-based and service-oriented architecture, as implemented at Bitly. We will explore how Bitly arrived at this architecture, the pros and cons of operating a streaming-based system, and examine some of the tools we built to make our service possible, particularly NSQ (https://github.com/bitly/nsq/), our message queue system.

Transcript

1. 16 x 9 Streaming Your Way to Web Scale: Scaling Bitly via Stream-Based Processing! Surge! September 25, 2014, 2:30pm 85  slides   23  Images   21  diagrams
2. Peter Herndon! peter@bit.ly! @tpherndon backend developer for Bitly
3. Scaling Bitly via Stream-Based Processing
4. or
5. Streaming Your Way to Web Scale
7. http://www.mongodb-is-web-scale.com/! (**NSFW due to language)
8. Define streaming — public API surface that puts events into message queues that are consumed by web services
9. Need to be asynchronous
10. we use Tornado
11. and Go
12. we had an older queue, but replaced it with a newer one.
13. written in Go
14. command v. event messages
15. Notification of an event
16. allows interested listeners to respond as they require
17. λ Lambda architecture - batch + stream processing
18. Part Two, NSQ
19. NSQ http://nsq.io
20. Look, Ma, real code! https://github.com/bitly/nsq https://github.com/bitly/go-nsq https://github.com/bitly/pynsq https://github.com/jehiah/nsqauth-contrib
21. Servers! • nsqd! • nsqlookupd! • nsqadmin! • pynsqauthd Parts is parts! (part 1)
22. Clients! • go-nsq! • pynsq Parts is parts! (part 2)
23. Utilities! • nsq_tail! • nsq_to_file! • nsq_to_nsq! • nsq_stat Parts is parts! (part 3)
24. two basic building blocks of distributed NSQ
25. nsqd & nsqlookupd put into context, look at evolution of a website
26. Basic Web App Web App Database Basic web app. In Python, Django + Postgres, Flask + Postgres, Tornado + Postgres
27. Scaling the Mountain (of Load) Web App Database Web App Web App First bottleneck: web layer
28. Cache Rules Everything Around Me Database Web App Web App Web App Cache Remove web layer bottleneck, next is DB, so add caching layer
29. You Want Me to Replicate You Database Database Web App Web App Web App Cache Works for a while, but DB requests still take too long, so replicate
30. Shards Here, Shards There Web App Web App Web App Cache Database Database Database Database Database Database …and then shard
31. It’s Off to Work I Go! Database Web App Cache Queue Worker But individual requests still take too long, because doing too much work. So add message queue and worker. In Python, Celery
32. Message From a Bottle(.py) Database Web App Cache Queue Worker Web app sends messages to queue
33. Working the Event Database Web App Cache Queue Worker Worker pulls message off queue and processes
34. Write Here, Write Now Database Web App Cache Queue Worker Worker writes results to database, file system, etc.
35. Write Here, Write Now (redux) Web App Database Worker Queue Instead, imagine worker writes results
36. Sending Out an SMS Web App Database Worker Queue and web app writes event messages to queue local to the web service
37. Listen, listen, LISTEN Web App Database Worker Queue Queue but worker is listening to a queue running on another server
38. Workin’ On a Chain(ed) Gang Web App Database Worker Queue Web App Database Worker Queue Web App Database Worker Queue
39. Look it up! Web App Database Worker Queue Queue Worker finds queue with topic via nsqlookupd
40. if  __name__  ==  "__main__":          tornado.options.parse_command_line()          logatron_client.setup()                    Reader(                  topic=settings.get('nsqd_output_topic'),                  channel='queuereader_spam_metrics',                  validate_method=validate_message,                  message_handler=count_spam_actions,                  lookupd_http_addresses=settings.get('nsq_lookupd')          )          run() /<service>/queuereader_<service>.py How do I find things?
41. Sending Out an SMS Web App Database Worker Queue topic: ‘spam_api’ First time app writes to a TOPIC in the local nsqd
42. Sending Out an SMS Web App Database Worker Queue topic: ‘spam_api’ nsqlookupd nsqd creates the topic and registers it with nsqlookupd
43. Where Am I Again? Web App Database Worker nsqd topic: 'spam_counter' nsqd topic: 'spam_api' nsqlookupd topic: 'spam_api'? Worker in another service looking for a topic asks nsqlookupd, replies with address
44. Talkin’ ‘Bout Something Web App Database Worker nsqd topic: 'spam_counter' nsqd topic: 'spam_api' channel: 'spam_counter' queuereader connects to nsqd, registers a channel
45. Cross-Town Traffic nsqd topic: 'spam_api' Worker Worker Worker channel: 'spam_counter' messages are divided by # of subscribers to a channel; allows horizontal scaling
46. Channeling the Ghost nsqd topic: 'spam_api' Worker Worker Worker channel: 'spam_counter' Worker channel: 'nsq_to_file'' full copy of all messages to each channel
48. nsqadmin
49. nsqadmin How we manage our message queues
50. nsqadmin
51. nsqadmin
53. pynsqauthd
54. It’s made of PEOPLE! https://github.com/jehiah/nsqauth-contrib
58. pynsq client
59. It’s still made of PEOPLE! https://github.com/bitly/pynsq
60. • settings.py! • <service>_api.py! • queuereader_<service>.py! • README.md /<service> Queuereaders are part of streaming architecture
61. if  __name__  ==  "__main__":          tornado.options.parse_command_line()          logatron_client.setup()                    Reader(                  topic=settings.get('nsqd_output_topic'),                  channel='queuereader_spam_metrics',                  validate_method=validate_message,                  message_handler=count_spam_actions,                  lookupd_http_addresses=settings.get('nsq_lookupd')        )          run() /<service>/queuereader_<service>.py, 1 of 4
62. if  __name__  ==  "__main__":          tornado.options.parse_command_line()          logatron_client.setup()                    Reader(                  topic=settings.get('nsqd_output_topic'),                  channel='queuereader_spam_metrics',                  validate_method=validate_message,                  message_handler=count_spam_actions,                  lookupd_http_addresses=settings.get('nsq_lookupd')        )          run() /<service>/queuereader_<service>.py, 1 of 4
63. def  validate_message(message):          if  message.get('o')  ==  '+'  and  message.get('l'):                  return  True          if  message.get('o')  ==  '-­‐'  and  message.get('l')\            and  message.get('bl'):                     return  True          return  False /<service>/queuereader_<service>.py, 2 of 4
64. if  __name__  ==  "__main__":          tornado.options.parse_command_line()          logatron_client.setup()                    Reader(                  topic=settings.get('nsqd_output_topic'),                  channel='queuereader_spam_metrics',                  validate_method=validate_message,                  message_handler=count_spam_actions,                  lookupd_http_addresses=settings.get('nsq_lookupd')        )          run() /<service>/queuereader_<service>.py, 1 of 4
65. def  count_spam_actions(message,  nsq_msg):          key_section  =  statsd_keys[message['o']]          key  =  key_section.get(message['l'],  key_section['default'])          statsd.incr(key)                    if  key  ==  'remove_by_manual':                  key_section  =  statsd_keys['-­‐manual']                  key  =  key_section.get(message['bl'],  key_section['default'])                  statsd.incr(key)                    return  nsq_msg.finish() /<service>/queuereader_<service>.py, 3 of 4
66. def  count_spam_actions(message,  nsq_msg):          key_section  =  statsd_keys[message['o']]          key  =  key_section.get(message['l'],  key_section['default'])          statsd.incr(key)                    if  key  ==  'remove_by_manual':                  key_section  =  statsd_keys['-­‐manual']                  key  =  key_section.get(message['bl'],  key_section['default'])                  statsd.incr(key)                    return  nsq_msg.finish() /<service>/queuereader_<service>.py, 3 of 4
67. if  __name__  ==  "__main__":          tornado.options.parse_command_line()          logatron_client.setup()                    Reader(                  topic=settings.get('nsqd_output_topic'),                  channel='queuereader_spam_metrics',                  validate_method=validate_message,                  message_handler=count_spam_actions,                  lookupd_http_addresses=settings.get('nsq_lookupd')        )          run() /<service>/queuereader_<service>.py, 4 of 4
68. if  __name__  ==  "__main__":          tornado.options.parse_command_line()          logatron_client.setup()                    Reader(                  topic=settings.get('nsqd_output_topic'),                  channel='queuereader_spam_metrics',                  validate_method=validate_message,                  message_handler=count_spam_actions,                  lookupd_http_addresses=settings.get('nsq_lookupd')        )          run() /<service>/queuereader_<service>.py, 4 of 4
70. utilities
71. nsq_tail! nsq_to_file! to_nsq! nsq_to_nsq! nsq_stat Parts is parts! (part 3, redux)
72. Features & Guarantees! (aka Trade-Offs) Distributed, No SPOF || Horizontally Scalable || TLS || statsd integration || Easy to Deploy || Cluster Administration
73. Messages NOT Durable
74. Delivered at least once
75. Un-ordered Delivery
76. Eventually-Consistent Discovery
77. A Thousand Points of Light (well, 58)
78. A Thousand Points of Light (well, 58)
79. Whoa… 8.2 billion decodes per month
80. Streaming Architecture Easy to build new services Easy to scale individual components horizontally Durable in the face of single component failure Distributed
81. THINGS TO THINK ABOUT Monitoring, monitoring, monitoring Failure modes — how can things fail? How does your application as a whole handle the failure of individual components? Measurement — metrics show the range Timeouts — connection timeouts, DNS timeouts — a slow network is the same as a failed service
82. NSQ http://nsq.io
83. Duke of URL https://github.com/bitly/nsq https://github.com/bitly/go-nsq https://github.com/bitly/pynsq https://github.com/jehiah/nsqauth-contrib
84. Web Scale - http://www.mongodb-is-web-scale.com! Waterfall - https://www.flickr.com/photos/desatur8/14949285342! Tornado - https://www.flickr.com/photos/indigente/798304! John de Lancie - https://www.flickr.com/photos/cayusa/1394930005! Ben Whishaw - https://www.flickr.com/photos/rossendalewadey/6032496676! Command Key - https://www.flickr.com/photos/klash/3175479797! iPhone6 Event - https://www.flickr.com/photos/notionscapital/15067798867! Wait for iPhone - https://www.flickr.com/photos/josh_gray/662814907! NSQ Logo - http://nsq.io! ! All other photos by T. Peter Herndon! ! Photo Credits
85. Questions?
86. Peter Herndon! peter@bit.ly! @tpherndon