Scaling Realtime at DISQUS

tl;dr: #Python #Architecture #Scalability #DISQUS

What does it take to add realtime functionality to a truly “web scale” app. The result is the DISQUS realtime system, a highly concurrent system for allowing web clients to subscribe to arbitrary events in the DISQUS infrastructure.

Transcript

1. Adam Hitchcock @NorthIsUp Scaling Realtime at DISQUS Thursday, March 21, 13 i am an infrastructure engineer at disqus focus on realtime/firehose but also a lot of other small service-y things.
2. Thursday, March 21, 13 A common misconception
3. Adam Hitchcock @NorthIsUp Scaling Realtime at DISQUS Thursday, March 21, 13 thanks for watching me instead of the other talks going on right now.
4. we’re hiring disqus.com/jobs If this is interesting to you... Thursday, March 21, 13 round peg + square hole == we want you! We are looking for interesting people, not necessarily ones who fit in to the boxes on our jobs page. If this is interesting, please come talk to me!
5. what is DISQUS? Thursday, March 21, 13 How many of you will raise your hand? How many of you are python people? How many of you know what disqus is?
6. Thursday, March 21, 13 - community platform - main product is a comment widget (javascript)
7. why do realtime? ๏ getting new data to the user asap ๏ for increased engagement ๏ and it looks awesome ๏ and we can sell (or trade) it Thursday, March 21, 13 We define this as ‘less than 10 seconds’ but my goal was less than one.
8. http://github.com/NorthIsUp/orbital2 http://map.labs.disqus.com Thursday, March 21, 13
9. DISQUS sees a lot of tra c Google Analytics: Feb 2013 - March 2012 Thursday, March 21, 13 the problem is that our old realtime system wouldn’t scale - at max capacity - old system less than 100 thousand concurrent users
10. realertime ๏ currently active on all DISQUS sites ๏ tested ‘dark’ on our existing network ๏ during testing: ๏ 1.5 million concurrently connected users ๏ 45 thousand new connections per second ๏ 165 thousand messages/second ๏ <.2 seconds latency end to end Thursday, March 21, 13 - realtime replaced by realERtime - i’ll re-visit on what dark means later on the testing slides - describe heavy tail distribution of popularity - end to end does NOT include the DISQUS app DEMO IT then “so how did we build this?”
11. so, how did we do it? Thursday, March 21, 13
12. Node.js and MongoDB! Thursday, March 21, 13
13. Node.js and MongoDB! Thursday, March 21, 13
14. This is PyCon. We used Python. Thursday, March 21, 13
15. and some other Technology You Know™ Thursday, March 21, 13 we are a small team, we don’t have time to NIH all the things python is our normal django site + some formatting and processing code
16. thoonk redis queue some python glue nginx push stream and long(er) polling Thursday, March 21, 13 HANDS FOR: - redis - PUB SUB - nginx - thoonk OFF THE SHELF
17. architecture overview Thursday, March 21, 13
18. old-june memcache New Posts memcache DISQUS embed clients DISQUS poll memcache ever 5 seconds Thursday, March 21, 13 Old system that wouldn’t scale
19. june-july redis pub/sub New Posts redis pub/sub DISQUS embed clients DISQUS HA Proxy Flask FE cluster Thursday, March 21, 13 iteration 1: - 165K messages/second - got us past launch - rampant inefficiency
20. HA Proxy july-october Flask FE cluster redis queue “python glue” Gevent server New Posts redis pub/sub DISQUS embed clients redis pub/sub DISQUS “python glue” Gevent server Thursday, March 21, 13 iteration 2: - up to ~1.5 milion concurrent users - moved all the hard work to earlier in the pipeline - very happy with it
21. HA Proxy august-october Flask FE cluster redis queue “python glue” Gevent server New Posts redis pub/sub DISQUS embed clients redis pub/sub DISQUS “python glue” Gevent server 2 servers 14 BIG servers 6 servers 5 servers Thursday, March 21, 13 also all those arrows on the right == a lot of sockets
22. HA Proxy august-october Flask FE cluster redis queue “python glue” Gevent server New Posts redis pub/sub DISQUS embed clients redis pub/sub DISQUS “python glue” Gevent server 2 servers 6 servers 5 servers 2 for redundancy 14 BIG servers lots of servers, we can do better Thursday, March 21, 13
23. “python glue” Gevent server october-now nginx + push stream module redis queue New Posts ngnix pub endpoint DISQUS embed clients http post DISQUS Thursday, March 21, 13 replaced flask + haproxy + redis with nginx push-stream-module
24. “python glue” Gevent server october-now nginx + push stream module redis queue New Posts ngnix pub endpoint DISQUS embed clients http post DISQUS 2 servers 5 servers Why still 5 for this? Network memory restriction, we can’t fix this without kernel hacking, tweaking, etc. (if you know how, tell us, then apply for a job, then fix it for us) Thursday, March 21, 13 - c1Million problem
25. october-now django Formatter Publishers thoonk queue http post ngnix pub endpoint DISQUS embed clients other realtime stu nginx + push stream module New Posts Thursday, March 21, 13 Looking into the tech in these components...
26. thoonk redis queue some python glue nginx push stream and long(er) polling Thursday, March 21, 13 python is our normal django site + some formatting and processing code redis is thoonk queue
27. the thoonk queue ๏ django post_save and post_delete hooks ๏ thoonk is a queue on top of redis ๏ implemented as a DFA ๏ provides job semantics ๏ useful for end to end acking ๏ reliable job processing in distributed system ๏ did I mention it’s on top of redis? ๏ uses zset to store items == ranged queries Thursday, March 21, 13
28. thoonk redis queue some python glue nginx push stream and long(er) polling Thursday, March 21, 13
29. the python glue ๏ listens to a thoonk queue ๏ cleans & formats message ๏ this is the final format for end clients ๏ compress data now ๏ publish message to nginx and other firehoses ๏ forum:id, thread:id, user:id, post:id Formatter Publishers Thursday, March 21, 13 - django post save & post delete signals - thoonk was easy and fun! - end to end ack via thoonk. (not removed until fully published to nginx) - allows for multiple publishers, we publish to nginx, pubsubhubbub, commercial consumers.
30. gevent is nice # the code is too big to show here, so just import it # http://bitly.com/geventspawn from realertime.lib.spawn import Watchdog from realertime.lib.spawn import TimeSensitiveBackoff Thursday, March 21, 13 - /lots/ of greenlets, need a way to manage them NAY! manage themselves - sleep(0) == canonical yield in gevent, just an FYI
31. data pipelines class Pipeline(object): def parse(self, data): raise NotImplemented('No ParserMixin used') def compute(self, data, parsed_data): raise NotImplemented('No ComputeMixin used') def publish(self, data, parsed_data, computed_data): raise NotImplemented('No PublisherMixin used') def handle(self, data): parsed_data = self.parse(data) computed_data = self.compute(data, parsed_data) return self.publish(data, parsed_data, computed_data) Thursday, March 21, 13 Each message for a pipeline is in its own green thread since 99% of the time the publish step is i/o bound.
32. Example Mixins class JSONParserMixin(Pipeline): def parse(self, data): return json.loads(data) class AnnomizeDataMixin(Pipeline): def compute(self, data, parsed_data): return {} class SuperSecureEncryptDataMixin(Pipeline): def compute(self, data, parsed_data): return parsed_data.encode('rot13') class HTTPPublisher(Pipeline): def publish(self, data, parsed_data, computed_data): u = urllib2.urlopen(self.dat_url, computed_data) return u class FilePublisher(Pipeline): def publish(self, data, parsed_data, computed_data): with open(self.output, 'a') as f: f.write(computed_data) Thursday, March 21, 13
33. Finished Pipeline class JSONAnnonHTTPPipeline( JSONParserMixin, AnnomizeDataMixin, HTTPPublisherMixin): pass class JSONSecureHTTPPipeline( JSONParserMixin, SuperSecureEncyptionMixin, HTTPPublisherMixin): pass class JSONAnnonFilePipeline( JSONParserMixin, AnnomizeDataMixin, FilePublisherMixin): pass Thursday, March 21, 13 Composing for fun and profit: - easier to reason about - sort of declarative (again reasoning) - testing is done in the same way: composing test classes
34. real live DISQUS code class FEOrbitalNginxMultiplexer( SchemaTransformerMixin, JSONFormatterMixin, SelfChannelsMixin, HTTPPublisherMixin): def __init__(self, domains, api_version=1): schema_namespace = 'orbital' self.channels = ('orbital', ) super(FEOrbitalNginxMultiplexer, self).__init__(domains=domain class FEPublicAckingMultiplexer( PublicTransformerMixin, JSONFormatterMixin, FEChannelsMixin, ThoonkQueuePubSubPublisherMixin): def __init__(self, domains, api_version): schema_namespace = 'general' super(FEPublicAckingMultiplexer, self).__init__(domains=domain Thursday, March 21, 13
35. thoonk redis queue some python glue nginx push stream and long(er) polling Thursday, March 21, 13
36. nginx push stream ๏ follow John Watson (@wizputer) for updated #humblebrags as we ramp up tra c ๏ an example config can be found here: http://bit.ly/disqus-nginx-push-stream http://wiki.nginx.org/HttpPushStreamModule Thursday, March 21, 13
37. nginx push stream ๏ Replaced webservers and Redis Pub/Sub ๏ But starting with Pub/Sub was important for us ๏ Encouraged us to over publish on keys Thursday, March 21, 13
38. nginx push stream ๏ Turned on for 70% of our network... ๏ ~950K subscribers (peak single machine) ๏ peak 40 MBytes/second (per machine) ๏ CPU usage is still well under 15% ๏ 99.845% active writes (the socket is written to often enough to come up as ACTIVE) http://wiki.nginx.org/HttpPushStreamModule Thursday, March 21, 13
39. config push stream location = /pub { allow 127.0.0.1; deny all; push_stream_publisher admin; set $push_stream_channel_id $arg_channel; } location ^~ /sub/ { # to maintain api compatibility we need this location ~ /sub/(.*)/(.*)$ { # Url encoding things? $1%3A2$2 set $push_stream_channels_path $1:$2; push_stream_subscriber streaming; push_stream_content_type application/json; } } http://wiki.nginx.org/HttpPushStreamModule Thursday, March 21, 13 and nginx does the rest WARNING, libraries like requests will automatically urlencode things like colons
40. examples # Subs curl -s 'localhost/sub/forum/cnn' curl -s 'localhost/sub/thread/907824578' curl -s 'localhost/sub/user/northisup' # Pubs curl -s -X POST 'localhost/pub?channel=forum:cnn' \ -d '{"some sort": "of json data"}' curl -s -X POST 'localhost/pub?channel=thread:907824578' \ -d '{"more": "json data"}' curl -s -X POST 'localhost/pub?channel=user:northisup' \ -d '{"the idea": "I think you get it by now"}' http://wiki.nginx.org/HttpPushStreamModule Thursday, March 21, 13
41. measure nginx location = /push-stream-status { allow 127.0.0.1; deny all; push_stream_channels_statistics; set $push_stream_channel_id $arg_channel; } http://wiki.nginx.org/HttpPushStreamModule Thursday, March 21, 13 actually used this to build a realtime stream of popular threads on disqus
42. thoonk redis queue some python glue nginx push stream and long(er) polling Thursday, March 21, 13
43. long(er) polling onProgress: function () { var self = this; var resp = self.xhr.responseText; var advance = 0; var rows; // If server didn't push anything new, do nothing. if (!resp || self.len === resp.length) return; // Server returns JSON objects, one per line. rows = resp.slice(self.len).split('\n'); _.each(rows, function (obj) { advance += (obj.length + 1); obj = JSON.parse(obj); self.trigger('progress', obj); }); self.len += advance; } Thursday, March 21, 13 Burak yigit Kaya @madBYK because on a busy thread this matters, 99% of the time, doesn’t matter (IGN E3) - peak post rate ~40 msg/sec - peak delivery ~164K msg/sec
44. Soon... EventSource // Currently EventSource has CORS issues ev = EventSource(dat_url); ev.addEventListener("Post", handlePostEvent); // EventSource wire format id: 1234 event: Post data: {"some sort": "of json data"} // what handlePostEvent sees var handlePostEvent = function (event) { event.data == 1234; event.data == '{"some sort": "of json data"}'; }); Thursday, March 21, 13 yeah, that is way easier. WARNING! CORS issues now, on track to be fixed just need to set the headers correctly on the inbound messages. DISQUS is talking more about eventsource at “html5 dev conf”
45. test, measure, repeat Thursday, March 21, 13
46. test ๏ Darktime ๏ use existing network to load test ๏ (user complaints when it didn’t work...) ๏ Darkesttime ๏ load testing a single thread ๏ have knobs you can twiddle Thursday, March 21, 13
47. measure ๏ measure all the things! ๏ especially when the numbers don’t line up ๏ measuring is hard in distributed systems ๏ try to express things as +1 and -1 if you can ๏ Sentry for measuring exceptions Thursday, March 21, 13 so when you have a talk you #humblebrag like “peak delivery rate” and stuff scales! gauges and aggregation
48. pretty graphs Thursday, March 21, 13 because, you know, graphs are cool
49. how does it really scale? POPE white smoke francis announced Thursday, March 21, 13 first was the white smoke, second was felix 245 MBytes/second. 6TB over the day. only 12% cpu
50. maths Thursday, March 21, 13
51. it’s been a busy few weeks Thursday, March 21, 13
52. wha? ๏ People do weird stu with your stu ๏ turned o this server in Oct 2012 ๏ Still getting 100 req/sec Thursday, March 21, 13
53. lessons ๏ do hard (computation) work early ๏ end-to-end acks are good, but expensive ๏ redis/nginx pubsub is e ectively free Thursday, March 21, 13 - data processing and json formatting done once not 1000x times - gziping done once not 1000x times - defer setting up the work in the generator until as late as possible - ditched e2e acks from the fe, cost way too much
54. If this was interesting to you... psst, we’re hiring disqus.com/jobs Thursday, March 21, 13 we have jobs on jobs page, but we also have jobs we don’t we make jobs for talented peopl
55. special thanks ๏ the team at DISQUS ๏ like je a.k.a. @nfluxx who had to review all my code ๏ and especially our dev-ops guys ๏ like john watson a.k.a. @wizputer who found the nginx-push-stream module psst, we’re hiring disqus.com/jobs Thursday, March 21, 13
56. slide full o’ links ๏ Nginx push stream module http://wiki.nginx.org/HttpPushStreamModule ๏ Thoonk (redis queue) http://github.com/andyet/thoonk.py ๏ Sentry (distributed traceback aggregation) http://github.com/dcramer/sentry ๏ Gevent (python coroutines and greenlets) http://gevent.org/ ๏ Scales (in-app metrics) http://github.com/Greplin/scales code.disqus.com Thursday, March 21, 13
57. Come find me here! PyCon 2013 Santa Clara Convention Center Hall A-B Santa Clara, CA 10’x15’ 10’x15’ 10’x15’ 10’x15’ 10’x15’ 10’x15’ 10’x15’ 10’x15’ 10’x15’ 10’x15’ 10’x15’ 10’x15’ HS ARE 10’x10’ UNLESS NOTED 5’ BOOTHS 0’ BOOTHS 0’ BOOTH BOOTH S ARE 10’ UNLESS NOTED 20’ 20’ 8’ 8’ LUNCH & BREAKS 10’x15’ 10’x15’ 10’x15’ 10’x15’ 10’x15’ 10’x15’ 10’x15’ 10’x15’ 8’ 20’ 20’ 10’ 20’ 19’ Revised 1/9/2013 10’x15’ 10’x15’ 10’x15’ 10’x15’ 10’x15’ 10’x15’ 10’x15’ 10’x15’ 10’x15’ 10’x15’ 10’x15’ 10’x15’ 10’x20’ 10’x15’ 10’x15’ 10’x20’ 10’x15’ 10’x15’ 10’x15’ 10’x15’ 10’x15’ 10’x15’ 10’x15’ 10’x15’ 8’x20’ 10’x15’ 10’x15’ 10’x15’ 10’x15’ 10’x15’ 10’x15’ 10’x15’ 10’x15’ Thursday, March 21, 13
58. we are still hiring psst, we’re hiring disqus.com/jobs Thursday, March 21, 13
59. Questions I have ๏ What is the best kernel config for webscale concurrency. Nginx? ๏ I <3 gevent, but what if I want to pypy? ๏ Nginx + lua? Seems kind of awesome. ๏ Composing data pipelines: good or bad? ๏ I didn’t have time to mention: ๏ Kafka, what is it good for? ๏ Seriously, why not RabbitMQ? Thursday, March 21, 13
60. Adam Hitchcock @NorthIsUp DISQUSsion? Thursday, March 21, 13