Save 37% off PRO during our Black Friday Sale! »

How my ops guy totally ruined my realtime talk last week with a single nginx module

887048987be67f10649a9dfacced6606?s=47 Adam
October 23, 2012

How my ops guy totally ruined my realtime talk last week with a single nginx module

tl;dr: #Python #Architecture #Scalability #nginx #DISQUS

Disqus' realtime infrastructure, presented at Keeping It Realtime Conf 2012.

887048987be67f10649a9dfacced6606?s=128

Adam

October 23, 2012
Tweet

Transcript

  1. Adam Hitchcock @NorthIsUp Making DISQUS Realtime Thursday, March 21, 13

    i am an infrastructure engineer at disqus focus on realtime/firehose
  2. Adam Hitchcock @NorthIsUp Making DISQUS Realtime Thursday, March 21, 13

    thanks for watching me instead of the mini ipad thing. more of a story -this was the title of my talk, but it is now... more of a story
  3. Adam Hitchcock @NorthIsUp Making DISQUS Realtime Thursday, March 21, 13

    more of a story -this was the title of my talk, but it is now... more of a story
  4. Adam Hitchcock @NorthIsUp How my ops guy totally changed my

    talk last week with a single nginx module Thursday, March 21, 13
  5. what is DISQUS? Thursday, March 21, 13 How many of

    you will raise your hand? How many of you are python people? How many of you know what disqus is?
  6. Thursday, March 21, 13 - community platform - main product

    is a comment widget (javascript)
  7. why do realtime? ๏ getting new data to the user

    asap ๏ for increased engagement ๏ and it looks awesome ๏ and we can sell (or trade) it Thursday, March 21, 13 We define this as ‘less than 10 seconds’ but my goal was less than one.
  8. DISQUS sees a lot of tra c Google Analytics: Sept

    2012 - Oct 2012 Thursday, March 21, 13 the problem is that... - at max capacity - old system less than 100 thousand concurrent users
  9. realertime ๏ currently active on all DISQUS 2012 sites ๏

    tested ‘dark’ on our existing network ๏ 1.5 million concurrently connected users ๏ 45 thousand new connections per second ๏ 165 thousand messages/second ๏ <.2 seconds latency end to end Thursday, March 21, 13 - wtf: describe disqus 12 - i’ll re-visit on what dark means later on the testing slides - describe heavy tail distribution of popularity - end to end does NOT include the DISQUS app DEMO IT then “so how did we build this?”
  10. so, how did we do it? Thursday, March 21, 13

    last time I gave this talk I had a joke here. It was at euro python and...
  11. thoonk redis queue some python glue nginx push stream and

    long(er) polling Thursday, March 21, 13 python is our normal django site + some formatting and processing code redis is thoonk queue
  12. architecture overview Thursday, March 21, 13

  13. architecture overview redis queue nginx + push stream module “python

    glue” Gevent server New Posts ngnix /pub endpoint DISQUS embed clients http post DISQUS Thursday, March 21, 13 - HARDWARE - HA - 3 flows - new info -> pubsub - new subscriptions - pubsub -> subscriptions
  14. architecture overview django Formatter Publishers thoonk queue New Posts http

    post ngnix pub endpoint DISQUS embed clients other realtime stu nginx + push stream module Thursday, March 21, 13 - post save and post delete hooks - other realtime stuff + thoonk
  15. thoonk redis queue some python glue nginx push stream and

    long(er) polling Thursday, March 21, 13 python is our normal django site + some formatting and processing code redis is thoonk queue
  16. the thoonk queue ๏ thoonk is a queue on top

    of redis ๏ implemented as a DFA ๏ provides job semantics ๏ useful for end to end acking ๏ reliable job processing in distributed system ๏ did I mention it’s on top of redis? Thursday, March 21, 13 uses zset to store items so you have ranged queries (can’t do that on rabbit)
  17. thoonk redis queue some python glue nginx push stream and

    long(er) polling Thursday, March 21, 13
  18. the python glue ๏ listens to a thoonk queue ๏

    cleans & formats message ๏ this is the final format for end clients ๏ compress data now ๏ publish message to nginx and other firehoses ๏ forum:id, thread:id, user:id, post:id Formatter Publishers Thursday, March 21, 13 - django post save & post delete signals - thoonk was easy and fun! - end to end ack via thoonk. (not removed until fully published to nginx) - allows for multiple publishers, we publish to nginx, pubsubhubbub, commercial consumers.
  19. thoonk redis queue some python glue nginx push stream and

    long(er) polling Thursday, March 21, 13
  20. nginx push stream ๏ follow John Watson (@wizputer) for updated

    #humblebrags as we ramp up tra c ๏ an example config can be found here: http://bit.ly/disqus-nginx-push-stream http://wiki.nginx.org/HttpPushStreamModule Thursday, March 21, 13
  21. nginx push stream ๏ Turned on for ~50% of our

    network... ๏ ~950K subscribers (peak single machine) ๏ 144 Mbits/second (per machine) ๏ CPU usage is still well under 50% http://wiki.nginx.org/HttpPushStreamModule Thursday, March 21, 13 Have NOT tested SSL yet
  22. config push stream location = /pub { allow 127.0.0.1; deny

    all; push_stream_publisher admin; set $push_stream_channel_id $arg_channel; } location ^~ /sub/ { # to maintain api compatibility we need this location ~ /sub/(.*)/(.*)$ { set $push_stream_channels_path $1:$2; push_stream_subscriber streaming; push_stream_content_type application/json; } } http://wiki.nginx.org/HttpPushStreamModule Thursday, March 21, 13 and nginx does the rest
  23. examples # Subs curl -s 'localhost/sub/forum/cnn' curl -s 'localhost/sub/thread/907824578' curl

    -s 'localhost/sub/user/northisup' # Pubs curl -s -X POST 'localhost/pub?channel=forum:cnn' \ -d '{"some sort": "of json data"}' curl -s -X POST 'localhost/pub?channel=thread:907824578' \ -d '{"more": "json data"}' curl -s -X POST 'localhost/pub?channel=user:northisup' \ -d '{"the idea": "I think you get it by now"}' http://wiki.nginx.org/HttpPushStreamModule Thursday, March 21, 13
  24. measure nginx location = /push-stream-status { allow 127.0.0.1; deny all;

    push_stream_channels_statistics; set $push_stream_channel_id $arg_channel; } http://wiki.nginx.org/HttpPushStreamModule Thursday, March 21, 13 actually used this to build a realtime stream of popular threads on disqus
  25. thoonk redis queue some python glue nginx push stream and

    long(er) polling Thursday, March 21, 13
  26. long(er) polling onProgress: function () { var self = this;

    var resp = self.xhr.responseText; var advance = 0; var rows; // If server didn't push anything new, do nothing. if (!resp || self.len === resp.length) return; // Server returns JSON objects, one per line. rows = resp.slice(self.len).split('\n'); _.each(rows, function (obj) { advance += (obj.length + 1); obj = JSON.parse(obj); self.trigger('progress', obj); }); self.len += advance; } Thursday, March 21, 13 because on a busy thread this matters, 99% of the time, doesn’t matter (IGN E3) - peak post rate ~40 msg/sec - peak delivery ~164K msg/sec
  27. test, measure, repeat Thursday, March 21, 13

  28. test ๏ Darktime ๏ use existing network to loadtest ๏

    (user complaints when it didn’t work...) ๏ Darkesttime ๏ load testing a single thread ๏ have knobs you can twiddle Thursday, March 21, 13
  29. measure ๏ measure all the things! ๏ especially when the

    numbers don’t line up ๏ measuring is hard in distributed systems ๏ try to express things as +1 and -1 if you can ๏ Sentry for measuring exceptions Thursday, March 21, 13 so when you have a talk you #humblebrag like “peak delivery rate” and stuff scales! gauges and aggregation
  30. lessons ๏ do hard (computation) work early ๏ end-to-end acks

    are good, but expensive ๏ redis/nginx pubsub is e ectively free Thursday, March 21, 13 - data processing and json formatting done once not 1000x times - gziping done once not 1000x times - defer setting up the work in the generator until as late as possible - ditched e2e acks from the fe, cost way too much
  31. special thanks ๏ the team at DISQUS ๏ like je

    who had to review all my code ๏ and especially our dev-ops guys ๏ like john watson a.k.a. @wizputer a.k.a the one who made me rewrite this talk psst, we’re hiring disqus.com/jobs Thursday, March 21, 13 Tell me your thoughts! @NorthIsUp
  32. slide full o’ links ๏ Nginx push stream module http://wiki.nginx.org/HttpPushStreamModule

    ๏ Thoonk (redis queue) http://github.com/andyet/thoonk.py ๏ Sentry (distributed traceback aggregation) http://github.com/dcramer/sentry ๏ Gevent (python coroutines and greenlets) http://gevent.org/ ๏ Scales (in-app metrics) http://github.com/Greplin/scales code.disqus.com Thursday, March 21, 13 Tell me your thoughts! @NorthIsUp
  33. Adam Hitchcock @NorthIsUp DISQUSsion? Thursday, March 21, 13 Tell me

    your thoughts! @NorthIsUp