The Asynchronous Plumber - PyConUK 2012

As internal HTTP REST services, popularised by Amazon, have been gaining a lot of attention - newly emerging HTTP/1.1 extensions like Server Sent Messages and WebSockets are the perfect accompaniment to asynchronous tech stacks like Tornado, Twisted, Gevent and Node.js, not just for powering 10 minute chat demos.

Presented at http://2012.pyconuk.net/ by Wes Mason.

Transcript

1. THE ASYNCHRONOUS PLUMBER Wes Mason @1stvamp www.serverdensity.com
2. WHO? @1stvamp (twitter/github) Product Engineer at ServerDensity I like Python and evented code
3. THE WEB IS OUR API Everything does HTTP REST is great - let's do more of this Almost everything can do HTTP asynchronously
4. OUR INTERNAL SERVICES RESTful Stateless Mostly thinlayers Common documented API Callable from frontend
5. BRAVE NEW (ASYNC) WORLD Longpolling, WebSockets and Server Sent Events UIs are responsive or GTFO Libraries like socket.io make this pretty easy for devs
6. COME BACK TO ME Jobs running in parallel using task systems What we want: async responses to the browser What happens: Jobs complete and...die. *sniff*
7. COME BACK TO ME (CONT.) HTTP or WebSocket request kicks off a task Task runs asynchronous to requests Task completed and returns data... ...how?
8. SEGWAY TO CELERY Great: Takes care of distributing tasks to clusters Not so great: can't do multiple pubsub
9. PUBLISH AND SUBSCRIBE Need results back from Celery by account ID Pubsub is ideal for this Each account has multiple users What we want: one task per account, same result to each user
10. MANY TO MANY WITH PUBSUB Redis let's you do this (but we don't know Redis) In theory AMQP let's you do M2M pubsub (and Celery does AMQP by default)
11. ...WELL, KIND OF > > > r e s u l t = t a s k . d e l a y ( f o o ) A s y n c R e s u l t ( i d = " s o m e - u n i q u e - i d " ) > > > r e s u l t . g e t ( ) . . .
12. CUE THE HAXX Threads (urgh) Polling MongoDB (we do know Mongo) Using MongoDB as Celery backend and keeping results (hello race conditions!)
13. YOU HAD ME HOOKED The stack's just web APIs.. ..why not just use webhooks to return data? They're async, just like handling our frontend requests They can be tested the same way as our other APIs
14. THE WEBHOOK REGISTRY Register/unregister callbacks by a given key (e.g. an account ID) Web handler accepts data payloads via HTTP POST Webhook URI persisted somewhere (e.g. MongoDB)
15. SKYHOOOKS TO REACH THE TORNADO Open source library with registry for Tornado - soon Gevent and Twisted. Registers callbacks and handling of multiple instance URIs Available now: github.com/serverdensity/skyhooks
16. MEANWHILE, BACK ON THE [SERVER] FARM Tasks running somewhere, somehow Data returning from one or more tasks One or more I N S E R T _ S E R V E R _ H E R E (e.g. Tornado) instances at one or more URIs awaiting data
17. # i m p o r t r e q u e s t s , i m p o r t p y m o n g o # s e t u p c o l l e c t i o n e t c . e t c . h o o k s = w e b h o o k _ c o l l e c t i o n . f i n d ( { " a c c o u n t _ i d " : i d } ) i f d a t a a n d l e n ( d a t a ) > 0 : f o r h o o k i n h o o k s : r e q u e s t s . p o s t ( h o o k [ " u r l " ] , d a t a ) N.B. s k y h o o k s does this for you
18. IS THIS THE ONLY WAY? ZeroMQ? Crossroads.io has bi-di-pubsub MongoDB trailing cursors Do you need it?
19. SYS.EXIT(0) Code: github.com/serverdensity Slides: github.com/1stvamp/pycon-2012-talk Jonathan "Joff" Oliver: about.me/joffie Grab me and say hi : - )