one system to rule them all

A description of the workflow engine we use in iWantMyName. It is built on top of RabbitMQ and CouchDB.

RabbitMQ + CouchDB = Awesome

Transcript

1. one system to rule them all
3. the challenge
4. • write a backend that scales well • handle various frontends (web, iPhone, ...) • runs “somewhere in the cloud” • easy to maintain and extend
5. the solution
6. RabbitMQ +CouchDB =Awesome
7. Part I
8. • AMQP based Message Queue • lots of additional transports (XMPP, HTTP, STOMP, ...) • erlang based RabbitMQ
9. RabbitMQ daemon daemon daemon daemon frontend frontend frontend
10. • every functionality gets a backed daemon • every daemon has a queue associated • communication via a JSON message that gets passed around
11. the JSON message
13. { "response": { "expiry": 710, "ts": "1234425918", "name": "Lenz Gschwendtner", "last4": 0, "cc_type": "master" }, "data": { "options": { "billing_id": "E16003D4-B312-BEA23D9AB789" }, "command": "billing_info" }, "meta": { "lang": "en", "reply_to": "B312-E16003D4-BEA23D9AB789", "platform": "iwmn" } }
14. • RPC style communication • if the flow through the system is static • fast transactions works great for ...
15. gets fiddly when ... • there are asynchronous callbacks from outside • message flow can change based on various factors • tasks can take days to complete
16. Part II
17. workflows
19. ... again ...
20. • various frontends • backend daemons • JSON messages what we have ...
21. what we need • a way to describe message flow • a easy flexible way to write and update it
22. CouchDB
23. ... to the rescue
25. workflow • really only a path definition through backends • some sort of condition management • a backend definition
26. { "_id": "f7a4408898e5...a05b4181045", "_rev": "3-633060011", "name": "billing_info", "type": "workflow", "user": "iwmn", "path": [ "logger", "billing", "billing_iwmn_cc" ] }
27. ... and the backend definition
28. { "_id": "33adf2e3efc3...f423929f057ac", "_rev": "3-3507820969", "conn": "rpc", "type": "worker", "name": "billing_iwmn_cc", "location": "billing.cc", "description": "the CC interface", "command": "billing_verify_cc" }
29. now we need a workflow daemon
30. RabbitMQ daemon daemon daemon daemon frontend frontend frontend workflow
31. where are we now? • we can call backends as before • we can call workflows via the workflow queue • we can manage workflows in CouchDB
32. Part III
33. asynchronous callbacks
35. problem • we start a workflow • we call some external API • the external API somehow notifies us (Mail, Callback URL, ...) about progress • we need to continue in the workflow
36. we need a persistent message store
37. { "_id": "22cd37afd06c82129adb665d3150f570", "_rev": "3-2669449653", "status": "done", "last_update": 1252296819, "name": "E16003D4-F8DB-11DD-B312-BEA23D9AB789", "message": { "response": { "expiry": 710, "name": "Lenz Gschwendtner", "last4": 0, "cc_type": "master" }, "data": { "options": { "billing_id": "E16003D4-F8DB-11DD-B312-BEA23D9AB789" }, "command": "billing_info" }, "meta": { "lang": "de", "name": "E16003D4-F8DB-11DD-B312-BEA23D9AB789", "orig_reply_to": "D234ADE8-9B64-11DE-A92C-E5EF11C6FBDB", "reply_to": "workflow", "user": "iwmn", "log": [ "logger", "billing", "billing.iwmn.info" ], "platform": "iwmn" } }, "created": 1252296815, "type": "bill" }
38. one more problem how do we correlate messages
39. ... with some CouchDB awesome ...
40. • the backend daemon for the callback polls a view in CouchDB • we can search for any value in the request we sent • all we need is a view that searches for the very value this external API uses as correlation ID
41. RabbitMQ daemon daemon daemon daemon frontend frontend frontend workflow external callback
42. Lenz Gschwendtner CTO iWantMyName @norbu09 me
43. credits • RabbitMQ • CouchDB • Erlang and Perl
44. more credits • http://www.flickr.com/photos/djpd • http://www.flickr.com/photos/lilyapp • http://www.flickr.com/photos/b-tal • http://www.flickr.com/photos/mikebaird • http://www.flickr.com/photos/eyetwist