Ducksboard's architecture, a real-time online dashboard

A talk on how Ducksboard has been designed and built to enable real-time metrics updates coming from different web services.

Transcript

2. Aitor Guevara • @aitorciki • aitor@ducksboard.com
3. What?
4. web dashboard business metrics gorgeous real-time! and more...
5. Who?
6. Jan Urbanski PostgreSQL contributor & speaker Diego Mariño Abiquo founder ($15M funding) Aitor Guevara That’s me!
7. Marta Online marketing sovereign David Django master of everything Pepe Backend architect of forever Javi Frontend ruler of tomorrow
8. When?
9. jan 2011 here we go! jun 2011 public beta jan 2012 launch mar 2012 seed round
10. How?
12. Design principles
13. grabber / visualizer asynchronicity push vs pull scalability eat your own dog duck food
14. Components
15. Backend (grabber) Parser pulls data from SaaS Gateway write API API read API
16. Frontend (viewer) web application REST + Websocket APIs mobile app & others (?)
17. Database app settings data store parsers management
18. Message Queue real-time updates n-publishers / m-consumers scenarios
19. Technologies
20. Python everywhere! backend Twisted frontend Django + Backbone.js PostgreSQL + RabbitMQ
21. On Twisted async network engine network clients and servers callbacks and deferreds
22. On deferreds from twisted.internet import reactor from twisted.web.client import Agent def _got_response(response): print(response.code, response.phrase) reactor.stop() agent = Agent(reactor) d = agent.request('GET', 'http://www.google.com/') d.addCallback(_got_response) reactor.run()
23. Architecture
25. Parser 1 / metric / user config stored in PostgreSQL consumes many API kinds data sent to Gateway
27. Gateway stores data in PostgreSQL pushes data to RabbitMQ public API for custom data
29. API data read API (HTTP + Websocket) historic data from PostgreSQL real-time data from RabbitMQ
31. Frontend server Django app + SQLAlchemy JSON REST API nginx + uWSGI Celery handles async jobs
32. Frontend client Backbone.js Raphaël draws the charts Websocket enables real-time
33. On Backbone 1 model / entity n views / model (main, settings) jQuery templates (tmplItem())
35. PL/Python “trick” PostgreSQL starts parsers insert/update triggers parser HTTP interface
36. “Scalability”
37. standalone components HTTP communication nginx + DNS + haproxy RabbitMQ m-n pubsub
38. Security
39. HTTPS everywhere pgcrypto bcrypted user passwords relational integrity!
40. Code
41. Jenkins CI backend coverage: 84% frontend (server): 77% JS: PhantomJS + QUnit
42. Deployment
43. virtualenv private pip repository Jenkins builds packages Fabric deploys to nodes rollback support (symlinks)
44. Statics caching build “compiles” JS and CSS mtime hash filenames mtimed image urls
45. Platform
46. Linode 4 parser nodes (1K/process) 2 gateway nodes 1 API node (2K/node) 1 frontend node
47. 1 database node 1 build node 1 preprod node 1 monitoring node
48. Monitoring
49. Zabbix items graphs alerts on thresholds resources + components
50. Business metrics
51. Analytics Mixpanel Optimizely Ducksboard!
52. Questions?
53. Thank you! get yours at ducksboard.com Aitor Guevara • @aitorciki • aitor@ducksboard.com