Ducksboard's architecture, a real-time online dashboard

Ducksboard's architecture, a real-time online dashboard

A talk on how Ducksboard has been designed and built to enable real-time metrics updates coming from different web services.

Faed19f9125bf030108575b40a3b7aba?s=128

Aitor Guevara

July 19, 2012
Tweet

Transcript

  1. None
  2. Aitor Guevara • @aitorciki • aitor@ducksboard.com

  3. What?

  4. web dashboard business metrics gorgeous real-time! and more...

  5. Who?

  6. Jan Urbanski PostgreSQL contributor & speaker Diego Mariño Abiquo founder

    ($15M funding) Aitor Guevara That’s me!
  7. Marta Online marketing sovereign David Django master of everything Pepe

    Backend architect of forever Javi Frontend ruler of tomorrow
  8. When?

  9. jan 2011 here we go! jun 2011 public beta jan

    2012 launch mar 2012 seed round
  10. How?

  11. None
  12. Design principles

  13. grabber / visualizer asynchronicity push vs pull scalability eat your

    own dog duck food
  14. Components

  15. Backend (grabber) Parser pulls data from SaaS Gateway write API

    API read API
  16. Frontend (viewer) web application REST + Websocket APIs mobile app

    & others (?)
  17. Database app settings data store parsers management

  18. Message Queue real-time updates n-publishers / m-consumers scenarios

  19. Technologies

  20. Python everywhere! backend Twisted frontend Django + Backbone.js PostgreSQL +

    RabbitMQ
  21. On Twisted async network engine network clients and servers callbacks

    and deferreds
  22. On deferreds from twisted.internet import reactor from twisted.web.client import Agent

    def _got_response(response): print(response.code, response.phrase) reactor.stop() agent = Agent(reactor) d = agent.request('GET', 'http://www.google.com/') d.addCallback(_got_response) reactor.run()
  23. Architecture

  24. None
  25. Parser 1 / metric / user config stored in PostgreSQL

    consumes many API kinds data sent to Gateway
  26. None
  27. Gateway stores data in PostgreSQL pushes data to RabbitMQ public

    API for custom data
  28. None
  29. API data read API (HTTP + Websocket) historic data from

    PostgreSQL real-time data from RabbitMQ
  30. None
  31. Frontend server Django app + SQLAlchemy JSON REST API nginx

    + uWSGI Celery handles async jobs
  32. Frontend client Backbone.js Raphaël draws the charts Websocket enables real-time

  33. On Backbone 1 model / entity n views / model

    (main, settings) jQuery templates (tmplItem())
  34. None
  35. PL/Python “trick” PostgreSQL starts parsers insert/update triggers parser HTTP interface

  36. “Scalability”

  37. standalone components HTTP communication nginx + DNS + haproxy RabbitMQ

    m-n pubsub
  38. Security

  39. HTTPS everywhere pgcrypto bcrypted user passwords relational integrity!

  40. Code

  41. Jenkins CI backend coverage: 84% frontend (server): 77% JS: PhantomJS

    + QUnit
  42. Deployment

  43. virtualenv private pip repository Jenkins builds packages Fabric deploys to

    nodes rollback support (symlinks)
  44. Statics caching build “compiles” JS and CSS mtime hash filenames

    mtimed image urls
  45. Platform

  46. Linode 4 parser nodes (1K/process) 2 gateway nodes 1 API

    node (2K/node) 1 frontend node
  47. 1 database node 1 build node 1 preprod node 1

    monitoring node
  48. Monitoring

  49. Zabbix items graphs alerts on thresholds resources + components

  50. Business metrics

  51. Analytics Mixpanel Optimizely Ducksboard!

  52. Questions?

  53. Thank you! get yours at ducksboard.com Aitor Guevara • @aitorciki

    • aitor@ducksboard.com