Slide 1
Slide 1 text
No content
Slide 3
Slide 3 text
What?
Slide 4
Slide 4 text
web dashboard
business metrics
gorgeous
real-time!
and more...
Slide 5
Slide 5 text
Who?
Slide 6
Slide 6 text
Jan Urbanski
PostgreSQL
contributor &
speaker
Diego Mariño
Abiquo founder
($15M funding)
Aitor Guevara
That’s me!
Slide 7
Slide 7 text
Marta Online marketing sovereign
David Django master of everything
Pepe Backend architect of forever
Javi Frontend ruler of tomorrow
Slide 8
Slide 8 text
When?
Slide 9
Slide 9 text
jan 2011 here we go!
jun 2011 public beta
jan 2012 launch
mar 2012 seed round
Slide 10
Slide 10 text
How?
Slide 11
Slide 11 text
No content
Slide 12
Slide 12 text
Design
principles
Slide 13
Slide 13 text
grabber / visualizer
asynchronicity
push vs pull
scalability
eat your own dog duck food
Slide 14
Slide 14 text
Components
Slide 15
Slide 15 text
Backend (grabber)
Parser pulls data from SaaS
Gateway write API
API read API
Slide 16
Slide 16 text
Frontend (viewer)
web application
REST + Websocket APIs
mobile app & others (?)
Slide 17
Slide 17 text
Database
app settings
data store
parsers management
Slide 18
Slide 18 text
Message Queue
real-time updates
n-publishers / m-consumers
scenarios
Slide 19
Slide 19 text
Technologies
Slide 20
Slide 20 text
Python everywhere!
backend Twisted
frontend Django + Backbone.js
PostgreSQL + RabbitMQ
Slide 21
Slide 21 text
On Twisted
async network engine
network clients and servers
callbacks and deferreds
Slide 22
Slide 22 text
On deferreds
from twisted.internet import reactor
from twisted.web.client import Agent
def _got_response(response):
print(response.code, response.phrase)
reactor.stop()
agent = Agent(reactor)
d = agent.request('GET', 'http://www.google.com/')
d.addCallback(_got_response)
reactor.run()
Slide 23
Slide 23 text
Architecture
Slide 24
Slide 24 text
No content
Slide 25
Slide 25 text
Parser
1 / metric / user
config stored in PostgreSQL
consumes many API kinds
data sent to Gateway
Slide 26
Slide 26 text
No content
Slide 27
Slide 27 text
Gateway
stores data in PostgreSQL
pushes data to RabbitMQ
public API for custom data
Slide 28
Slide 28 text
No content
Slide 29
Slide 29 text
API
data read API (HTTP + Websocket)
historic data from PostgreSQL
real-time data from RabbitMQ
Slide 30
Slide 30 text
No content
Slide 31
Slide 31 text
Frontend server
Django app + SQLAlchemy
JSON REST API
nginx + uWSGI
Celery handles async jobs
Slide 32
Slide 32 text
Frontend client
Backbone.js
Raphaël draws the charts
Websocket enables real-time
Slide 33
Slide 33 text
On Backbone
1 model / entity
n views / model (main, settings)
jQuery templates (tmplItem())
Slide 34
Slide 34 text
No content
Slide 35
Slide 35 text
PL/Python “trick”
PostgreSQL starts parsers
insert/update triggers
parser HTTP interface
Slide 36
Slide 36 text
“Scalability”
Slide 37
Slide 37 text
standalone components
HTTP communication
nginx + DNS + haproxy
RabbitMQ m-n pubsub
Slide 38
Slide 38 text
Security
Slide 39
Slide 39 text
HTTPS everywhere
pgcrypto
bcrypted user passwords
relational integrity!
Slide 40
Slide 40 text
Code
Slide 41
Slide 41 text
Jenkins CI
backend coverage: 84%
frontend (server): 77%
JS: PhantomJS + QUnit
Slide 42
Slide 42 text
Deployment
Slide 43
Slide 43 text
virtualenv
private pip repository
Jenkins builds packages
Fabric deploys to nodes
rollback support (symlinks)
Slide 44
Slide 44 text
Statics caching
build “compiles” JS and CSS
mtime hash filenames
mtimed image urls
Slide 45
Slide 45 text
Platform
Slide 46
Slide 46 text
Linode
4 parser nodes (1K/process)
2 gateway nodes
1 API node (2K/node)
1 frontend node
Slide 47
Slide 47 text
1 database node
1 build node
1 preprod node
1 monitoring node
Slide 48
Slide 48 text
Monitoring
Slide 49
Slide 49 text
Zabbix
items graphs
alerts on thresholds
resources + components
Slide 50
Slide 50 text
Business
metrics
Slide 51
Slide 51 text
Analytics
Mixpanel
Optimizely
Ducksboard!
Slide 52
Slide 52 text
Questions?
Slide 53
Slide 53 text
Thank you!
get yours at ducksboard.com
Aitor Guevara • @aitorciki • [email protected]