Building Custom Analytics with InfluxDB and D3

Building Custom Analytics with InfluxDB and D3 Paul Dix [email protected]
@pauldix

InfluxDB An open source distributed analytics database.

Analytics Database? • Metrics, events, time series • Infrastructure metrics
• Application performance • User events/analytics • Business analytics

Events • Measurements • Exceptions • Page Views • User
actions • Commits • Deploys

Metrics or measurements • CPU load • Memory usage •
Average response times • Counts over fixed intervals

Things you want to know about over time

InfluxDB • Written in Go • No external dependencies •
3 full time developers (YC W13)

How data is organized • Databases (like in MySQL, Postgres,
etc) • Time series ◦ like tables, but you can have millions of them ◦ no need to define ahead of time • Points or events ◦ like rows, but schemaless like Mongo ◦ single level hash

[{ "name": "foo", "columns": [ "time", "sequence_number", "val1", "val2" ],
"points": [ [1384295094, 3, "paul", 23], [1384295094, 2, "john", 92], [1384295094, 1, "todd", 61] ] }, {...}] As JSON

Write points curl -X POST \ 'http://localhost:8086db/mydb/series?u=paul&p=pass' \ -d '[{"name":"foo",
"columns":["val"], "points": [[3]]}]'

Querying curl \ 'http://...:8086/db/mydb/series?u=paul&p=pass&q=...'

SQL(ish) Query Language select * from foo where time >
now() - 4h

[{ "name": "foo", "columns": [ "time", "sequence_number", "val1", "val2" ],
"points": [ [1384295094, 3, "paul", 23], [1384295094, 2, "john", 92], [1384295094, 1, "todd", 61] ] }, {...}] JSON data returned

select count(state) from user_events group by time(5m) where time >
now() - 7d and state = ‘MA’

[{ "name": "user_events", "columns": [ "time", "count" ], "points": [
[1389729000, 3], [1389728700, 5], [1389728400, 1] ] }, {...}] JSON data returned

select count(state), state from user_events group by time(5m), state where
time > now() - 7d

[{ "name": "user_events", "columns": [ "time", "count", "state" ], "points":
[ [1389729000, 3, "NY"], [1389729000, 5, "CO"], [1389728700, 1, "NY"] ] }, {...}] JSON data returned

Other functions • Percentiles • Min, max, first, last, sum,
count, stddev • Histogram • Lag, lead, funnel (next release)

Built in UI

# install influxdb brew update && brew install influxdb influxdb
-config=/usr/local/etc/influxdb.conf # set up the admin dashboard git clone https://github.com/influxdb/influxdb-admin cd influxdb-admin bundle install middleman server open http://localhost:4567 Building Custom UIs

Starting the Interface # make the dir in the local
admin interface # dir so it shows up on the drop down mkdir /usr/local/opt/influxdb/share/admin/interfaces/paul_owns # make the dir and file in the influxdb-admin repo mkdir source/interfaces/paul_owns touch source/interfaces/paul_owns/index.html # or slim, etc.

# line 36 in config.rb # activate :livereload Kill Live
Reload

<html><head> <link href="//netdna.bootstrapcdn.com/bootstrap/3.0.3/css/bootstrap. min.css" media="screen" rel="stylesheet" type="text/css"> </head> <body><div class="container"><div
class="row"> <div class="col-md-12"> <h1>Paul's Custom Dash</h1> </div></div></div> </body> </html>

Write some data [ { "name": "events", "columns": ["type", "email"],
"points": [ ["signup", "[email protected]"], ["paid", "[email protected]"] ] } ]

<div class="container"> ... <div class="row"> <div class="col-md-6" id="events_graph"></div> </div> <div>
Graph it!

q = "SELECT COUNT(type) FROM events GROUP BY time(1h), type
fill(0)" parent.influxdb.query(q, (points) => typeToSeries = {} points.forEach((point) => series = typeToSeries[point.type] if !series typeToSeries[point.type] = []; series = typeToSeries[point.type]; series.push({x: point.time / 1000, y: point.count}) )

lastColorUsed = -1 colors = d3.scale.category10() data = for type,
series of typeToSeries lastColorUsed += 1 { data: series.reverse(), color: colors(lastColorUsed), name: type }

graph = new Rickshaw.Graph( height: 200, element: document.querySelector("#events_graph"), renderer: 'area',
stroke: true, series: data ) graph.render()

<div class="container"> ... <div class="row"> <h3>Detail</h3> <div class="col-md-6" id="events_detail"></div> </div>
</div> Add some hover detail

hoverDetail = new Rickshaw.Graph.HoverDetail graph: graph, formatter: (series, x, y)
-> date = '<span class="date">' + new Date(x * 1000).toUTCString() + '</span>' swatch = '<span class="detail_swatch" style="background-color: ' + series.color + '"></span>' content = swatch + series.name + ": " + parseInt(y) + '<br>' + date renderDetail(series.name, x) content

renderDetail = (series, x) => startTime = x endTime =
startTime + 3600 q = "select * from events where time > " + startTime + "s and time < " + endTime + "s and type = '" + series + "'"

parent.influxdb.query(q, (points) => ul = "<ul>" points.forEach (point) => li
= "<li>" li += point.email li += "</li>" ul += li ul += "</ul>" $("#events_detail").html(ul) )

Many possibilities • Histograms over time • Pulling in context
• Drilling down

We’re looking for help • Dashboard builder • Explorer

Thanks! [email protected] @pauldix

Building Custom Analytics with InfluxDB and D3

Building Custom Analytics with InfluxDB and D3

More Decks by Paul Dix

Other Decks in Technology

Featured

Transcript