Realtime data visualization RethinkDB & Epoch

Ryan Paul RethinkDB Evangelist @segphault

Usage Scenarios • Business intelligence • Streaming analytics • System monitoring • Live location tracking

Why Realtime?

Why Realtime? because POLLING IS A BUMMER

Realtime Visualization Stack RethinkDB NodeJS Socket.io D3 Koa Vue Epoch Browser REST API

Track live updates with RethinkDB changefeeds

What is RethinkDB? • Open source database for building realtime web applications • NoSQL database that stores schemaless JSON documents • Distributed database that is easy to scale

Built for Realtime Apps • Subscribe to change notifications from database queries • No more polling — the database pushes changes to your app • Reduce the amount of plumbing needed to stream live updates

Subscribe to change notifications on database queries Changefeeds

r.table("users").changes() Changefeeds Track changes on the users table

Changefeeds • The changes command returns a cursor that receives updates • Each update includes the new and old value of the modified record

r.table("users") .filter({name: "Bob"}).delete() Changefeed output: { new_val: null, old_val: { id: '362ae837-2e29-4695-adef-4fa415138f90', name: 'Bob', ... } } Changefeeds

r.table("players") .orderBy({index: r.desc("score")}) .limit(3).changes() Track top three players by score Chain the changes command to an actual ReQL query: Changefeeds

Send live updates with Socket.io Messaging

What is Socket.io? • Realtime web framework • Supports communication between frontend and backend • Abstracts away network transport • Can send arbitrary JSON objects

What is Socket.io? • Officially supported NodeJS server implementation • Client libraries for native mobile platforms and frontend web apps • Third-party server implementations for other languages

Socket.io Server var server = require("http").createServer(); var io = require("socket.io")(server); server.listen(8000, () => console.log("Server started on port 8000")); io.on("connection", client => { client.emit("message", {text: "Hello World"}); });

Socket.io Client var socket = io(); socket.on("message", data => { console.log("Received message:", data.text); });

Display live data with Epoch graphing library

What is D3? • Library for building interactive data visualizations • Provides APIs for generating and manipulating SVG content • Extremely powerful but difficult to use

Realtime Challenges • Potential for very frequent drawing updates • Need to remove old data while painting new data • Increments for your axes can change dynamically

Epoch Realtime charting library for high performance visualizations

What is Epoch? • High-level library for realtime data visualizations • Builds on D3 but also uses to boost performance • Supports a fixed range of visualization types

No content

Realtime Line Graph var chart = $("#chart").epoch({ type: "time.line", axes: ["left", "bottom"], data: [ {label: "Writes", values: [...]}, {label: "Reads", values: [...]} ] });

Realtime Line Graph var server = require("http").createServer(); var io = require("socket.io")(server); var r = require("rethinkdbdash")(config.database); var bluebird = require("bluebird"); bluebird.coroutine(function*() { var stats = yield r.db("rethinkdb").table("stats") .get(["cluster"]).changes(); stats.each((err, item) => io.emit("stats", item)); })(); server.listen(8000, () => console.log("Server started on port 8000"));

Realtime Line Graph function timestamp() { return (new Date).getTime() / 1000; } var socket = io(); socket.on("stats", data => { var stats = data.new_val.query_engine; chart.push([ { time: timestamp(), y: stats.written_docs_per_sec}, { time: timestamp(), y: stats.read_docs_per_sec} ]); });

Realtime Pie Graph

Realtime Pie Graph var changes = yield r.table("votes").changes()("new_val"); changes.each((err, vote) => { io.to(`poll:${vote.poll}`).emit("vote", vote); }); ... io.on("connection", bluebird.coroutine(function*(client) { client.on("subscribe", (id, callback) => { r.table("polls").get(id).merge({ choices: r.row("choices").merge(c => ({ value: r.table("votes") .getAll(c("id"), {index: "choice"}).count() })) }).then(callback); client.join(`poll:${id}`); }); }));

Realtime Pie Graph created: function() { var client = io(); client.on("vote", this.updateVote); client.on("connect", () => client.emit("subscribe", this.id, this.setup)); }, methods: { setup: function(data) { this.poll = data; this.chart = $("#vote-pie").epoch({ type: "pie", inner: 90, data: data.choices}); }, updateVote: function(vote) { this.poll.choices.find(c => c.id == vote.choice).value += 1; this.chart.update(this.poll.choices); ...

Additional Resources • RethinkDB website:
 http://rethinkdb.com • Epoch:
 https://fastly.github.io/epoch/ • Socket.io:
 http://socket.io/