Realtime data visualization with RethinkDB and Epoch

Transcript

1. Realtime data visualization RethinkDB & Epoch
2. Ryan Paul RethinkDB Evangelist @segphault
3. Usage Scenarios • Business intelligence • Streaming analytics • System monitoring • Live location tracking
4. Why Realtime?
5. Why Realtime? because POLLING IS A BUMMER
6. Realtime Visualization Stack RethinkDB NodeJS Socket.io D3 Koa Vue Epoch Browser REST API
7. Track live updates with RethinkDB changefeeds
8. What is RethinkDB? • Open source database for building realtime web applications • NoSQL database that stores schemaless JSON documents • Distributed database that is easy to scale
9. Built for Realtime Apps • Subscribe to change notifications from database queries • No more polling — the database pushes changes to your app • Reduce the amount of plumbing needed to stream live updates
10. Subscribe to change notifications on database queries Changefeeds
11. r.table("users").changes() Changefeeds Track changes on the users table
12. Changefeeds • The changes command returns a cursor that receives updates • Each update includes the new and old value of the modified record
13. r.table("users") .filter({name: "Bob"}).delete() Changefeed output: { new_val: null, old_val: { id: '362ae837-2e29-4695-adef-4fa415138f90', name: 'Bob', ... } } Changefeeds
14. r.table("players") .orderBy({index: r.desc("score")}) .limit(3).changes() Track top three players by score Chain the changes command to an actual ReQL query: Changefeeds
15. Send live updates with Socket.io Messaging
16. What is Socket.io? • Realtime web framework • Supports communication between frontend and backend • Abstracts away network transport • Can send arbitrary JSON objects
17. What is Socket.io? • Officially supported NodeJS server implementation • Client libraries for native mobile platforms and frontend web apps • Third-party server implementations for other languages
18. Socket.io Server var server = require("http").createServer(); var io = require("socket.io")(server); server.listen(8000, () => console.log("Server started on port 8000")); io.on("connection", client => { client.emit("message", {text: "Hello World"}); });
19. Socket.io Client var socket = io(); socket.on("message", data => { console.log("Received message:", data.text); });
20. Display live data with Epoch graphing library
21. What is D3? • Library for building interactive data visualizations • Provides APIs for generating and manipulating SVG content • Extremely powerful but difficult to use
22. Realtime Challenges • Potential for very frequent drawing updates • Need to remove old data while painting new data • Increments for your axes can change dynamically
23. Epoch Realtime charting library for high performance visualizations
24. What is Epoch? • High-level library for realtime data visualizations • Builds on D3 but also uses <canvas> to boost performance • Supports a fixed range of visualization types
26. Realtime Line Graph var chart = $("#chart").epoch({ type: "time.line", axes: ["left", "bottom"], data: [ {label: "Writes", values: [...]}, {label: "Reads", values: [...]} ] });
27. Realtime Line Graph var server = require("http").createServer(); var io = require("socket.io")(server); var r = require("rethinkdbdash")(config.database); var bluebird = require("bluebird"); bluebird.coroutine(function*() { var stats = yield r.db("rethinkdb").table("stats") .get(["cluster"]).changes(); stats.each((err, item) => io.emit("stats", item)); })(); server.listen(8000, () => console.log("Server started on port 8000"));
28. Realtime Line Graph function timestamp() { return (new Date).getTime() / 1000; } var socket = io(); socket.on("stats", data => { var stats = data.new_val.query_engine; chart.push([ { time: timestamp(), y: stats.written_docs_per_sec}, { time: timestamp(), y: stats.read_docs_per_sec} ]); });
29. Realtime Pie Graph
30. Realtime Pie Graph var changes = yield r.table("votes").changes()("new_val"); changes.each((err, vote) => { io.to(`poll:${vote.poll}`).emit("vote", vote); }); ... io.on("connection", bluebird.coroutine(function*(client) { client.on("subscribe", (id, callback) => { r.table("polls").get(id).merge({ choices: r.row("choices").merge(c => ({ value: r.table("votes") .getAll(c("id"), {index: "choice"}).count() })) }).then(callback); client.join(`poll:${id}`); }); }));
31. Realtime Pie Graph created: function() { var client = io(); client.on("vote", this.updateVote); client.on("connect", () => client.emit("subscribe", this.id, this.setup)); }, methods: { setup: function(data) { this.poll = data; this.chart = $("#vote-pie").epoch({ type: "pie", inner: 90, data: data.choices}); }, updateVote: function(vote) { this.poll.choices.find(c => c.id == vote.choice).value += 1; this.chart.update(this.poll.choices); ...
32. Additional Resources • RethinkDB website:
 http://rethinkdb.com • Epoch:
 https://fastly.github.io/epoch/ • Socket.io:
 http://socket.io/