Realtime data visualization with RethinkDB and Epoch

Transcript

1. Realtime data visualization RethinkDB & Epoch

2. Ryan Paul RethinkDB Evangelist @segphault

3. Usage Scenarios • Business intelligence • Streaming analytics • System

   monitoring • Live location tracking

4. Why Realtime?

5. Why Realtime? because POLLING IS A BUMMER

6. Realtime Visualization Stack RethinkDB NodeJS Socket.io D3 Koa Vue Epoch

   Browser REST API

7. Track live updates with RethinkDB changefeeds

8. What is RethinkDB? • Open source database for building realtime

   web applications • NoSQL database that stores schemaless JSON documents • Distributed database that is easy to scale

9. Built for Realtime Apps • Subscribe to change notifications from

   database queries • No more polling — the database pushes changes to your app • Reduce the amount of plumbing needed to stream live updates

10. Subscribe to change notifications on database queries Changefeeds

11. r.table("users").changes() Changefeeds Track changes on the users table

12. Changefeeds • The changes command returns a cursor that receives

    updates • Each update includes the new and old value of the modified record

13. r.table("users") .filter({name: "Bob"}).delete() Changefeed output: { new_val: null, old_val: {

    id: '362ae837-2e29-4695-adef-4fa415138f90', name: 'Bob', ... } } Changefeeds

14. r.table("players") .orderBy({index: r.desc("score")}) .limit(3).changes() Track top three players by score

    Chain the changes command to an actual ReQL query: Changefeeds

15. Send live updates with Socket.io Messaging

16. What is Socket.io? • Realtime web framework • Supports communication

    between frontend and backend • Abstracts away network transport • Can send arbitrary JSON objects

17. What is Socket.io? • Officially supported NodeJS server implementation •

    Client libraries for native mobile platforms and frontend web apps • Third-party server implementations for other languages

18. Socket.io Server var server = require("http").createServer(); var io = require("socket.io")(server);

    server.listen(8000, () => console.log("Server started on port 8000")); io.on("connection", client => { client.emit("message", {text: "Hello World"}); });

19. Socket.io Client var socket = io(); socket.on("message", data => {

    console.log("Received message:", data.text); });

20. Display live data with Epoch graphing library

21. What is D3? • Library for building interactive data visualizations

    • Provides APIs for generating and manipulating SVG content • Extremely powerful but difficult to use

22. Realtime Challenges • Potential for very frequent drawing updates •

    Need to remove old data while painting new data • Increments for your axes can change dynamically

23. Epoch Realtime charting library for high performance visualizations

24. What is Epoch? • High-level library for realtime data visualizations

    • Builds on D3 but also uses <canvas> to boost performance • Supports a fixed range of visualization types
25. None

26. Realtime Line Graph var chart = $("#chart").epoch({ type: "time.line", axes:

    ["left", "bottom"], data: [ {label: "Writes", values: [...]}, {label: "Reads", values: [...]} ] });

27. Realtime Line Graph var server = require("http").createServer(); var io =

    require("socket.io")(server); var r = require("rethinkdbdash")(config.database); var bluebird = require("bluebird"); bluebird.coroutine(function*() { var stats = yield r.db("rethinkdb").table("stats") .get(["cluster"]).changes(); stats.each((err, item) => io.emit("stats", item)); })(); server.listen(8000, () => console.log("Server started on port 8000"));

28. Realtime Line Graph function timestamp() { return (new Date).getTime() /

    1000; } var socket = io(); socket.on("stats", data => { var stats = data.new_val.query_engine; chart.push([ { time: timestamp(), y: stats.written_docs_per_sec}, { time: timestamp(), y: stats.read_docs_per_sec} ]); });

29. Realtime Pie Graph

30. Realtime Pie Graph var changes = yield r.table("votes").changes()("new_val"); changes.each((err, vote)

    => { io.to(`poll:${vote.poll}`).emit("vote", vote); }); ... io.on("connection", bluebird.coroutine(function*(client) { client.on("subscribe", (id, callback) => { r.table("polls").get(id).merge({ choices: r.row("choices").merge(c => ({ value: r.table("votes") .getAll(c("id"), {index: "choice"}).count() })) }).then(callback); client.join(`poll:${id}`); }); }));

31. Realtime Pie Graph created: function() { var client = io();

    client.on("vote", this.updateVote); client.on("connect", () => client.emit("subscribe", this.id, this.setup)); }, methods: { setup: function(data) { this.poll = data; this.chart = $("#vote-pie").epoch({ type: "pie", inner: 90, data: data.choices}); }, updateVote: function(vote) { this.poll.choices.find(c => c.id == vote.choice).value += 1; this.chart.update(this.poll.choices); ...

32. Additional Resources • RethinkDB website:
 http://rethinkdb.com • Epoch:
 https://fastly.github.io/epoch/ •

    Socket.io:
 http://socket.io/