Realtime data visualization with RethinkDB and Epoch

Transcript

1. Realtime data visualization
   RethinkDB & Epoch
   

   View Slide

2. Ryan Paul
   RethinkDB
   Evangelist
   @segphault
   

   View Slide

3. Usage Scenarios
   • Business intelligence
   • Streaming analytics
   • System monitoring
   • Live location tracking
   

   View Slide

4. Why Realtime?
   

   View Slide

5. Why Realtime?
   because
   POLLING IS A BUMMER
   

   View Slide

6. Realtime Visualization Stack
   RethinkDB
   NodeJS Socket.io
   D3
   Koa
   Vue
   Epoch
   Browser
   REST API
   

   View Slide

7. Track live updates with
   RethinkDB changefeeds
   

   View Slide

8. What is RethinkDB?
   • Open source database for building
   realtime web applications
   • NoSQL database that
   stores schemaless JSON documents
   • Distributed database that is easy to
   scale
   

   View Slide

9. Built for Realtime Apps
   • Subscribe to change notifications
   from database queries
   • No more polling — the database
   pushes changes to your app
   • Reduce the amount of plumbing
   needed to stream live updates
   

   View Slide

10. Subscribe to change notifications
    on database queries
    Changefeeds
    

    View Slide

11. r.table("users").changes()
    Changefeeds
    Track changes on the users table
    

    View Slide

12. Changefeeds
    • The changes command returns a
    cursor that receives updates
    • Each update includes the new and
    old value of the modified record
    

    View Slide

13. r.table("users")
    .filter({name: "Bob"}).delete()
    Changefeed output:
    {
    new_val: null,
    old_val: {
    id: '362ae837-2e29-4695-adef-4fa415138f90',
    name: 'Bob',
    ...
    }
    }
    Changefeeds
    

    View Slide

14. r.table("players")
    .orderBy({index: r.desc("score")})
    .limit(3).changes()
    Track top three players by score
    Chain the changes command
    to an actual ReQL query:
    Changefeeds
    

    View Slide

15. Send live updates with
    Socket.io Messaging
    

    View Slide

16. What is Socket.io?
    • Realtime web framework
    • Supports communication
    between frontend and backend
    • Abstracts away network transport
    • Can send arbitrary JSON objects
    

    View Slide

17. What is Socket.io?
    • Officially supported NodeJS server
    implementation
    • Client libraries for native mobile
    platforms and frontend web apps
    • Third-party server implementations
    for other languages
    

    View Slide

18. Socket.io Server
    var server = require("http").createServer();
    var io = require("socket.io")(server);
    server.listen(8000, () =>
    console.log("Server started on port 8000"));
    io.on("connection", client => {
    client.emit("message", {text: "Hello World"});
    });
    

    View Slide

19. Socket.io Client
    var socket = io();
    socket.on("message", data => {
    console.log("Received message:", data.text);
    });
    

    View Slide

20. Display live data with
    Epoch graphing library
    

    View Slide

21. What is D3?
    • Library for building interactive
    data visualizations
    • Provides APIs for generating and
    manipulating SVG content
    • Extremely powerful but difficult to
    use
    

    View Slide

22. Realtime Challenges
    • Potential for very frequent
    drawing updates
    • Need to remove old data while
    painting new data
    • Increments for your axes can
    change dynamically
    

    View Slide

23. Epoch
    Realtime charting library for high
    performance visualizations
    

    View Slide

24. What is Epoch?
    • High-level library for realtime data
    visualizations
    • Builds on D3 but also uses
    to boost performance
    • Supports a fixed range of
    visualization types
    

    View Slide

25. View Slide

26. Realtime Line Graph
    var chart = $("#chart").epoch({
    type: "time.line",
    axes: ["left", "bottom"],
    data: [
    {label: "Writes", values: [...]},
    {label: "Reads", values: [...]}
    ]
    });
    

    View Slide

27. Realtime Line Graph
    var server = require("http").createServer();
    var io = require("socket.io")(server);
    var r = require("rethinkdbdash")(config.database);
    var bluebird = require("bluebird");
    bluebird.coroutine(function*() {
    var stats = yield r.db("rethinkdb").table("stats")
    .get(["cluster"]).changes();
    stats.each((err, item) => io.emit("stats", item));
    })();
    server.listen(8000, () =>
    console.log("Server started on port 8000"));
    

    View Slide

28. Realtime Line Graph
    function timestamp() {
    return (new Date).getTime() / 1000;
    }
    var socket = io();
    socket.on("stats", data => {
    var stats = data.new_val.query_engine;
    chart.push([
    { time: timestamp(), y: stats.written_docs_per_sec},
    { time: timestamp(), y: stats.read_docs_per_sec}
    ]);
    });
    

    View Slide

29. Realtime Pie Graph
    

    View Slide

30. Realtime Pie Graph
    var changes = yield r.table("votes").changes()("new_val");
    changes.each((err, vote) => {
    io.to(`poll:${vote.poll}`).emit("vote", vote);
    });
    ...
    io.on("connection", bluebird.coroutine(function*(client) {
    client.on("subscribe", (id, callback) => {
    r.table("polls").get(id).merge({
    choices: r.row("choices").merge(c => ({
    value: r.table("votes")
    .getAll(c("id"), {index: "choice"}).count()
    }))
    }).then(callback);
    client.join(`poll:${id}`);
    });
    }));
    

    View Slide

31. Realtime Pie Graph
    created: function() {
    var client = io();
    client.on("vote", this.updateVote);
    client.on("connect", () =>
    client.emit("subscribe", this.id, this.setup));
    },
    methods: {
    setup: function(data) {
    this.poll = data;
    this.chart = $("#vote-pie").epoch({
    type: "pie", inner: 90, data: data.choices});
    },
    updateVote: function(vote) {
    this.poll.choices.find(c =>
    c.id == vote.choice).value += 1;
    this.chart.update(this.poll.choices);
    ...
    

    View Slide

32. Additional Resources
    • RethinkDB website:

    http://rethinkdb.com
    • Epoch:

    https://fastly.github.io/epoch/
    • Socket.io:

    http://socket.io/
    

    View Slide