RethinkDB Training Course

Transcript

1. RethinkDB Distributed Databases

2. Ryan Paul RethinkDB Evangelist @segphault

3. Introduction What is RethinkDB?

4. What is RethinkDB? • Open source database for building realtime

   web applications • NoSQL database that stores schemaless JSON documents • Distributed database that is easy to scale

5. Built for Realtime Apps • Subscribe to change notifications from

   database queries • No more polling — the database pushes changes to your app • Reduce the amount of plumbing needed to stream live updates

6. Power and Convenience • Highly expressive query language • Relational

   features like table joins • Powerful admin UI with point- and-click cluster management

7. Introduction to ReQL RethinkDB Query Language

8. Introduction to ReQL • ReQL embeds natively into your programming

   language • Compose ReQL queries by chaining commands • ReQL queries are executed on the server

9. Sample ReQL Queries r.table("users") .filter(r.row("age").gt(30)) r.table("users") .pluck("last_name") .distinct().count() r.table("fellowship") .filter({species:

   "hobbit"}) .update({species: "halfling"})

10. Anatomy of a ReQL Query r.table("users") .pluck("last_name") .distinct().count() Number of

    unique last names

11. Anatomy of a ReQL Query r.table("users") .pluck("last_name") .distinct().count() Access a

    database table

12. Anatomy of a ReQL Query r.table("users") .pluck("last_name") .distinct().count() Isolate a

    document property

13. Anatomy of a ReQL Query r.table("users") .pluck("last_name") .distinct().count() Consolidate duplicate

    values

14. Anatomy of a ReQL Query r.table("users") .pluck("last_name") .distinct().count() Display the

    number of items

15. ReQL Commands • Transformations: map, orderBy, skip, limit, slice •

    Aggregations: group, reduce, count, sum, avg, min, max, distinct, contains • Documents: row, pluck, without, merge, append, difference, keys, hasFields, spliceAt • Writing: insert, update, replace, delete • Control: forEach, range, branch, do, coerceTo, expr

16. DEMO

17. Secondary Indexes • Queries performed against indexes are much faster

    • Can index on a single property, multiple fields, or arbitrary ReQL expressions

18. Querying an Index r.table("fellowship") .indexCreate("species") r.table("fellowship") .getAll("human", {index: "species"}) Find

    all humans in “fellowship”

19. Anonymous Functions r.range(5).map(function(i) { return i.mul(2); }) Multiply each value

    by 2 You can pass anonymous functions to commands like map and reduce:

20. Understanding ReQL • Anonymous function must return a valid ReQL

    expression • Client driver translates ReQL queries into wire protocol • Can’t mix local application functionality into ReQL queries • In JS use e.g. the mul and gt commands instead of the normal operators

21. The r.row command Multiply each value by 2 You can

    often use r.row instead of an anonymous function: r.range(5).map(r.row.mul(2))

22. Grouping r.table("songs") .group("artist") .count() Number of songs by each artist

    Use group to collect records into groups by a shared property:

23. Grouping [ { "group": "Morcheeba", "reduction": 10 }, { "group":

    "Pink Floyd", "reduction": 145 }, ... ]

24. Grouping • Commands chained after group will operate on the

    contents of the group • Use the ungroup command to operate on the grouped output

25. Ungroup r.table("songs") .group("artist").count() .ungroup().max("reduction") Artist with the most songs

26. Additional ReQL Features • Geospatial indexing for location- based queries

    • Support for storing binary objects • Date and time functions for time data • An http command for fetching remote JSON data

27. Realtime Updates Working with Changefeeds

28. Subscribe to change notifications on database queries Changefeeds

29. r.table("users").changes() Changefeeds Track changes on the users table

30. Changefeeds • The changes command returns a cursor that receives

    updates • Each update includes the new and old value of the modified record

31. Changefeeds r.table("users") .filter({name: "Bob"}).delete() Changefeed output: { new_val: null, old_val:

    { id: '362ae837-2e29-4695-adef-4fa415138f90', name: 'Bob', ... } }

32. Changefeeds r.table("players") .orderBy({index: r.desc("score")}) .limit(3).changes() Track top three players by

    score Chain the changes command to an actual ReQL query:

33. Changefeeds r.table("table").get(ID).changes() r.table("table").between(X, Y).changes() r.table("table").filter(CONDITION).changes() r.table("table").map(FN).changes() r.table("table").min(INDEX).changes() r.table("table").max(INDEX).changes() r.table("table").orderBy(CONDITION) .limit(N).changes()

    Commands that currently work with changefeeds:

34. DEMO

35. Building Web Apps Using RethinkDB in Node

36. Client Driver • Use a RethinkDB client driver to access

    the database in your app • Official drivers available for Ruby, Python, and JavaScript • Third-party drivers available for other languages like Go and Clojure

37. > Client Driver Install the JS client driver from NPM

    in your Node.js project: $ npm install rethinkdb --save

38. Client Driver var r = require("rethinkdb"); r.connect().then(function(conn) { return r.table("users")

    .insert({name: "Bob"}).run(conn) .finally(function() { conn.close(); }); }).then(function(output) { console.log(output); }); Add Bob to the “users” table

39. Client Driver var r = require("rethinkdb"); r.connect().then(function(conn) { return r.table("users")

    .insert({name: "Bob"}).run(conn) .finally(function() { conn.close(); }); }).then(function(output) { console.log(output); }); Import the RethinkDB module

40. Client Driver var r = require("rethinkdb"); r.connect().then(function(conn) { return r.table("users")

    .insert({name: "Bob"}).run(conn) .finally(function() { conn.close(); }); }).then(function(output) { console.log(output); }); Connect to the database

41. Client Driver var r = require("rethinkdb"); r.connect().then(function(conn) { return r.table("users")

    .insert({name: "Bob"}).run(conn) .finally(function() { conn.close(); }); }).then(function(output) { console.log(output); }); ReQL query that inserts a record

42. Client Driver var r = require("rethinkdb"); r.connect().then(function(conn) { return r.table("users")

    .insert({name: "Bob"}).run(conn) .finally(function() { conn.close(); }); }).then(function(output) { console.log(output); }); Run the query on a connection

43. Client Driver var r = require("rethinkdb"); r.connect().then(function(conn) { return r.table("users")

    .insert({name: "Bob"}).run(conn) .finally(function() { conn.close(); }); }).then(function(output) { console.log(output); }); Close connection when operation is complete

44. Client Driver var r = require("rethinkdb"); r.connect().then(function(conn) { return r.table("users")

    .insert({name: "Bob"}).run(conn) .finally(function() { conn.close(); }); }).then(function(output) { console.log(output); }); Display query response

45. Client Driver var r = require("rethinkdb"); r.connect().then(function(conn) { return r.table("users")

    .insert({name: "Bob"}).run(conn) .finally(function() { conn.close(); }); }).then(function(output) { console.log(output); }).error(function(err) { console.log("Failed:", err); }); Handle errors emitted by Promise

46. Query Composition • ReQL embeds natively in your programming language

    • Pass around ReQL expressions like any other code • You can assign ReQL expressions to variables or store them in functions

47. Query Composition var old = r.row("time").lt(r.now()); r.table("events") .filter(old) .delete().run(); Delete

    events that are in the past Store subexpressions in variables for reuse:

48. Query Composition function olderThan(t) { return r.table("events") .filter(r.row("time").lt(t)); } olderThan(r.now()).delete().run();

    Delete events that are in the past Encapsulate query logic in functions for reuse:

49. Using Express • Express is a Node.js framework for building

    web applications • It does URL routing, parameter parsing, and response handling • Easy to use Express to make simple REST APIS

50. Using Express var app = require("express")(); var r = require("rethinkdb");

    app.listen(8090); console.log("App listening on port 8090"); app.get("/fellowship/species/:species", function(req, res) { r.connect().then(function(conn) { return r.table("fellowship") .filter({species: req.params.species}).run(conn) .finally(function() { conn.close(); }); }) .then(function(cursor) { return cursor.toArray(); }) .then(function(output) { res.json(output); }) .error(function(err) { res.status(500).json({err: err}); }); }); Serve rows with specified species

51. Using Express var app = require("express")(); var r = require("rethinkdb");

    app.listen(8090); console.log("App listening on port 8090"); app.get("/fellowship/species/:species", function(req, res) { r.connect().then(function(conn) { return r.table("fellowship") .filter({species: req.params.species}).run(conn) .finally(function() { conn.close(); }); }) .then(function(cursor) { return cursor.toArray(); }) .then(function(output) { res.json(output); }) .error(function(err) { res.status(500).json({err: err}); }); }); Instantiate an Express app

52. Using Express var app = require("express")(); var r = require("rethinkdb");

    app.listen(8090); console.log("App listening on port 8090"); app.get("/fellowship/species/:species", function(req, res) { r.connect().then(function(conn) { return r.table("fellowship") .filter({species: req.params.species}).run(conn) .finally(function() { conn.close(); }); }) .then(function(cursor) { return cursor.toArray(); }) .then(function(output) { res.json(output); }) .error(function(err) { res.status(500).json({err: err}); }); }); Serve app on desired port

53. Using Express var app = require("express")(); var r = require("rethinkdb");

    app.listen(8090); console.log("App listening on port 8090"); app.get("/fellowship/species/:species", function(req, res) { r.connect().then(function(conn) { return r.table("fellowship") .filter({species: req.params.species}).run(conn) .finally(function() { conn.close(); }); }) .then(function(cursor) { return cursor.toArray(); }) .then(function(output) { res.json(output); }) .error(function(err) { res.status(500).json({err: err}); }); }); Define GET request with URL route

54. Using Express var app = require("express")(); var r = require("rethinkdb");

    app.listen(8090); console.log("App listening on port 8090"); app.get("/fellowship/species/:species", function(req, res) { r.connect().then(function(conn) { return r.table("fellowship") .filter({species: req.params.species}).run(conn) .finally(function() { conn.close(); }); }) .then(function(cursor) { return cursor.toArray(); }) .then(function(output) { res.json(output); }) .error(function(err) { res.status(500).json({err: err}); }); }); Perform RethinkDB query

55. Using Express var app = require("express")(); var r = require("rethinkdb");

    app.listen(8090); console.log("App listening on port 8090"); app.get("/fellowship/species/:species", function(req, res) { r.connect().then(function(conn) { return r.table("fellowship") .filter({species: req.params.species}).run(conn) .finally(function() { conn.close(); }); }) .then(function(cursor) { return cursor.toArray(); }) .then(function(output) { res.json(output); }) .error(function(err) { res.status(500).json({err: err}); }); }); Return query output JSON to user

56. Using Express var app = require("express")(); var r = require("rethinkdb");

    app.listen(8090); console.log("App listening on port 8090"); app.get("/fellowship/species/:species", function(req, res) { r.connect().then(function(conn) { return r.table("fellowship") .filter({species: req.params.species}).run(conn) .finally(function() { conn.close(); }); }) .then(function(cursor) { return cursor.toArray(); }) .then(function(output) { res.json(output); }) .error(function(err) { res.status(500).json({err: err}); }); }); Emit a 500 error if the query fails

57. Using Changefeeds r.connect().then(function(c) { return r.table("fellowship") .changes().run(c); }) .then(function(cursor) {

    cursor.each(function(err, item) { console.log(item); }); }); Display every change on the “fellowship” table

58. Using Changefeeds r.connect().then(function(c) { return r.table("fellowship") .changes().run(c); }) .then(function(cursor) {

    cursor.each(function(err, item) { console.log(item); }); }); Attach a changefeed to the table

59. Using Changefeeds r.connect().then(function(c) { return r.table("fellowship") .changes().run(c); }) .then(function(cursor) {

    cursor.each(function(err, item) { console.log(item); }); }); Iterate over every value passed into the cursor

60. Using Changefeeds r.connect().then(function(c) { return r.table("fellowship") .changes().run(c); }) .then(function(cursor) {

    cursor.each(function(err, item) { console.log(item); }); }); Display received changes in the console

61. Using Socket.io • Powerful framework for realtime client/server communication •

    Supports WebSockets, long polling, and other transports • Lets you send JSON messages between your app and frontend

62. Using Socket.io var sockio = require("socket.io"); var app = require("express")();

    var r = require("rethinkdb"); var io = sockio.listen(app.listen(8090)); r.connect().then(function(conn) { return r.table("players") .orderBy({index: r.desc("score")}) .limit(5).changes().run(conn); }) .then(function(cursor) { cursor.each(function(err, data) { io.sockets.emit("update", data); }); }); Broadcast score changes over Socket.io

63. Using Socket.io var sockio = require("socket.io"); var app = require("express")();

    var r = require("rethinkdb"); var io = sockio.listen(app.listen(8090)); r.connect().then(function(conn) { return r.table("players") .orderBy({index: r.desc("score")}) .limit(5).changes().run(conn); }) .then(function(cursor) { cursor.each(function(err, data) { io.sockets.emit("update", data); }); }); Load the Socket.io module

64. Using Socket.io var sockio = require("socket.io"); var app = require("express")();

    var r = require("rethinkdb"); var io = sockio.listen(app.listen(8090)); r.connect().then(function(conn) { return r.table("players") .orderBy({index: r.desc("score")}) .limit(5).changes().run(conn); }) .then(function(cursor) { cursor.each(function(err, data) { io.sockets.emit("update", data); }); }); Instantiate Socket.io server

65. Using Socket.io var sockio = require("socket.io"); var app = require("express")();

    var r = require("rethinkdb"); var io = sockio.listen(app.listen(8090)); r.connect().then(function(conn) { return r.table("players") .orderBy({index: r.desc("score")}) .limit(5).changes().run(conn); }) .then(function(cursor) { cursor.each(function(err, data) { io.sockets.emit("update", data); }); }); Attach a changefeed to the query

66. Using Socket.io var sockio = require("socket.io"); var app = require("express")();

    var r = require("rethinkdb"); var io = sockio.listen(app.listen(8090)); r.connect().then(function(conn) { return r.table("players") .orderBy({index: r.desc("score")}) .limit(5).changes().run(conn); }) .then(function(cursor) { cursor.each(function(err, data) { io.sockets.emit("update", data); }); }); Broadcast updates to all Socket.io connections

67. Socket.io Frontend <html> <head> <title>Real-time web app</title> <script src="/socket.io/socket.io.js"></script> <script>

    var socket = io.connect(); socket.on("update", function(data) { console.log("Update:", data); }); Receive Socket.io updates on frontend

68. Socket.io Frontend <html> <head> <title>Real-time web app</title> <script src="/socket.io/socket.io.js"></script> <script>

    var socket = io.connect(); socket.on("update", function(data) { console.log("Update:", data); }); Load the Socket.io client script

69. Socket.io Frontend <html> <head> <title>Real-time web app</title> <script src="/socket.io/socket.io.js"></script> <script>

    var socket = io.connect(); socket.on("update", function(data) { console.log("Update:", data); }); Connect to the Socket.io server

70. Socket.io Frontend <html> <head> <title>Real-time web app</title> <script src="/socket.io/socket.io.js"></script> <script>

    var socket = io.connect(); socket.on("update", function(data) { console.log("Update:", data); }); Create handler for “update” messages

71. Socket.io Frontend <html> <head> <title>Real-time web app</title> <script src="/socket.io/socket.io.js"></script> <script>

    var socket = io.connect(); socket.on("update", function(data) { console.log("Update:", data); }); Display update in browser console

72. DEMO

73. Cluster Configuration Sharding and replication

74. Sharding and Replication • RethinkDB is designed for clustering and

    easy scalability • To add a new server to the cluster, just launch it with the join option • Configure sharding and replication per table • Any feature that works with a single database will work in a sharded cluster

75. Add a Server to a Cluster $ rethinkdb --join server:29015

    >

76. Cluster Configuration • Interactively configure your cluster with the web

    UI • Programmatically configure your cluster with simple ReQL commands • Fine-grained cluster control via full ReQL access to system tables

77. DEMO

78. Reconfigure Command r.table("users") .reconfigure({shards: 2, replicas: 1}) Configure table with

    2 shards and 1 replica The reconfigure command changes table settings:

79. Config Command r.table("users").config() { id: "31c92680-f70c-4a4b-a49e-b238eb12c023", name: "users", db: "mydatabase",

    primary_key: "id", shards: [ {primary_replica: "a", "replicas": ["a", "b"]}, {primary_replica: "d", "replicas": ["c", "d"]} ], write_acks: "majority", durability: "hard" }

80. System Tables • table_config: table configurations, including sharding and replication

    • server_config: server names and tags • db_config: database UUIDs and names

81. System Tables • current_issues: cluster errors and problems • stats:

    statistics for the cluster • logs: server logs and messages • jobs: currently-running operations

82. Scalability Tricks • useOutdated: reduce query time by not guaranteeing

    that you use the latest data • durability: can set to soft if you don’t want to wait for writes to be written to disk

83. Additional Resources • RethinkDB website:
 http://rethinkdb.com • RethinkDB cookbook:
 http://rethinkdb.com/docs/cookbook

    • RethinkDB installation:
 http://rethinkdb.com/docs/install/