Tweet
Tweet

Slide 1

Slide 1 text

RethinkDB Distributed Databases

Slide 2

Slide 2 text

Ryan Paul RethinkDB Evangelist @segphault

Slide 3

Slide 3 text

Introduction What is RethinkDB?

Slide 4

Slide 4 text

What is RethinkDB? • Open source database for building realtime web applications • NoSQL database that stores schemaless JSON documents • Distributed database that is easy to scale

Slide 5

Slide 5 text

Built for Realtime Apps • Subscribe to change notifications from database queries • No more polling — the database pushes changes to your app • Reduce the amount of plumbing needed to stream live updates

Slide 6

Slide 6 text

Power and Convenience • Highly expressive query language • Relational features like table joins • Powerful admin UI with point- and-click cluster management

Slide 7

Slide 7 text

Introduction to ReQL RethinkDB Query Language

Slide 8

Slide 8 text

Introduction to ReQL • ReQL embeds natively into your programming language • Compose ReQL queries by chaining commands • ReQL queries are executed on the server

Slide 9

Slide 9 text

Sample ReQL Queries r.table("users") .filter(r.row("age").gt(30)) r.table("users") .pluck("last_name") .distinct().count() r.table("fellowship") .filter({species: "hobbit"}) .update({species: "halfling"})

Slide 10

Slide 10 text

Anatomy of a ReQL Query r.table("users") .pluck("last_name") .distinct().count() Number of unique last names

Slide 11

Slide 11 text

Anatomy of a ReQL Query r.table("users") .pluck("last_name") .distinct().count() Access a database table

Slide 12

Slide 12 text

Anatomy of a ReQL Query r.table("users") .pluck("last_name") .distinct().count() Isolate a document property

Slide 13

Slide 13 text

Anatomy of a ReQL Query r.table("users") .pluck("last_name") .distinct().count() Consolidate duplicate values

Slide 14

Slide 14 text

Anatomy of a ReQL Query r.table("users") .pluck("last_name") .distinct().count() Display the number of items

Slide 15

Slide 15 text

ReQL Commands • Transformations: map, orderBy, skip, limit, slice • Aggregations: group, reduce, count, sum, avg, min, max, distinct, contains • Documents: row, pluck, without, merge, append, difference, keys, hasFields, spliceAt • Writing: insert, update, replace, delete • Control: forEach, range, branch, do, coerceTo, expr

Slide 16

Slide 16 text

DEMO

Slide 17

Slide 17 text

Secondary Indexes • Queries performed against indexes are much faster • Can index on a single property, multiple fields, or arbitrary ReQL expressions

Slide 18

Slide 18 text

Querying an Index r.table("fellowship") .indexCreate("species") r.table("fellowship") .getAll("human", {index: "species"}) Find all humans in “fellowship”

Slide 19

Slide 19 text

Anonymous Functions r.range(5).map(function(i) { return i.mul(2); }) Multiply each value by 2 You can pass anonymous functions to commands like map and reduce:

Slide 20

Slide 20 text

Understanding ReQL • Anonymous function must return a valid ReQL expression • Client driver translates ReQL queries into wire protocol • Can’t mix local application functionality into ReQL queries • In JS use e.g. the mul and gt commands instead of the normal operators

Slide 21

Slide 21 text

The r.row command Multiply each value by 2 You can often use r.row instead of an anonymous function: r.range(5).map(r.row.mul(2))

Slide 22

Slide 22 text

Grouping r.table("songs") .group("artist") .count() Number of songs by each artist Use group to collect records into groups by a shared property:

Slide 23

Slide 23 text

Grouping [ { "group": "Morcheeba", "reduction": 10 }, { "group": "Pink Floyd", "reduction": 145 }, ... ]

Slide 24

Slide 24 text

Grouping • Commands chained after group will operate on the contents of the group • Use the ungroup command to operate on the grouped output

Slide 25

Slide 25 text

Ungroup r.table("songs") .group("artist").count() .ungroup().max("reduction") Artist with the most songs

Slide 26

Slide 26 text

Additional ReQL Features • Geospatial indexing for location- based queries • Support for storing binary objects • Date and time functions for time data • An http command for fetching remote JSON data

Slide 27

Slide 27 text

Realtime Updates Working with Changefeeds

Slide 28

Slide 28 text

Subscribe to change notifications on database queries Changefeeds

Slide 29

Slide 29 text

r.table("users").changes() Changefeeds Track changes on the users table

Slide 30

Slide 30 text

Changefeeds • The changes command returns a cursor that receives updates • Each update includes the new and old value of the modified record

Slide 31

Slide 31 text

Changefeeds r.table("users") .filter({name: "Bob"}).delete() Changefeed output: { new_val: null, old_val: { id: '362ae837-2e29-4695-adef-4fa415138f90', name: 'Bob', ... } }

Slide 32

Slide 32 text

Changefeeds r.table("players") .orderBy({index: r.desc("score")}) .limit(3).changes() Track top three players by score Chain the changes command to an actual ReQL query:

Slide 33

Slide 33 text

Changefeeds r.table("table").get(ID).changes() r.table("table").between(X, Y).changes() r.table("table").filter(CONDITION).changes() r.table("table").map(FN).changes() r.table("table").min(INDEX).changes() r.table("table").max(INDEX).changes() r.table("table").orderBy(CONDITION) .limit(N).changes() Commands that currently work with changefeeds:

Slide 34

Slide 34 text

DEMO

Slide 35

Slide 35 text

Building Web Apps Using RethinkDB in Node

Slide 36

Slide 36 text

Client Driver • Use a RethinkDB client driver to access the database in your app • Official drivers available for Ruby, Python, and JavaScript • Third-party drivers available for other languages like Go and Clojure

Slide 37

Slide 37 text

> Client Driver Install the JS client driver from NPM in your Node.js project: $ npm install rethinkdb --save

Slide 38

Slide 38 text

Client Driver var r = require("rethinkdb"); r.connect().then(function(conn) { return r.table("users") .insert({name: "Bob"}).run(conn) .finally(function() { conn.close(); }); }).then(function(output) { console.log(output); }); Add Bob to the “users” table

Slide 39

Slide 39 text

Client Driver var r = require("rethinkdb"); r.connect().then(function(conn) { return r.table("users") .insert({name: "Bob"}).run(conn) .finally(function() { conn.close(); }); }).then(function(output) { console.log(output); }); Import the RethinkDB module

Slide 40

Slide 40 text

Client Driver var r = require("rethinkdb"); r.connect().then(function(conn) { return r.table("users") .insert({name: "Bob"}).run(conn) .finally(function() { conn.close(); }); }).then(function(output) { console.log(output); }); Connect to the database

Slide 41

Slide 41 text

Client Driver var r = require("rethinkdb"); r.connect().then(function(conn) { return r.table("users") .insert({name: "Bob"}).run(conn) .finally(function() { conn.close(); }); }).then(function(output) { console.log(output); }); ReQL query that inserts a record

Slide 42

Slide 42 text

Client Driver var r = require("rethinkdb"); r.connect().then(function(conn) { return r.table("users") .insert({name: "Bob"}).run(conn) .finally(function() { conn.close(); }); }).then(function(output) { console.log(output); }); Run the query on a connection

Slide 43

Slide 43 text

Client Driver var r = require("rethinkdb"); r.connect().then(function(conn) { return r.table("users") .insert({name: "Bob"}).run(conn) .finally(function() { conn.close(); }); }).then(function(output) { console.log(output); }); Close connection when operation is complete

Slide 44

Slide 44 text

Client Driver var r = require("rethinkdb"); r.connect().then(function(conn) { return r.table("users") .insert({name: "Bob"}).run(conn) .finally(function() { conn.close(); }); }).then(function(output) { console.log(output); }); Display query response

Slide 45

Slide 45 text

Client Driver var r = require("rethinkdb"); r.connect().then(function(conn) { return r.table("users") .insert({name: "Bob"}).run(conn) .finally(function() { conn.close(); }); }).then(function(output) { console.log(output); }).error(function(err) { console.log("Failed:", err); }); Handle errors emitted by Promise

Slide 46

Slide 46 text

Query Composition • ReQL embeds natively in your programming language • Pass around ReQL expressions like any other code • You can assign ReQL expressions to variables or store them in functions

Slide 47

Slide 47 text

Query Composition var old = r.row("time").lt(r.now()); r.table("events") .filter(old) .delete().run(); Delete events that are in the past Store subexpressions in variables for reuse:

Slide 48

Slide 48 text

Query Composition function olderThan(t) { return r.table("events") .filter(r.row("time").lt(t)); } olderThan(r.now()).delete().run(); Delete events that are in the past Encapsulate query logic in functions for reuse:

Slide 49

Slide 49 text

Using Express • Express is a Node.js framework for building web applications • It does URL routing, parameter parsing, and response handling • Easy to use Express to make simple REST APIS

Slide 50

Slide 50 text

Using Express var app = require("express")(); var r = require("rethinkdb"); app.listen(8090); console.log("App listening on port 8090"); app.get("/fellowship/species/:species", function(req, res) { r.connect().then(function(conn) { return r.table("fellowship") .filter({species: req.params.species}).run(conn) .finally(function() { conn.close(); }); }) .then(function(cursor) { return cursor.toArray(); }) .then(function(output) { res.json(output); }) .error(function(err) { res.status(500).json({err: err}); }); }); Serve rows with specified species

Slide 51

Slide 51 text

Using Express var app = require("express")(); var r = require("rethinkdb"); app.listen(8090); console.log("App listening on port 8090"); app.get("/fellowship/species/:species", function(req, res) { r.connect().then(function(conn) { return r.table("fellowship") .filter({species: req.params.species}).run(conn) .finally(function() { conn.close(); }); }) .then(function(cursor) { return cursor.toArray(); }) .then(function(output) { res.json(output); }) .error(function(err) { res.status(500).json({err: err}); }); }); Instantiate an Express app

Slide 52

Slide 52 text

Using Express var app = require("express")(); var r = require("rethinkdb"); app.listen(8090); console.log("App listening on port 8090"); app.get("/fellowship/species/:species", function(req, res) { r.connect().then(function(conn) { return r.table("fellowship") .filter({species: req.params.species}).run(conn) .finally(function() { conn.close(); }); }) .then(function(cursor) { return cursor.toArray(); }) .then(function(output) { res.json(output); }) .error(function(err) { res.status(500).json({err: err}); }); }); Serve app on desired port

Slide 53

Slide 53 text

Using Express var app = require("express")(); var r = require("rethinkdb"); app.listen(8090); console.log("App listening on port 8090"); app.get("/fellowship/species/:species", function(req, res) { r.connect().then(function(conn) { return r.table("fellowship") .filter({species: req.params.species}).run(conn) .finally(function() { conn.close(); }); }) .then(function(cursor) { return cursor.toArray(); }) .then(function(output) { res.json(output); }) .error(function(err) { res.status(500).json({err: err}); }); }); Define GET request with URL route

Slide 54

Slide 54 text

Using Express var app = require("express")(); var r = require("rethinkdb"); app.listen(8090); console.log("App listening on port 8090"); app.get("/fellowship/species/:species", function(req, res) { r.connect().then(function(conn) { return r.table("fellowship") .filter({species: req.params.species}).run(conn) .finally(function() { conn.close(); }); }) .then(function(cursor) { return cursor.toArray(); }) .then(function(output) { res.json(output); }) .error(function(err) { res.status(500).json({err: err}); }); }); Perform RethinkDB query

Slide 55

Slide 55 text

Using Express var app = require("express")(); var r = require("rethinkdb"); app.listen(8090); console.log("App listening on port 8090"); app.get("/fellowship/species/:species", function(req, res) { r.connect().then(function(conn) { return r.table("fellowship") .filter({species: req.params.species}).run(conn) .finally(function() { conn.close(); }); }) .then(function(cursor) { return cursor.toArray(); }) .then(function(output) { res.json(output); }) .error(function(err) { res.status(500).json({err: err}); }); }); Return query output JSON to user

Slide 56

Slide 56 text

Using Express var app = require("express")(); var r = require("rethinkdb"); app.listen(8090); console.log("App listening on port 8090"); app.get("/fellowship/species/:species", function(req, res) { r.connect().then(function(conn) { return r.table("fellowship") .filter({species: req.params.species}).run(conn) .finally(function() { conn.close(); }); }) .then(function(cursor) { return cursor.toArray(); }) .then(function(output) { res.json(output); }) .error(function(err) { res.status(500).json({err: err}); }); }); Emit a 500 error if the query fails

Slide 57

Slide 57 text

Using Changefeeds r.connect().then(function(c) { return r.table("fellowship") .changes().run(c); }) .then(function(cursor) { cursor.each(function(err, item) { console.log(item); }); }); Display every change on the “fellowship” table

Slide 58

Slide 58 text

Using Changefeeds r.connect().then(function(c) { return r.table("fellowship") .changes().run(c); }) .then(function(cursor) { cursor.each(function(err, item) { console.log(item); }); }); Attach a changefeed to the table

Slide 59

Slide 59 text

Using Changefeeds r.connect().then(function(c) { return r.table("fellowship") .changes().run(c); }) .then(function(cursor) { cursor.each(function(err, item) { console.log(item); }); }); Iterate over every value passed into the cursor

Slide 60

Slide 60 text

Using Changefeeds r.connect().then(function(c) { return r.table("fellowship") .changes().run(c); }) .then(function(cursor) { cursor.each(function(err, item) { console.log(item); }); }); Display received changes in the console

Slide 61

Slide 61 text

Using Socket.io • Powerful framework for realtime client/server communication • Supports WebSockets, long polling, and other transports • Lets you send JSON messages between your app and frontend

Slide 62

Slide 62 text

Using Socket.io var sockio = require("socket.io"); var app = require("express")(); var r = require("rethinkdb"); var io = sockio.listen(app.listen(8090)); r.connect().then(function(conn) { return r.table("players") .orderBy({index: r.desc("score")}) .limit(5).changes().run(conn); }) .then(function(cursor) { cursor.each(function(err, data) { io.sockets.emit("update", data); }); }); Broadcast score changes over Socket.io

Slide 63

Slide 63 text

Using Socket.io var sockio = require("socket.io"); var app = require("express")(); var r = require("rethinkdb"); var io = sockio.listen(app.listen(8090)); r.connect().then(function(conn) { return r.table("players") .orderBy({index: r.desc("score")}) .limit(5).changes().run(conn); }) .then(function(cursor) { cursor.each(function(err, data) { io.sockets.emit("update", data); }); }); Load the Socket.io module

Slide 64

Slide 64 text

Using Socket.io var sockio = require("socket.io"); var app = require("express")(); var r = require("rethinkdb"); var io = sockio.listen(app.listen(8090)); r.connect().then(function(conn) { return r.table("players") .orderBy({index: r.desc("score")}) .limit(5).changes().run(conn); }) .then(function(cursor) { cursor.each(function(err, data) { io.sockets.emit("update", data); }); }); Instantiate Socket.io server

Slide 65

Slide 65 text

Using Socket.io var sockio = require("socket.io"); var app = require("express")(); var r = require("rethinkdb"); var io = sockio.listen(app.listen(8090)); r.connect().then(function(conn) { return r.table("players") .orderBy({index: r.desc("score")}) .limit(5).changes().run(conn); }) .then(function(cursor) { cursor.each(function(err, data) { io.sockets.emit("update", data); }); }); Attach a changefeed to the query

Slide 66

Slide 66 text

Using Socket.io var sockio = require("socket.io"); var app = require("express")(); var r = require("rethinkdb"); var io = sockio.listen(app.listen(8090)); r.connect().then(function(conn) { return r.table("players") .orderBy({index: r.desc("score")}) .limit(5).changes().run(conn); }) .then(function(cursor) { cursor.each(function(err, data) { io.sockets.emit("update", data); }); }); Broadcast updates to all Socket.io connections

Slide 67

Slide 67 text

Socket.io Frontend Real-time web app var socket = io.connect(); socket.on("update", function(data) { console.log("Update:", data); }); Receive Socket.io updates on frontend

Slide 68

Slide 68 text

Socket.io Frontend Real-time web app var socket = io.connect(); socket.on("update", function(data) { console.log("Update:", data); }); Load the Socket.io client script

Slide 69

Slide 69 text

Socket.io Frontend Real-time web app var socket = io.connect(); socket.on("update", function(data) { console.log("Update:", data); }); Connect to the Socket.io server

Slide 70

Slide 70 text

Socket.io Frontend Real-time web app var socket = io.connect(); socket.on("update", function(data) { console.log("Update:", data); }); Create handler for “update” messages

Slide 71

Slide 71 text

Socket.io Frontend Real-time web app var socket = io.connect(); socket.on("update", function(data) { console.log("Update:", data); }); Display update in browser console

Slide 72

Slide 72 text

DEMO

Slide 73

Slide 73 text

Cluster Configuration Sharding and replication

Slide 74

Slide 74 text

Sharding and Replication • RethinkDB is designed for clustering and easy scalability • To add a new server to the cluster, just launch it with the join option • Configure sharding and replication per table • Any feature that works with a single database will work in a sharded cluster

Slide 75

Slide 75 text

Add a Server to a Cluster $ rethinkdb --join server:29015 >

Slide 76

Slide 76 text

Cluster Configuration • Interactively configure your cluster with the web UI • Programmatically configure your cluster with simple ReQL commands • Fine-grained cluster control via full ReQL access to system tables

Slide 77

Slide 77 text

DEMO

Slide 78

Slide 78 text

Reconfigure Command r.table("users") .reconfigure({shards: 2, replicas: 1}) Configure table with 2 shards and 1 replica The reconfigure command changes table settings:

Slide 79

Slide 79 text

Config Command r.table("users").config() { id: "31c92680-f70c-4a4b-a49e-b238eb12c023", name: "users", db: "mydatabase", primary_key: "id", shards: [ {primary_replica: "a", "replicas": ["a", "b"]}, {primary_replica: "d", "replicas": ["c", "d"]} ], write_acks: "majority", durability: "hard" }

Slide 80

Slide 80 text

System Tables • table_config: table configurations, including sharding and replication • server_config: server names and tags • db_config: database UUIDs and names

Slide 81

Slide 81 text

System Tables • current_issues: cluster errors and problems • stats: statistics for the cluster • logs: server logs and messages • jobs: currently-running operations

Slide 82

Slide 82 text

Scalability Tricks • useOutdated: reduce query time by not guaranteeing that you use the latest data • durability: can set to soft if you don’t want to wait for writes to be written to disk

Slide 83

Slide 83 text

Additional Resources • RethinkDB website:
 http://rethinkdb.com • RethinkDB cookbook:
 http://rethinkdb.com/docs/cookbook • RethinkDB installation:
 http://rethinkdb.com/docs/install/