Automatic Failover in RethinkDB

Slide 1

Slide 1 text

Automatic Failover Using Raspberry Pis to understand automatic failover RethinkDB Meetup San Francisco, California July 27, 2015

Slide 2

Slide 2 text

Jorge Silva @thejsj Developer Evangelist @ RethinkDB

Slide 3

Slide 3 text

Distributed Systems What makes RethinkDB distributed?

Slide 4

Slide 4 text

What is RethinkDB? • Open source database for building realtime web applications • NoSQL database that stores schemaless JSON documents • Distributed database that is easy to scale

Slide 5

Slide 5 text

What makes it distributed? • Allows simple sharding and replication of tables • Allows you to easily connect nodes to a cluster using `--join`

Slide 6

Slide 6 text

The problem • When one of your nodes goes down, you needed to manually decide what to do

Slide 7

Slide 7 text

Automatic Failover RethinkDB 2.1

Slide 8

Slide 8 text

What's new in 2.1 • RethinkDB 2.1 introduces automatic failover • It uses Raft as the consensus algorithm

Slide 9

Slide 9 text

Replicas • Primary replicas serve as the authoritative copy of the data • Secondary replicas serve as a mirror of the primary replica

Slide 10

Slide 10 text

Automatic Failover • In RethinkDB, automatic failover takes care of promoting secondary replicas into primary replicas when a primary replica is unavailable • The cluster picks new primaries by voting. New server need a majority vote.

Slide 11

Slide 11 text

Automatic Failover Cluster with Raspberry Pis

Slide 12

Slide 12 text

Step #1: Start RethinkDB // Check RethinkDB is running $ ssh [email protected]

Slide 13

Slide 13 text

Step #2: ssh into raspberry pis // Check devices in network $ nmap -sn *.*.*.0/24 // ssh into raspberry pis $ ssh pi@redisgeek $ ssh pi@pishark

Slide 14

Slide 14 text

Step #3: Start RethinkDB // Start RethinkDB in both Raspberry Pis pi@mrpi1 ~ $ rethinkdb \ -n redisgeek \ -t pi -t redisgeek \ --bind all \ --join 104.236.171.225

Slide 15

Slide 15 text

Step #4: Check servers r.db('rethinkdb').table('server_config')

Slide 16

Slide 16 text

Step #5: Insert test data // Insert data into table r.table('data') .insert( // Insert data form Reddit r.http('reddit.com/r/rethinkdb.json') ('data')('children').map(r.row('data')) ) // Query data r.table('data')

Slide 17

Slide 17 text

Step #6: Check replica

Slide 18

Slide 18 text

Automatic Failover Demo #1

Slide 19

Slide 19 text

Step #1: Move data // Move all data to `redisgeek` r.table('data') .reconfigure({ shards: 1, replicas: { 'redisgeek': 1 }, primaryReplicaTag: 'redisgeek' })

Slide 20

Slide 20 text

Step #2: Disconnect primary

Slide 21

Slide 21 text

Step #3: Query data // Query table r.table('data') // Returns Error

Slide 22

Slide 22 text

What happened? • We move all our data in 'redisgeek' • We disconnected 'redisgeek' from the network • Because we can't communicate with 'redisgeek' (primary replica), our data in inaccessible

Slide 23

Slide 23 text

Step #4: Replicate data // Configure 3 replicas r.table('data') .reconfigure({ shards: 1, replicas: 3 })

Slide 24

Slide 24 text

Step #5: Check replicas

Slide 25

Slide 25 text

Step #6: Disconnect primary

Slide 26

Slide 26 text

Step #7: Query data // Query table r.table('data') // We have data!

Slide 27

Slide 27 text

Step #8: Insert data // Insert data r.table('data').insert({ hello: 'world' })

Slide 28

Slide 28 text

What happened? • Node gets promoted to primary replica

Slide 29

Slide 29 text

Step #8: Reconnect primary • 'redisgeek' comes back as primary

Slide 30

Slide 30 text

Automatic Failover Replication and failover #2

Slide 31

Slide 31 text

Step #1: Make main primary // Make `main` the primary replica r.table('data') .reconfigure({ shards: 1, replicas: { 'main': 1, 'redisgeek': 1, 'pishark': 1 }, primaryReplicaTag: 'main' })

Slide 32

Slide 32 text

Step #2: Disconnect secondaries

Slide 33

Slide 33 text

Step #3: Query data // Query table r.table('data').count() // 25

Slide 34

Slide 34 text

Step #4: Attempt Insert r.table('data') .insert({ hello: 'world' }) // Error

Slide 35

Slide 35 text

What happened? • Because the primary replica is connected data can be read • Because a majority of nodes are disconnected, data can't be written

Slide 36

Slide 36 text

Questions • RethinkDB website:  http://rethinkdb.com • New Failover Documentation:  http://docs.rethinkdb.com/2.1/docs/ failover/ • Email me: [email protected] • Tweet: @thejsj, @rethinkdb