Replication and Replica Sets

Presented at MongoDB Bangalore 2012

1. 1 Sridhar Nanjundeswaran, 10gen sridhar@10gen.com @snanjund October 26, 2012
2. 2 • Introduction to Replica Sets • Developing with Replica Sets • Operational Considerations • Behind the Curtain
3. Introduction to Replica Sets Why? What is it? Configuration Options
4. 4 • How many have faced node failures? • How many have been woken to do fail overs? • How many have experienced issues due to n/w latency? • Different uses for data – Normal processing – Simple analytics
5. 5 Node 1 Node 2 Node 3
6. 6 Node 1 Node 2 Node 3 RS Init
7. 7 Node 1 Node 2 Node 3 Heartbeat Election
8. 8 Node 1 Secondary Node 2 Secondary Node 3 Primary Replication Replication Heartbeat
9. 9 Node 1 Secondary Node 2 Secondary Node 3 Primary Heartbeat Primary Election
10. 10 Node 1 Secondary Node 2 Primary Node 3 Primary Heartbeat Replication
11. 11 Node 1 Secondary Node 2 Primary Node 3 Recovery Heartbeat Replication
12. 12 Node 1 Secondary Node 2 Primary Node 3 Secondary Replication Heartbeat Replication
13. 13 Node 1 Secondary Node 2 Arbiterr Node 3 Primary Replication Heartbeat
14. 14 > conf = { _id : "mySet", members : [ {_id : 0, host : "A”, priority : 3}, {_id : 1, host : "B", priority : 2}, {_id : 2, host : "C”}, {_id : 3, host : "D", hidden : true}, {_id : 4, host : "E", hidden : true, slaveDelay : 3600} ] } > rs.initiate(conf)
15. 15 > conf = { _id : "mySet”, members : [ {_id : 0, host : "A”, priority : 3}, {_id : 1, host : "B", priority : 2}, {_id : 2, host : "C”}, {_id : 3, host : "D", hidden : true}, {_id : 4, host : "E", hidden : true, slaveDelay : 3600} ] } > rs.initiate(conf) Primary DC
16. 16 > conf = { _id : "mySet”, members : [ {_id : 0, host : "A”, priority : 3}, {_id : 1, host : "B", priority : 2}, {_id : 2, host : "C”}, {_id : 3, host : "D", hidden : true}, {_id : 4, host : "E", hidden : true, slaveDelay : 3600} ] } > rs.initiate(conf) Secondary DC Default priority = 1
17. 17 > conf = { _id : "mySet”, members : [ {_id : 0, host : "A”, priority : 3}, {_id : 1, host : "B", priority : 2}, {_id : 2, host : "C”}, {_id : 3, host : "D", hidden : true}, {_id : 4, host : "E", hidden : true, slaveDelay : 3600} ] } > rs.initiate(conf) Analytics node
18. 18 > conf = { _id : "mySet”, members : [ {_id : 0, host : "A”, priority : 3}, {_id : 1, host : "B", priority : 2}, {_id : 2, host : "C”}, {_id : 3, host : "D", hidden : true}, {_id : 4, host : "E", hidden : true, slaveDelay : 3600} ] } > rs.initiate(conf) Backup node
19. Developing with Replica Sets Consistency Write Preference Read Preference
20. 20 Application Primary Secondary v1 Insert Read
21. 21 Application 1 Primary Secondary v1 Insert Read Application 2 Read
22. 22 Application 1 Primary Secondary v1 Insert Read Application 2 Does not exist Read
23. 23 Application 1 Primary Secondary v1 Insert Read Application 2 Read from primary is always consistent Read
24. 24 Application 1 Primary Secondary v1 v1 Insert Read Replicate Application 2 Read
25. 25 Application 1 Primary Secondary v1 v1 Insert Read Replicate Application 2 Read Read
26. 26 Application 1 Primary Secondary v1 v1 Insert Read Replicate Application 2 Reads V1 Read Read
27. 27 Application 1 Primary Secondary v1 v1 Insert Read Replicate Application 2 Read Read v2 Update
28. 28 Application 1 Primary Secondary v1 v1 Insert Read Replicate Application 2 Read Read v2 Update Read
29. 29 Application 1 Primary Secondary v1 v1 Insert Read Replicate Application 2 Read Read v2 Update Read Read
30. 30 Application 1 Primary Secondary v1 v1 Insert Read Replicate Application 2 Read Read v2 Update Read Read
31. 31 Application 1 Primary Secondary v1 v1 Insert Read Replicate Application 2 Read Read v2 Update v2 Replicate Read Read
32. 32 Application 1 Primary Secondary v1 v1 Insert Read Replicate Application 2 Read Read v2 Update v2 Replicate Read Read Read
33. 33 Application 1 Primary Secondary v1 v1 Insert Read Replicate Application 2 Read Read v2 Update v2 Replicate Read Read Read
34. 34 • Fire and forget • Wait for error • Wait for journal sync • Wait for replication
35. 35 Driver Primary write apply in memory
36. 36 Driver Primary getLastError apply in memory write
37. 37 Driver Primary getLastError apply in memory write j:true Write to journal
38. 38 Secondary replicate Driver Primary getLastError apply in memory write W:majority
39. 39 • Since 2.0.0 • Control over where data is written to • Each member can have one or more tags e.g. – Dc : "ny” – dc: "ny",
ip: "192.168",
rack: "row3rk7” • Replica set defines rules for where data resides • Rules can change without changing app code
40. 40 { _id : "mySet", members : [ {_id : 0, host : "A", tags : {"dc": "ny"}}, {_id : 1, host : "B", tags : {"dc": "ny"}}, {_id : 2, host : "C", tags : {"dc": "sf"}}, {_id : 3, host : "D", tags : {"dc": "sf"}}, {_id : 4, host : "E", tags : {"dc": "cloud"}}] settings : { getLastErrorModes : { allDCs : {"dc" : 3}, someDCs : {"dc" : 2}} } } > db.blogs.insert({...}) > db.runCommand({getLastError : 1, w : "allDCs"})
41. 41 Driver Primary (sf) getLastError apply in memory write W:allDCs Secondary (ny) replicate Secondary (cloud) replicate
42. 42 • 5 modes (new in 2.2) – PRIMARY(only) - Default – PRIMARYPREFERRED – SECONDARY (only) – SECONDARYPREFERRED – NEAREST
43. 43 • Custom read preferences • Control where you read from – E.g. { "disk": "ssd", "use": "reporting" } • Use in conjunction with standard read preferences – Except primary
44. Operational Considerations Upgrade/Maintenance Common Deployment Scenarios
45. 45 • No downtime • Rolling upgrade/maintenance – Start with Secondary – Primary last
46. 46 • Single datacenter • Single switch & power • Points of failure: – Power – Network – Datacenter – Two node failure • Automatic recovery of single node crash Member 1 Member 2 Member 3
47. 47 • Multi datacenter • DR node for safety • Can’t do multi data center durable write safely since only 1 node in distant DC Member 1 Member 2 Member 3 DC 1 DC 2
48. 48 • Three data centers • Can survive full data center loss • Can do w= { dc : 2 } to guarantee write in 2 data centers (with tags) Member 1 Member 2 Member 3 Member 4 DC 2 Member 5 - DR DC 1 DC 3
49. Behind the Curtain Schema Oplog
50. 50 • Local DB (not replicated) – system.replset – oplog.rs • Capped collection • Idempotent version of operation stored
51. 51 • Heartbeat every 2 seconds – Times out in 10 seconds • Missed heartbeat considered node down
52. 52 > db.replsettest.insert({_id:1,value:1}) { "ts" : Timestamp(1350539727000, 1), "h" : NumberLong("6375186941486301201"), "op" : "i", "ns" : "test.replsettest", "o" : { "_id" : 1, "value" : 1 } } > db.replsettest.update({_id:1},{$inc:{value:10}}) { "ts" : Timestamp(1350539786000, 1), "h" : NumberLong("5484673652472424968"), "op" : "u", "ns" : "test.replsettest", "o2" : { "_id" : 1 }, "o" : { "$set" : { "value" : 11 } } }
53. 53 > db.replsettest.update({},{$set:{name : ”foo”}, false, true}) { "ts" : Timestamp(1350540395000, 1), "h" : NumberLong("- 4727576249368135876"), "op" : "u", "ns" : "test.replsettest", "o2" : { "_id" : 2 }, "o" : { "$set" : { "name" : "foo" } } } { "ts" : Timestamp(1350540395000, 2), "h" : NumberLong("- 7292949613259260138"), "op" : "u", "ns" : "test.replsettest", "o2" : { "_id" : 3 }, "o" : { "$set" : { "name" : "foo" } } } { "ts" : Timestamp(1350540395000, 3), "h" : NumberLong("- 1888768148831990635"), "op" : "u", "ns" : "test.replsettest", "o2" : { "_id" : 1 }, "o" : { "$set" : { "name" : "foo" } } }
54. 54 • Full read preference support in mongos – Drivers too • Improved RS lag logging • rs.syncFrom command • buildIndexes setting • replIndexPrefetch setting
55. 55 • Use replica sets • Easy to setup – Try on a single machine • Check doc page for RS tutorials – http://docs.mongodb.org/manual/replication/#t utorials
56. 56 @mongodb © Copyright 2010 10gen Inc. conferences, appearances, and meetups http://www.10gen.com/events http://bit.ly/mongofb Facebook | Twitter | LinkedIn http://linkd.in/joinmongo download at mongodb.org We’re Hiring ! sridhar@10gen.com @snanjund