MongoDB UK 2012: MongoDB Oplog Magic

Mihnea Giurgea, Technical Lead, uberVU.
MongoDB Oplogs are quite easy to manipulate, and are potentially very rewarding, especially when you need to perform live database migrations. This talk will give a short intro into replication and how oplogs work, then a deeper dive into what can be achieved by redirecting oplogs from one cluster to another. For example, you could force two distinct clusters to mirror each other, or just have one cluster copy another.

Transcript

2. Mongo Replica Set • asynchronous replication • multiple nodes that are copies of each other • one primary, multiple secondaries (slaves) • automatic election • writes are only handled by primary only!
3. Mongo Replica Set
4. Oplog • oplog - special collection (capped) • oplog - records each write operation • replicas “tail the oplog” for new updates • new ops are replayed on secondaries
5. Oplog example: insert {u'h': -1469300750073380169L, u'ns': u'mydb.tweets', u'o': {u'_id': ObjectId('4e95ae77a20e6164850761cd'), u'content': u'Lorem ipsum', u'nr': 16}, u'op': u'i', u'ts': Timestamp(1318432375, 1)}
6. Oplog example: update {u'h': -5295451122737468990L, u'ns': u'mydb.tweets', u'o': {u'$set': {u'content': u'Lorem ipsum'}}, u'o2': {u'_id': ObjectId('4e95ae3616692111bb000001')}, u'op': u'u', u'ts': Timestamp(1318432339, 1)}
7. Q: What we want? A: Cross-cluster oplog replay (replay oplogs from one mongo cluster to another). Custom oplog replay
8. Custom oplog replay Q: How do we do that? A: Using OplogReplay (now Open Source) ./oplog-replay localhost:27017 localhost:27018
9. How it works? A: Very similar to MongoDB internal oplog :) tail the oplog for new entries: apply oplog entry save timestamp of last entry
10. How it works? • last timestamp is persisted on destination > oplogreplay.settings.findOne() { "_id" : "misc-lastts", "value" : { "t" : 1335960424000, "i" : 770 } } • restarting will replay entries newer than last timestamp
11. Other features? TODO - explain what else can it do? - also db & collection regexp, start from point-in-time
12. Want more? TODO - show how it can be easily extended? inheritance + skip deletes
13. Inverted pyramid Recent data is more important TODO - add picture here (see notes)
14. Inverted pyramid Q: How can we store historical data cheaper? A: Keep data in two distinct mongo clusters: • recent - only last 30 days, more resources • historical - all data, but less resources (or even more clusters...)
15. Advantages Q: Why bother with distinct mongo clusters? A: Several reasons: • different # of shards • different # of replica sets • more / less RAM • adjust storage size
16. Implementation (1) Setup an oplog-replay between clusters: TODO - add picture ( recent ) ---oplogreplat---> ( historical )
17. Implementation (2) Modify your code to know about separation def get_data(since, until): results = [] T = compute_time_threshold() if since <= T: results += get_hist_data(since, T) if until > T: results += get_rcnt_data(T, until) return results
18. Splitting Replica Sets TODO - show a diagram of before and after split before split after split | blogs |blogs | | posts ------> | posts | | comments | | comments |
19. Splitting Replica Sets • split one mongo cluster in two • with no downtime
20. Splitting Replica Sets Q: How it's done? A:
21. Splitting Replica Sets i. create new node (Secondary), wait for it to catch up ii. stop node, remove from ReplicaSet iii. hack its internal state to look like a NEW replica set iv. stop oplogreplay from point-in-time v. redirect your app code (all at once or one at a time, depending on your application needs)
22. Limitations TODO - mongos → one oplogreplay per shard, but balancer deletes is an issue!
23. Limitations TODO - how to overcome balancer issue?
24. Limitations TODO - if the oplogreplay falls behind for too long, there is no recovery procedure
25. OplogReplay @ uberVU TODO - recent / historical split
26. OplogReplay @ uberVU TODO - admin migration